JP2021018219A - Three-dimensional shape measurement system, three-dimensional shape measurement method, and three-dimensional shape measurement program - Google Patents

Abstract

To provide a three-dimensional shape measurement system that is advantageous to excluding a decoding point by interreflection and performing a three-dimensional shape measurement even as to an overlapping point where decoding information on a direct reflection component cannot be separated from decoding information on an interreflection component under only an epipolar constraint.SOLUTION: A three-dimensional shape measurement system 100 comprises: a projection part 2 that projects a coded pattern image row in sequence; an imaging unit 1 that images an object to which the coded pattern image is projected in the projection part 2 in sequence, and acquires an imaging image row; a decoding unit 51 that acquires decoding information as separating decoding information on a direct reflection component from decoding information on an interreflection component as to an overlapping point on the basis of the coded pattern image row and imaging image row; a three-dimensional shape measurement unit 53 that obtains decompression by a geometric calculation, using the decoding information, and thereby measures a three-dimensional shape of the object; and a geometry constraint processing unit 522 that performs geometry constraint processing of excluding a decompression point off a prescribed constraint range.SELECTED DRAWING: Figure 2

Description

The present invention is a three-dimensional shape measuring system that measures a three-dimensional shape of an object by projecting a coded pattern image onto the object and photographing the object on which the coded pattern image is projected. The measurement method relates to a three-dimensional shape measurement program.

The three-dimensional measurement technique is a technique useful for inspecting scratches on an object (work) at a production site and grasping a randomly placed object (work) by an arm robot. Among them, the optical measurement method based on the combination of a projector and a camera (hereinafter referred to as "Pro Cam") is a surface measurement method, so it can measure in real time, can reduce costs, and has industrial application value. high. As a three-dimensional measurement method based on the Procam system, a Gray code method, a phase shift method (for example, Non-Patent Documents 1 and 2), and a high-speed measurement method using a special projection pattern (for example, Non-Patent Document 3) are known. ing.

However, when the mirror surface of the object is strong (for example, when the object is made of metal), mirror surface mutual reflection occurs in the recess of the object. As a result, the reflected light of the projected light is observed in a pixel other than the pixel expected from the positional relationship between the camera and the projector (problem 1), or a plurality of reflected lights are superposed on one pixel of the camera and observed (problem 1). Problem 2). The general measurement method using Pro Cam is based on Pro Cam, and the three-dimensional measurement method is based on the premise that the material of the object is Lambertian reflection. If the object has a strong mirror surface, the above problems 1 and 2 make it difficult to accurately measure the concave portion of the object in three dimensions.

For problem 1, the mutually reflected light observed outside the epipolar line can be limited to the epipolar line of the camera image (epipolar constraint) when the directly reflected light is observed. A method for stabilizing the association by removing it has been proposed (for example, Non-Patent Documents 4 to 7). These existing methods require the projection and imaging of a large number of binary patterns for the stability of reflection separation, and immediacy cannot be expected without the use of special equipment.

The Gray code method, which can encode with the minimum number of pixels among the coding methods using binary patterns, allows coordinate coding with the number of projected images _NGC = log ₂ M, where M is the number of pixels in the parallax direction of the projected image. Is.

However, since this method assumes that each camera pixel is affected only by a single projector pixel, the projected light from different pixels on the epipolar line of the projected image is mirror-plane mutual reflection as in Problem 2. If it is observed by one camera pixel, the corresponding point of the mutually reflected light cannot be removed, and an erroneous shape is required.

Brenner, C., Boehm, J. and Guehring, J .: Photogrammetric calibration and accuracy evaluation of a crosspattern stripe projector, Videometrics VI, Vol. 3641, International Society for Optics and Photonics, pp. 164-173 (1998). Sato, K. and Inokuchi, S .: Three-dimensional surface measurement by space encoding range imaging, Journal of Robotic Systems, Vol. 2, pp. 27-39 (1985). Sagawa, R., Furukawa, R. and Kawasaki, H .: Dense 3D reconstruction from high frame-rate video using a static grid pattern, IEEE transactions on pattern analysis and machine intelligence, Vol. 36, No. 9, pp. 1733 -1747 (2014). Furuse, T., Hiura, S. and Sato, K .: Synchronous detection for robust 3-d shape measurement against interreection and subsurface scattering, International Conference on Image Analysis and Processing, Springer, pp. 276-285 (2011). Nayar, SK, Krishnan, G., Grossberg, MD and Raskar, R .: Fast separation of direct and global components of a scene using high frequency illumination, ACM Transactions on Graphics (TOG), Vol. 25, No. 3, pp . 935-944 (2006). O'Toole, M., Achar, S., Narasimhan, SG and Kutulakos, KN: Homogeneous codes for energy-efficient illumination and imaging, ACM Transactions on Graphics (ToG), Vol. 34, No. 4, p. 35 ( 2015). O'Toole, M., Mather, J. and Kutulakos, KN: 3d shape and indirect appearance by structured light transport, Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, IEEE, pp. 3246-3253 (2014) .. Mirdehghan, P., Chen, W. and Kutulakos, K.N .: Optimal Structured Light a la Carte, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6248-6257 (2018).

Therefore, the present invention eliminates the restoration point due to mutual reflection even for the superimposed point where it is not possible to separate whether it is the decoding information of the direct reflection component or the decoding information of the mutual reflection component only by the epipolar constraint. An object of the present invention is to provide a three-dimensional shape measurement system which is advantageous for measuring.

The three-dimensional shape measurement system according to one aspect of the present invention includes a projection means for sequentially projecting a plurality of types of coded pattern images forming a coded pattern image sequence, and an object on which the coded pattern image is projected by the projection means. Decoding to acquire decoding information by performing decoding based on the imaging means for acquiring an imaged image sequence composed of a plurality of captured images by sequentially imaging an object and the coded pattern image string and the captured image string. Decoding means for obtaining the decoding information by separating the superimposed point where the directly reflected light and the indirect reflected light overlap into the decoding information of the directly reflected component and the decoding information of the mutual reflection component. A means, a three-dimensional shape measuring means for measuring the three-dimensional shape of the object by obtaining a restoration point by geometric calculation using the decoding information, and a geometry for excluding the restoration point outside a predetermined constraint range. It has a configuration including a superimposing point processing means for performing constraint processing.

With this configuration, the decoding information of the direct reflection component and the decoding information of the mutual reflection component are separately obtained at the superimposition point by decoding, and when the three-dimensional shape measurement is performed using them, the mutual reflection Corresponding points of direct reflection are restored outside the restoration range by using the property that the corresponding points of direct reflection are restored within the restoration range by eliminating the restoration points outside the restoration range. It is possible to measure (restore) the three-dimensional shape by appropriately excluding the restoration points due to mutual reflection even for the overlapping points that cannot be separated only by the epipolar constraint that restores.

In the above three-dimensional shape measurement system, the superimposition point processing means may eliminate the decoding information of the mutual reflection component at the superimposition point by epipolar constraint processing that limits the restoration point on the epipolar line, and the three-dimensional The shape measuring means may measure the three-dimensional shape by using the decoding information in which the decoding information of the mutual reflection component is excluded by the epipolar constraint process.

With this configuration, since the decoding information of the mutual reflection component is excluded by the epipolar constraint, the decoding information of the mutual reflection component that cannot be separated by the geometry constraint can be excluded.

In the above three-dimensional shape measurement system, the superimposition point processing means may include a constraint range setting means for setting the constraint range in the geometry constraint process.

With this configuration, geometry constraint processing can be performed according to the set constraint range.

In the above three-dimensional shape measurement system, the constraint range setting means may set the constraint range according to a designation by the user.

With this configuration, the user can set the constraint range.

In the above three-dimensional shape measuring system, the restraint range setting means may set the restraint range according to the shape of the object.

With this configuration, the restraint range corresponding to the shape of the object can be set.

In the above three-dimensional shape measuring system, the restraint range setting means may set the restraint range according to the position of the object.

With this configuration, the restraint range corresponding to the position of the object can be set.

The three-dimensional shape measuring system may further include a detection means for detecting the object, and the restraint range setting means may set the restraint range according to the detection result of the detection means.

With this configuration, the constraint range can be set based on the detection of the actual object.

The three-dimensional shape measuring system may further include a second three-dimensional shape measuring means for measuring the shape of the object with a coarser accuracy than the three-dimensional shape measuring means, and the restraint range setting means may be provided. Based on the measurement result of the second three-dimensional shape measuring means, the restraint range may be set so as to include the object.

With this configuration, the restraint range can be set based on the rough shape measurement of the target object by the second three-dimensional measuring means.

In the above three-dimensional shape measurement system, the coded pattern image sequence may be generated based on a random number.

With this configuration, it is possible to generate a coded pattern image string having low zero average normalization cross-correlation between the signals of each pixel obtained in the coded pattern image string and having high orthogonality.

In the above three-dimensional shape measurement system, the decoding means identifies the superposition point by comparing the ratio of the maximum normalized cross-correlation value to the second-order normalized cross-correlation value with a predetermined threshold value. You can do it.

With this configuration, the superimposition points can be identified by utilizing the high orthogonality of the coded pattern image sequence.

The three-dimensional shape measurement method of one aspect of the present invention includes a projection step of sequentially projecting a plurality of types of coded pattern images forming a coded pattern image sequence, and an object on which the coded pattern image is projected in the projection step. Directly reflected light and indirect reflection on the captured image based on the imaging step of acquiring an captured image sequence consisting of a plurality of captured images by sequentially imaging an object, the coded pattern image sequence, and the captured image sequence. Three-dimensional shape measurement that measures the three-dimensional shape of the object by obtaining the restoration point by geometric calculation using the decoding step to search for the corresponding point including the overlapping point where the light overlaps and the decoding information. It has a configuration including a step and a superimposing point processing step for performing a geometry constraint processing for excluding the restoration point outside a predetermined constraint range.

Even with this configuration, the decoding information of the direct reflection component and the decoding information of the mutual reflection component are separately obtained by decoding at the superimposition point, and when three-dimensional shape measurement is performed using them, mutual reflection is performed. Correspondence of direct reflection by eliminating the restoration points outside the restoration range by using the property that the points restored by the correspondence points of are restored outside the restoration range and the correspondence points of direct reflection are restored within the restoration range. It is possible to measure (restore) the three-dimensional shape by appropriately excluding the restoration points due to mutual reflection even for the overlapping points that cannot be separated only by the epipolar constraint that restores the points.

The three-dimensional shape measurement program according to one aspect of the present invention includes a projection step in which a computer projects a plurality of types of coded pattern images forming a coded pattern image sequence on a projection means in order, and a projection step on the imaging means. Based on the imaging step of sequentially imaging an object on which the coded pattern image is projected to acquire an captured image sequence composed of a plurality of captured images, and the coded pattern image sequence and the captured image sequence. The object is obtained by geometrically calculating a restoration point using the decoding step of searching for a corresponding point including an overlapping point in which the directly reflected light and the indirect reflected light overlap on the captured image and the decoding information. It has a configuration in which a three-dimensional shape measurement step for measuring the three-dimensional shape of the above and a superposition point processing step for performing a geometry constraint process for excluding the restoration point outside a predetermined constraint range are executed.

Even with this configuration, the decoding information of the direct reflection component and the decoding information of the mutual reflection component are separately obtained by decoding at the superimposition point, and when three-dimensional shape measurement is performed using them, mutual reflection is performed. Correspondence of direct reflection by eliminating the restoration points outside the restoration range by using the property that the points restored by the correspondence points of are restored outside the restoration range and the correspondence points of direct reflection are restored within the restoration range. It is possible to measure (restore) the three-dimensional shape by appropriately excluding the restoration points due to mutual reflection even for the overlapping points that cannot be separated only by the epipolar constraint that restores the points.

According to the present invention, even for a superposed point that cannot be separated only by the epipolar constraint that restores the corresponding point of direct reflection, the restoration point due to mutual reflection can be appropriately excluded to measure (restore) the three-dimensional shape.

FIG. 1 is a diagram showing an application scene of the three-dimensional shape measurement system according to the embodiment of the present invention. FIG. 2 is a block diagram showing a functional configuration in the three-dimensional shape measurement system according to the embodiment of the present invention. FIG. 3 is a diagram showing a coded pattern image sequence projected by the projection unit of the embodiment of the present invention. FIG. 4 is a flowchart of coded pattern image generation according to the embodiment of the present invention. FIG. 5 is a map showing the maximum ZNCC value obtained by decoding according to the embodiment of the present invention and the ZNCC value at the second position. FIG. 6 is a flowchart of decoding according to the embodiment of the present invention. FIG. 7 is a diagram showing an example in which the direct reflection component and the indirect reflection component of the embodiment of the present invention are superimposed. FIG. 8 is an explanatory diagram of the geometry constraint processing according to the embodiment of the present invention. FIG. 9 is a diagram showing the result of association (when N = 30) according to the embodiment of the present invention. FIG. 10 is a diagram showing a shape measurement result by the three-dimensional shape measuring unit according to the embodiment of the present invention. FIG. 11 is a graph showing the effect of the geometry constraint processing according to the embodiment of the present invention. FIG. 12 is a flowchart of three-dimensional shape measurement according to the embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the embodiments described below are examples of cases where the present invention is carried out, and the present invention is not limited to the specific configurations described below. In carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted.

FIG. 1 is a diagram showing an application scene of the three-dimensional shape measurement system according to the embodiment of the present invention. The three-dimensional shape measuring system 100 includes a projector 10, a camera 20, and a three-dimensional shape measuring device 30. The projector 10 projects a coded pattern image that changes at a predetermined frame rate. The camera 20 is installed so as to photograph the work W, which is an object of three-dimensional shape measurement. The camera 20 captures a coded pattern image projected by the projector 10 and reflected by the work W at a predetermined frame rate to generate an captured image.

The geometrical relationship between the projector 10 and the camera 20, that is, the relationship between the position and the posture is fixed at the time of operation, and this geometrical relationship is known in the three-dimensional shape measuring device 30. Further, the projector 10 and the camera 20 are both connected to the three-dimensional shape measuring device 30 so as to be able to communicate by wire or wirelessly.

The three-dimensional shape measuring device 30 measures the three-dimensional shape including the position information of the work W based on the coded pattern image projected by the projector 10 and the captured image captured by the camera 20. The three-dimensional shape measuring device 30 may be realized by, for example, a general-purpose computer executing the three-dimensional shape measuring program of the present embodiment.

In the example of FIG. 1, the three-dimensional shape measuring device 30 is connected to the arm robot 40. The arm robot 40 includes an arm having a plurality of joints, and performs a task of gripping the work W with respect to the work W. The result of the three-dimensional shape measurement of the work W obtained by the three-dimensional shape measuring device 30 is given to the arm robot 40. The arm robot 40 operates based on the position information of the work W and the three-dimensional shape information. As a result, the arm robot 40 can appropriately work on the target work W, for example, can appropriately pick the work W.

FIG. 2 is a block diagram showing a functional configuration in the three-dimensional shape measurement system according to the embodiment of the present invention. The three-dimensional shape measurement system 100 includes an imaging unit 1, a projection unit 2, a synchronization control unit 3, a coded pattern image generation unit 4, an information processing unit 5, and a three-dimensional information output unit 6. ..

The imaging unit 1 is composed of a camera 20. The projection unit 2 is composed of a projector 10. The synchronization control unit 3, the coding pattern image generation unit 4, the information processing unit 5, and the three-dimensional information output unit 6 may be provided in the three-dimensional shape measuring device 30, or only a part thereof has a three-dimensional shape. It may be provided in the measuring device 30, and a part of the other may be provided in the projector 10, the high-speed camera 20, or another device connected to the three-dimensional shape measuring device 30.

The imaging unit 1 performs imaging to obtain an captured image. The imaging unit 1 continuously performs imaging at a high frame rate of about 1000 to 3000 fps to obtain an captured image sequence composed of a plurality of continuous captured images. In the present embodiment, the imaging unit 1 detects only the brightness value of the incident light and generates a monochrome image as an captured image. Further, in the present embodiment, the image pickup unit 1 obtains a captured image using an image pickup device having a resolution of 1024 × 768 pixels.

The projection unit 2 projects a coded pattern image based on the coded pattern image generation information generated by the coded pattern image generation unit 4. The projection unit 2 sequentially projects a plurality of coded pattern images constituting the coded pattern image sequence at about 60 fps. As the projection unit 2, a DLP projector capable of high-speed driving (for example, about 5000 Hz) using a DMD may be used. In this case, the camera 20 constituting the imaging unit 1 may be a high-speed camera. Further, in the present embodiment, the projection unit 2 uses a DMD of 480 × 360 pixels and has a 1-dot size of 5 × 5 pixels.

The coded pattern image generation unit 4 generates coded pattern image generation information and supplies it to the projection unit 2. FIG. 3 is a diagram showing a coded pattern image sequence projected by the projection unit of the embodiment of the present invention. The coded pattern image generation unit 4 generates a black-and-white binary random dot pattern as a coded pattern image based on the time-series modulation signal data of random dots related to decoding.

The coded pattern image generation unit 4 generates a coded pattern image using a Mersenne Twister or an M-sequence random number as a random number. As a result, it is possible to obtain a coded pattern image sequence having low zero average normalization cross-correlation between the signals of each pixel and high orthogonality.

The coded pattern image generation unit 4 generates a coded pattern image sequence composed of a plurality of different coded pattern images at a speed higher than the frame rate of the projection unit 2.

The coded pattern image generation unit 4 repeatedly outputs N types of coded pattern image generation information (N = 6912 in the present embodiment) in order. That is, the coded pattern image generation unit 4 generates information on the coded pattern image for the time frame of t = 1, information on the generation of the coded pattern image for the time frame of t = 2, and for the time frame of t = 3. The generation information of the coded pattern image, ..., The generation information of the coded pattern image for the time frame of t = N is generated in order.

In the present embodiment, the coded pattern image generation unit 4 generates a black-and-white binary random dot pattern as the coded pattern image. As a result, the orthogonality of the coded signal of each dot can be easily increased as compared with other pattern images, and since the signal of each dot is spatially independent, the number of shots can be increased or decreased flexibly. It can be carried out.

FIG. 4 is a flowchart of coded pattern image generation according to the embodiment of the present invention. First, the coded pattern image generation unit 4 specifies pixels and generates a random number from 0 to 255 based on an appropriate seed (step S41). Then, if the generated random number is 128 or less (YES in step S42), the coded pattern image generation unit 4 sets the pixel value of the pixel to 255 (step S43), and the generated random number is the threshold value. If it is less than 128 (NO in step S42), the pixel value of the pixel is set to 0 (step S44).

If the pixel value has not been determined for all the pixels of the N images (NO in step S45), the specified pixel is advanced by one, steps S41 to S44 are repeated, and all the pixels of all the images are covered. When the pixel value is determined (YES in step S45), the process ends. By this process, N black-and-white binary random pattern images are generated.

The synchronization control unit 3 synchronizes the imaging unit 1 with the projection unit 2. As a result, the imaging unit 1 can reliably capture an image of each time frame of the coded pattern image sequence that changes in time series by the projection unit 2.

The information processing unit 5 generates an image sequence composed of a plurality of captured images obtained by the imaging unit 1 in a time series, and a plurality of coded pattern images generated in a time series generated by the coded pattern image generation unit 4. Get information. The coded pattern image generation information acquired by the information processing unit 5 from the coded pattern image generation unit 4 is also referred to as a calibration parameter. The information processing unit 5 measures the three-dimensional shape of the object W based on the captured image and the coded pattern image generation information (calibration parameter). The information processing unit 5 includes a decoding unit 51, a superposition point processing unit 52, and a three-dimensional shape measurement unit 53.

Based on the coded pattern image sequence and the captured image sequence, the decoding unit 51 separates the decoding information into the direct reflection component and the mutual reflection component for each pixel in the region where the projected light is superimposed. Here, as described above, in the present embodiment, since the coded pattern image that changes according to the time series is projected, N steps are taken for each pixel by N images captured corresponding to N time frames. A dot pattern that changes with is obtained.

The decoding unit 51 uses zero-mean normalized cross-correlation (ZNCC) for the correlation calculation. The decoding unit 51 calculates the ZNCC from the observation signal obtained from each captured image and the fully coded signal, obtains the signal having the maximum ZNCC value zncc ₁ and the second-order ZNCC value zncc ₂ , and the ratio r of these. _{By discriminating} a point where _zncc = znccc ₁ / znccc ₂ is _{equal to} or higher than the threshold value th _znccc as a direct reflection point and identifying other points as superimposition points, the superimposition points and other points are distinguished. While doing so, extract the corresponding points. Here, the superimposing point is a point where the directly reflected light and the indirect reflected light overlap due to mutual reflection or the like. In the present embodiment, the threshold value th _zncc is set to 2.0, and restoration is performed using a corresponding point having a ZNCC value of 0.4 or more.

FIG. 5 is a map showing the maximum ZNCC value obtained by decoding according to the embodiment of the present invention and the ZNCC value at the second position. In FIG. 5, the largest ZNCC value (“First” in FIG. 5) is located on the left side and the second ZNCC value (“Second” in FIG. 5) is located on the right side. Further, in FIG. 5, the upper row shows the number of shots N = 50, and the lower row shows the number of shots N = 20. From FIG. 5, it can be seen that as N increases in the region exposed to the projected light, the difference between the maximum ZNCC value and the second-position ZNCC value increases, and the decoding stability increases.

The decoding unit 51 searches for which signal of the sum of the two coded signals has the maximum ZNCC value at the extracted superimposition point. At this time, if the ZNCC value is obtained for all combinations of coded signals, the calculation becomes enormous. Therefore, the decoding unit 51 superimposes the signals by using the following equation (1) if the signals are orthogonal to each other. Find the ZNCC value of the point signal.

ここで、ＣＣはＺＮＣＣであり、ｆは直接反射光の符号化信号であり、ｇは相互反射光の符号化信号であり、ｈは重畳点の観測信号を表す。

Here, CC is ZNCC, f is a coded signal of the directly reflected light, g is a coded signal of the mutually reflected light, and h is an observation signal of the superimposition point.

FIG. 6 is a flowchart of decoding according to the embodiment of the present invention. First, the decoding unit 51 acquires time-series modulated signal data of random dots related to decoding used for generating the coded pattern image string obtained from the coded pattern image generation unit 4 (step S61). .. The decoding unit 51 inputs the captured image sequence obtained by imaging the coded pattern image string from the imaging unit 1 (step S62).

The decoding unit 51 acquires a signal for luminance modulation in the time direction in each pixel of the captured image (step S63). The decoding unit 51 calculates ZNCC using the signal obtained at each pixel and the time-series modulated signal data of random dots (step S64). The decoding unit 51 adopts the signal having the smallest ZNCC value and performs decoding (step S65).

If the result of the decoding is not a composite signal of the sum of the two signals (NO in step S66), the decoding unit 51 acquires the signal as one decoding information (step S67). When the result of the decoding is a composite signal of the sum of the two signals (YES in step S66), the decoding unit 51 separates the two decoding information and acquires the signal (step S68).

In this way, the decoding unit 51 is in a state where one corresponding point of direct reflection and mutual reflection is obtained on the projected image with respect to the superimposed point, and the superimposed point processing unit 52 is restored at the corresponding point of mutual reflection. Eliminate the points that have been made. The superimposition point processing unit 52 includes an epipolar constraint processing unit 521, a geometry constraint processing unit 522, and a constraint range setting unit 523.

The epipolar constraint processing unit 521 eliminates the decoding information of the mutual reflection component at the superposition point by removing the corresponding points on the epipolar line from the decoding information obtained from the decoding unit 51 by the epipolar constraint processing.

Regarding the decoding information obtained from the decoding unit 51, the geometry constraint processing unit 522 removes some of the decoding information by the epipolar constraint processing unit 521, and then further, the mutual overlap points that cannot be excluded by the epipolar constraint alone. Decoding information of the reflection component is excluded by geometry constraint.

For this purpose, the three-dimensional shape measuring unit 53 uses the decoding information after the epipolar constraint processing is performed, that is, the decoding information from which the decoding information of the mutual reflection component is excluded, to determine the restoration point by geometric calculation. By obtaining it, the three-dimensional shape is measured (restored), and the restored point is output to the geometry constraint processing unit 522. The geometry constraint processing unit 522 determines whether or not the restoration point obtained from the three-dimensional shape measurement unit 53 is within the restoration range, and excludes the restoration point outside the restoration range.

The constraint range setting unit 523 sets the range of the geometry constraint in the geometry constraint processing unit 522. The geometry constraint processing unit 522 performs geometry constraint processing in the constraint range set by the constraint range setting unit 523 to eliminate restoration points due to mutual reflection. The setting of the range of geometry constraint in the constraint range setting unit 523 will be described in detail later.

FIG. 7 is a diagram showing an example in which the direct reflection component and the indirect reflection component of the embodiment of the present invention are superimposed. In the example shown in FIG. 7, since the work W has a mirror surface property, the light projected from the two points 85 and 86 on the epipolar line 83 on the projected image 81 projected from the projector 10 is the captured image of the camera 20. It is observed at one point 87 on the epipolar line 84 on 82.

FIG. 8 is an explanatory diagram of the geometry constraint processing according to the embodiment of the present invention. In FIG. 8, the restoration range 91 in the depth direction is set to include the work W. The restoration point 94 using the correspondence point 92 is on the surface of the work W, and the restoration point 95 using the correspondence point 93 is in a different place from the work W. Further, FIG. 8 shows the optical center 95 of the camera 20 and the optical center 96 of the projector 10. Of the two corresponding points of the directly reflected light correspondence point 94 and the mutual reflected light correspondence point 93, the point (restoration point) measured three-dimensionally at the mutual reflected light correspondence point 93 is restored outside the restoration range 91. Therefore, the geometry constraint processing unit 522 eliminates this restoration point.

As a result, the corresponding points obtained by the mutually reflected light in FIG. 7 are likely to be restored to a position away from the restoration range 91 of the work W as shown in FIG. 8, and thus the geometry constraint processing unit 522 By eliminating the corresponding points, it is possible to effectively eliminate the restoration points of the mutually reflected light for many overlapping points in the case of FIG. 7.

The three-dimensional information output unit 6 outputs the restoration points remaining in the geometry constraint processing in the geometry constraint processing unit 522 as the measurement result of the three-dimensional shape.

FIG. 9 is a diagram showing the result of association (when N = 30) according to the embodiment of the present invention. FIG. 9A is an image captured by the camera 20 of the experimental scene. As shown in FIG. 9A, in this example, there is a white box on the left and an aluminum L-shaped angle is arranged on the right. FIG. 9B shows the result when the geometric constraint is not used. The coordinate coding information is also shown in the upper left of FIG. 9B.

FIG. 9C shows the result of performing the epipolar constraint process, and FIG. 9E shows the result of performing the epipolar constraint process and the geometry constraint process. 9 (D) shows a part of the results in FIGS. 9 (C) and 9 (E) in an enlarged manner and side by side.

In the case of FIG. 9 (B) in which the geometric constraint is not used, overlapping points occur on the L-shaped angle and the decoding error is relatively large, but in FIG. 9 (C) in which the epipolar constraint process is performed. As a result, the coding error due to the superposition point is greatly reduced. Further, when the geometry constraint processing is also performed, as shown in FIG. 9 (D), coding errors due to mutual reflection (see the black frame portions in FIGS. 9 (C) and 9 (D)) can be further reduced. I understand.

FIG. 10 is a diagram showing a shape measurement result by the three-dimensional shape measuring unit according to the embodiment of the present invention. FIG. 10 (A) is the result of using the conventional method, FIG. 10 (B) is the result of not using the geometric constraint, and FIG. 10 (C) is the result of performing the epipolar constraint process. As a result, FIG. 10 (D) shows the result when the epipolar constraint process and the geometry constraint process are performed.

Although the mutual reflection component cannot be determined by the result of the conventional method shown in FIG. 10 (A), the mutual reflection component can be determined in FIGS. 10 (B) to 10 (D). Focusing on the measurement points near the dent of the L-shaped angle, FIG. 10 (D) is the densest and can be measured with less noise due to the effect of performing geometry constraint processing in addition to epipolar constraint processing. ..

FIG. 11 is a graph showing the effect of the geometry constraint processing according to the embodiment of the present invention. The epipolar common point is composed of a near point and a far point, and the epipolar common point where two points on the epipolar line on the projected image are superimposed with the projected light from a distant position due to mutual reflection is defined as a "far point". , The epipolar shared point on which the leaked light of the neighboring dots is superimposed is defined as the "near point".

In FIG. 11, each graph shows the number of points generated when the number of projected sheets N is changed. The vertical axis of FIG. 11A shows the change in the points of the superimposition point and the epipolar common point, and the vertical axis of FIG. 11B shows the change of the points of the far point and the near point constituting the epipolar common point. , The vertical axis of FIG. 11C shows the change in the number of distant points removed by the geometry constraint process.

Looking at the overall tendency, when the number of shots N is relatively small (N = 10 to 29), the reliability of the direct reflection signal is low and there are many points where it is not correctly decoded, so that many overlapping points occur. There are few epipolar sharing points. On the other hand, when the number of shots N is relatively large (N = 30 to 50), the reliability of the direct reflection signal is high and the number of points to be correctly decoded increases, so that the number of superimposition points decreases and the epipolar shared point. Tends to increase.

As shown in FIGS. 11B and 11C, when the number of projected images N is large, the vocalization ratio between the far point and the near point is about 1: 4, and the restoration point due to mutual reflection among the far points About 60% of the above can be eliminated by geometry constraint processing. In this example, restoration with less noise is possible with a minimum number of projected images N = 30. As described above, when the number of projected sheets N is reduced, the number of overlapping points at which the corresponding points of the directly reflected light cannot be determined only by the epipolar constraint processing increases, and the effect of the geometry constraint processing becomes remarkable.

Next, the setting of the constraint range of the geometry constraint process in the constraint range setting unit 523 will be described. As is clear from the above description, the constraint range must completely include the object, and the smaller the constraint range, the more effectively the geometry constraint process can eliminate the restoration point of the mutually reflected light. Such a restraint range can be set by various methods.

The constraint range does not necessarily have to be set as a three-dimensional closed space, and may be released in the one-dimensional or two-dimensional direction. For example, the constraint range may be set only in the depth direction and released in the plane direction perpendicular to the depth direction, or the constraint range may be set within the projection angle of view of the projection unit 2 and the constraint range in the projection depth direction may be set. The conditions do not have to be set.

First, the constraint range can be set according to the user's specification. In this case, the constraint range designation from the user is input to the constraint range setting unit 523, and the constraint range setting unit 523 sets the constraint range according to this designation. The user can specify an arbitrary range within the coordinates set in relation to the imaging unit 1 and the projection unit 2. For example, when the constraint range is a rectangular parallelepiped box shape, the user can specify the constraint range by specifying the height, position, and posture of the box.

In particular, when the shape, position, and posture of the object are known, the user can specify the restraint range according to the known shape, position, and posture. For example, in the case of 3D shape measurement for working on an object as described above, the position and posture of the object may be unknown, but 3D for shape inspection of the object. In the case of shape measurement, the shape, position, and posture of the object may be known. When the shape, position, and posture of the object are known, the user can specify the restraint range in advance according to the known shape, position, and posture of the object.

Further, the restraint range setting unit 523 may set the restraint range according to the shape of the object. As described above, when the object has an L-shaped angle, the restraint range may be set to an L-shaped angle shape including the L-shaped angle according to the shape.

Further, the three-dimensional shape measurement system 100 may further include a detection unit for detecting an object, and the restraint range setting unit 523 may set the restraint range according to the detection result of the detection unit. That is, when the position of the object is unknown, the detection unit detects the object using an existing method, and the constraint range setting unit 523 sets the constraint range at the position where the object is detected by the detection unit. You can do it.

Further, the three-dimensional shape measurement system 100 may further include a second three-dimensional shape measurement unit that measures the shape of the object with a coarser accuracy than the above-mentioned three-dimensional shape measurement. Then, the restraint range setting unit 523 may set the restraint range so that the object is included based on the measurement result of the second three-dimensional shape measurement unit. Since the rough shape, position, and posture of the object can be obtained by the second three-dimensional shape measuring unit, the restraint range setting unit 523 sets the restraint range so as to include it.

Alternatively, an object having a shape, position, and posture that should be a restraint range is placed in the space where the object should exist, and the object is measured by the second three-dimensional shape measurement unit, and the restraint range setting unit 523 determines. The measured shape, position, and posture may be used as they are as the restraint range.

FIG. 12 is a flowchart of three-dimensional shape measurement according to the embodiment of the present invention. First, the signal processing unit 5 acquires the geometric calibration parameters obtained in advance (step S121). The projection unit 2 projects the coded pattern image sequence generated by the coded pattern image generation unit 4 (step S122). The imaging unit 1 acquires a plurality of projected coded pattern image sequences to acquire the captured image sequence (step S123).

The coding unit 51 decodes the captured image using the captured image sequence (step S124). The epipolar constraint processing unit 521 removes the decoding information of the mutual reflection component from the decoding information by performing the epipolar constraint processing on the superposed point (step S125). The three-dimensional shape measuring unit 53 measures (restores) the three-dimensional shape from the decoding information including the remaining overlapping points, and outputs the three-dimensional shape to the geometry constraint processing unit 522 (step S126).

The geometry constraint processing unit 522 excludes the restoration points outside the restoration range from the restoration points obtained from the three-dimensional shape measurement unit 53 (step S127), and the three-dimensional information output unit 6 sets the remaining restoration points as the object. It is output as three-dimensional shape data indicating the three-dimensional shape (step S128).

As described above, according to the present embodiment, the decoding information of the direct reflection component and the decoding information of the mutual reflection component are separately obtained at the superimposition point by decoding, and the three-dimensional shape measurement is performed using them. When the above is performed, the points restored at the corresponding points of mutual reflection are restored outside the restoration range, and the corresponding points of direct reflection are restored within the restoration range, and the restoration points outside the restoration range are excluded. By doing so, it is possible to measure (restore) the three-dimensional shape by appropriately excluding the restoration points due to mutual reflection even for the overlapping points that cannot be separated only by the epipolar constraint that restores the corresponding points of direct reflection.

(Modification example 1)
In the above embodiment, the coded pattern image projected on the object is captured by the camera 20 (imaging unit 1), but an event camera may be used instead of the camera 20. The event camera is a camera capable of detecting an event for each pixel according to the amount of change in the brightness value. Event cameras have a wider dynamic range than ordinary cameras, and are advantageous in that they are compact and can be used with low memory. By utilizing this property of the event camera, the three-dimensional shape can be effectively measured even when the object is a translucent object other than metal.

(Modification 2)
In the above embodiment, a black-and-white binary random dot pattern is used as the coded pattern image, but a line-type M-sequence image or the like may be used instead.

(Modification 3)
In the above embodiment, the three-dimensional shape measurement system 100 is configured by using one projector 10 and one camera 20, but the number of projectors 10 and cameras 20 may be plural. For example, a combination of one projector and a plurality of cameras 20, a combination of a plurality of projectors 10 and one camera 20, a combination of a plurality of projectors 10 and a plurality of projectors 10 and a plurality of cameras 20. 3D shape measurement system 100 may be configured with.

When there are a plurality of projectors 10, each projector 10 projects a unique coded pattern image sequence, and when the ZNCC of the coded signal generated by each projector is relatively low, the decoding information Separation is possible.

According to the present invention, even for a superposed point that cannot be separated only by the epipolar constraint that restores the corresponding point of direct reflection, the restoration point due to mutual reflection can be appropriately excluded to measure (restore) the three-dimensional shape of the object. It is useful as a three-dimensional shape measurement system or the like that measures the three-dimensional shape of the object by projecting the coded pattern image onto the object and photographing the object on which the coded pattern image is projected.

1 Imaging unit 2 Projection unit 3 Synchronous control unit 4 Coding pattern image generation unit 5 Information processing unit 51 Decoding unit 52 Superimposition point processing unit 521 Epipolar constraint processing unit 522 Geometry constraint processing unit 523 Constraint range setting unit 53 3D shape measurement Part 6 3D shape output part 10 Projector 20 Camera 30 3D shape measuring device 40 Arm robot 100 3D shape measuring system W work (object)

Claims (12)

A projection means that sequentially projects multiple types of coded pattern images that form a coded pattern image sequence,
An imaging means that acquires an image sequence composed of a plurality of captured images by sequentially imaging an object on which the coded pattern image is projected by the projection means.
It is a decoding means that obtains decoding information by performing decoding based on the coded pattern image sequence and the captured image sequence, and is directly reflected at an overlapping point where directly reflected light and indirect reflected light overlap. Decoding means for acquiring the decoding information by separating the decoding information of the component and the decoding information of the mutual reflection component,
A three-dimensional shape measuring means for measuring the three-dimensional shape of the object by obtaining a restoration point by geometric calculation using the decoded information.
Superimposition point processing means that performs geometry constraint processing that excludes the restoration points outside the predetermined constraint range, and
3D shape measurement system equipped with. The superimposition point processing means eliminates the decoding information of the mutual reflection component at the superimposition point by epipolar constraint processing that limits the restoration point on the epipolar line.
The three-dimensional shape according to claim 1, wherein the three-dimensional shape measuring means measures the three-dimensional shape by using the decoding information from which the decoding information of the mutual reflection component is excluded by the epipolar constraint process. Measurement system. The three-dimensional shape measuring system according to claim 1 or 2, wherein the superimposing point processing means includes a constraint range setting means for setting the constraint range in the geometry constraint process. The three-dimensional shape measuring system according to claim 3, wherein the restraint range setting means sets the restraint range according to a designation by a user. The three-dimensional shape measuring system according to claim 3, wherein the restraint range setting means sets the restraint range according to the shape of the object. The three-dimensional shape measuring system according to claim 3, wherein the restraint range setting means sets the restraint range according to the position of the object. Further provided with a detection means for detecting the object,
The three-dimensional shape measurement system according to claim 3, wherein the restraint range setting means sets the restraint range according to the detection result of the detection means. A second three-dimensional shape measuring means for measuring the shape of the object with a coarser accuracy than the three-dimensional shape measuring means is further provided.
The three-dimensional shape measuring system according to claim 3, wherein the restraint range setting means sets the restraint range so as to include the object based on the measurement result of the second three-dimensional shape measuring means. The three-dimensional shape measurement system according to any one of claims 1 to 6, wherein the coded pattern image sequence is generated based on a random number. 3. The three according to claim 9, wherein the decoding means identifies the superposition point by comparing the ratio of the maximum normalized cross-correlation value to the second-ranked normalized cross-correlation value with a predetermined threshold value. Dimensional shape measurement system. A projection step that sequentially projects multiple types of coded pattern images that form a coded pattern image sequence,
An imaging step in which an object on which the coded pattern image is projected is sequentially imaged in the projection step, and an captured image sequence composed of a plurality of captured images is acquired.
Based on the coded pattern image sequence and the captured image sequence, a decoding step of searching for a corresponding point including a superposed point in which the directly reflected light and the indirect reflected light overlap on the captured image, and a decoding step.
A three-dimensional shape measurement step of measuring the three-dimensional shape of the object by obtaining a restoration point by geometric calculation using the decoding information, and
A superposition point processing step that performs a geometry constraint process that excludes the restored points outside the predetermined constraint range, and
A three-dimensional shape measurement method equipped with. On the computer
A projection step of sequentially projecting a plurality of types of coded pattern images forming a coded pattern image sequence on the projection means,
An imaging step in which the imaging means sequentially images an object on which the coded pattern image is projected in the projection step to acquire an image sequence composed of a plurality of captured images.
Based on the coded pattern image sequence and the captured image sequence, a decoding step of searching for a corresponding point including a superposed point in which the directly reflected light and the indirect reflected light overlap on the captured image, and a decoding step.
A three-dimensional shape measurement step of measuring the three-dimensional shape of the object by obtaining a restoration point by geometric calculation using the decoding information, and
A superposition point processing step that performs a geometry constraint process that excludes the restored points outside the predetermined constraint range, and
3D shape measurement program to execute.

Images