JP2022086354A - 3D measurement system and 3D measurement method - Google Patents

Abstract

To provide a system and method capable of performing highly accurate three-dimensional measurement for an object having a glossy surface.SOLUTION: A three-dimensional measurement system includes: a projection section that irradiates an object with sinusoidal pattern light a plurality of times while shifting its phase; an imaging section that captures light reflected by the object; and a measurement processing section that generates a phase image from the captured image using a phase shift method. The measurement processing section inputs at least one of images before or after the measurement processing section performs the phase shift method to a CNN which has learned so that an image obtained by reducing the effect of a specular reflection component is generated.SELECTED DRAWING: Figure 2

Description

The present invention relates to a three-dimensional measurement system and a three-dimensional measurement method for an object having a glossy surface.

The three-dimensional measurement technique is used, for example, in a factory production line to recognize a work position when gripping a work which is an object with a robot hand. As a three-dimensional measurement technique, for example, a method is known in which a pattern image is projected from a projector onto a work, and the reflected light from the work is photographed and analyzed by a camera.

In Patent Document 1, a projection unit that sequentially projects a coded pattern image sequence composed of a plurality of types of coded pattern images and an object on which the coded pattern image is projected by the projection unit are imaged in order, and a plurality of images are taken. An imaging unit that acquires a captured image sequence consisting of captured images, a decoding unit that performs decoding corresponding to mutual reflection using a coded pattern image sequence and a captured image sequence, and a decoding result in the decoding unit and A three-dimensional shape measurement system including a measurement unit that measures the three-dimensional shape of an object by associating a coded pattern image with a captured image using the geometrical relationship between the projection unit and the imaging unit. Is disclosed.

Japanese Unexamined Patent Publication No. 2020-02640

However, it is known that when an active three-dimensional measurement method such as a phase shift method is used for an object having a large specular reflection ratio such as a metal having a glossy surface, the measurement accuracy deteriorates or measurement becomes impossible. Has been done.

Therefore, an object of the present invention is to provide a system and a method capable of three-dimensionally measuring an object having a glossy surface with high accuracy.

The three-dimensional measurement system of the present invention is
A projection unit that irradiates an object with a sinusoidal pattern light multiple times while shifting the phase.
An image pickup unit that captures the reflected light reflected by the object, and
A measurement processing unit that generates a phase image from a captured image using the phase shift method,
Have,
The measurement processing unit inputs at least one image before and after the measurement processing unit executes the phase shift method to the CNN trained to generate an image in which the influence of the specular reflection component is reduced.

In the three-dimensional measurement system of the present invention, it is preferable that the measurement processing unit generates a further phase image from one of the phase image and the image obtained by processing the phase image with CNN by using the epipolar constraint condition.

In the three-dimensional measurement system of the present invention, the measurement processing unit is
Generate a phase image from the captured image using the phase shift method,
The phase image is input to the CNN to generate a phase image in which the influence of the specular reflection component is reduced.
Further phase images are generated from the phase image with the influence of the specular reflection component reduced by using the epipolar constraint condition.
It is preferable to generate a three-dimensional point cloud from the further phase image using the principle of triangulation.

In the three-dimensional measurement method of the present invention,
Irradiate the object with a sine and cosine pattern light multiple times while shifting the phase.
The reflected light reflected by the object is photographed,
Generate a phase image from the captured image using the phase shift method,
At least one of the images before and after performing the phase shift method is input to the CNN trained to generate an image with reduced influence of the specular reflection component.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a system and a method capable of three-dimensionally measuring an object having a glossy surface with high accuracy.

It is a figure which shows the structure of the 3D measurement system by one Embodiment of this invention. It is a figure which shows the flow of the 3D measurement method by one Embodiment of this invention. It shows a sinusoidal pattern light phase-shifted from each other. The CNN constructed in one embodiment of the present invention is shown. It is a figure for demonstrating the correction by epipolar restraint. It is a figure which shows the flow of the 3D measurement method by the modified embodiment of this invention. In this embodiment, two scenes generated by a simulator for generating CNN training data are shown. (A) shows an image of an object piled up in bulk, and (b) shows an image of a three-dimensional point cloud obtained by this embodiment.

FIG. 1 is a diagram showing a configuration of a three-dimensional measurement system according to an embodiment of the present invention.
The three-dimensional measurement system 10 includes a projection unit 1 which is a projector, an image pickup unit 2 which is a camera, and a measurement processing unit 3 which is a computer.
The projection unit 1 irradiates the object 4 with a sinusoidal pattern light a plurality of times while shifting the phase. The image pickup unit 2 captures the reflected light reflected by the object 4.

FIG. 2 is a diagram showing a flow of a three-dimensional measurement method according to an embodiment of the present invention.
In step S1, the measurement processing unit 3 generates a phase image A from the captured image by using a phase shift method.
In step S2, the measurement processing unit 3 inputs the phase image A to the CNN (convolutional neural network) to generate the phase image B in which the influence of the mirror reflection component is reduced.
In step S3, the measurement processing unit 3 generates the phase image C from the phase image B by using the epipolar constraint condition. Note that this step S3 is optional and can be omitted.
In step S4, the measurement processing unit 3 generates a three-dimensional point cloud from the phase image C using the principle of triangulation.

First, the phase shift method will be described.
3 (a) to 3 (d) show sinusoidal pattern lights phase-shifted from each other.
The projection unit 1 irradiates the object 4 with a plurality of phase-shifted sinusoidal pattern lights as shown in FIG. In the phase shift method, as the number of pattern lights increases, the measurement accuracy improves, but the amount of calculation also increases. Therefore, in the present embodiment, the 4step phase shift method using four pattern lights is used, but the number of pattern lights can be any number as long as it is three or more.

Second, CNN will be described.
FIG. 4 shows a CNN constructed in one embodiment of the present invention.
In a normal CNN, local information is lost in the process of convolving an image. However, in the three-dimensional measurement of bulk objects, it is necessary to restore the information of pixels that are difficult to measure due to the influence of the specular reflection component from the local image features around them. Therefore, it is necessary to directly transmit the local image features output in the upper layer close to the input of the CNN to the lower layer by Skip Connection. Therefore, in this embodiment, a CNN based on Unit, which has a Skip Connection and can solve a regression problem on a pixel-by-pixel basis, is used.
In this way, the measurement processing unit 3 trains the CNN in advance so as to generate the phase image B in which the influence of the specular reflection component is reduced from the phase image A.
It is also possible to use a CNN having a structure in which Units are stacked in two stages (see FIGS. 6 (c) and 6 (d) described later). In this case, the influence of the specular reflection component can be reduced more accurately.

Third, epipolar restraint will be described.
FIG. 5 is a diagram for explaining the correction by the epipolar constraint.
When the observation point x is photographed by the camera at the point _{OL, the observation point x is projected onto the point x L on} the projection surface of the camera. At this time, the points x ₁ , x ₂ , and x ₃ between the observation point x and the point x _L are projected onto the epipolar line of the projection surface of the projector at the point _OR .
Thus, in a projector-camera system, there is a condition that the projector pixel corresponding to any camera pixel exists on the corresponding epipolar line, and this epipolar line is calculated in advance using external parameters and internal parameters. be able to.
When there is a point (phase) deviating from the epipolar line, the measurement processing unit 3 draws a perpendicular line from the point onto the epipolar line, and corrects the intersection of the perpendicular line and the epipolar line as a correct point.

In the above-described embodiment, the measurement processing unit 3 executes each step of the phase shift method, the CNN, and the epipolar constraint in this order. First, a phase image A is generated from the captured image using the phase shift method, and the phase image A is input to the CNN to generate a phase image B in which the influence of the mirror reflection component is reduced, thereby photographing in a real environment. The number of images to be displayed can be reduced.
For example, it is assumed that 100 images are used as input data of CNN. When inputting captured images to CNN, 100 images must be captured. On the other hand, when a phase image is first generated from a captured image by first using the phase shift method and the phase image is input to the CNN, the phase image is generated by combining the captured image and the simulation data, for example, 10 images. All you have to do is take an image (the remaining 90 images use simulation data), and you can reduce the time and effort of taking a picture.
However, the order of each step is not limited to the above-described embodiment, and various modifications are possible, and examples of the modifications will be described below.

6 (a) to 6 (d) are diagrams showing a flow of a three-dimensional measurement method according to a modified embodiment of the present invention.
In FIG. 6 (a), the CNN is executed before the phase shift method. That is, the measurement processing unit 3 inputs a captured image, which is an image before executing the phase shift method, to the CNN to generate an image in which the influence of the mirror reflection component is reduced, and then uses the phase shift method to generate a phase. Generate image A.
In the present specification, the expression "generate a phase image from a captured image" is used not only when a phase image is directly generated from a captured image but also as an image processed from a captured image (for example, as shown in FIG. 6A). In addition, a case where a phase image is generated from an image) obtained by processing a captured image in CNN is also included.

In FIG. 6 (b), CNN is performed after epipolar restraint. That is, the measurement processing unit 3 generates a phase image A from the captured image using the phase shift method, generates a phase image B using the epipolar constraint condition, inputs the phase image B to the CNN, and mirror-reflects the image. A phase image C with reduced influence of components is generated.

In FIG. 6 (c), the CNN is executed twice. That is, the measurement processing unit 3 inputs the phase image A to the CNN to generate a phase image B in which the influence of the mirror reflection component is reduced, and inputs the phase image B to the CNN to further influence the influence of the mirror reflection component. Generates a reduced phase image C.

In FIG. 6D, two CNNs are executed before and after the phase shift method. That is, the measurement processing unit 3 inputs the captured image to the CNN to generate an image in which the influence of the mirror reflection component is reduced, generates the phase image A by using the phase shift method, and inputs the phase image A to the CNN. Then, a phase image B in which the influence of the mirror reflection component is reduced is generated.

Hereinafter, examples of the present invention will be described.
In this embodiment, each step was executed in the order shown in FIG.
The configurations of the projector and the camera used in this embodiment are as follows.
Projector: Panasonic PT-VMZ50J
Resolution: 1920 x 1080
Effective luminous flux: 5000 lm
Contrast: 3,000,000: 1
Camera: Baumer VCxU-124
Lens: VS TECHNOLOGY VS-1614H1N
Resolution: 800 x 600
Field of view: 400 x 300 mm
WD: 550 mm
Spatial resolution: 0.50 x 0.50 mm / pixel
In addition, Zhang's method was used for calibration between the projector and the camera. The calibration test results are as follows.

7 (a) and 7 (b) show two scenes generated by the simulator for generating the learning data of CNN in this embodiment.
Blender is used for the simulator, and the settings of Blender are as follows.
Metallic: 1.00
Specular: 0.50
Roughness: 0.30
Diffuse reflection count: 4
Number of specular reflections: 4
Number of renders: 1024 / pixel

As a first experiment, the reduction of the influence of the specular reflection component by the phase shift method, CNN and epipolar constraint was confirmed by simulation. That is, the phase image A generated by the phase shift method, the phase image B generated by the CNN, and the phase image C generated by the epipolar constraint were converted into a three-dimensional point cloud, respectively.
The accuracy of the 3D point group converted from the phase image B is higher than that of the 3D point group converted from the phase image A, and the accuracy is higher than that of the 3D point group converted from the phase image C. It was confirmed visually that it was.
This means that the influence of the specular reflection component could be reduced with each step.

As the second experiment, the measurement accuracy error was evaluated by simulation.
In the conventional method (the method in which CNN was removed from this example), the average was 7.21 mm, the standard deviation (σ) was 6.39 mm, and 3σ was 19.17 mm.
In this example, the average was 0.21 mm, the standard deviation (σ) was 0.68 mm, and 3σ was 1.64 mm.
As a result, it was found that in this embodiment, the measurement accuracy was greatly improved as compared with the conventional method.

As the third experiment, three-dimensional measurement was performed in the actual environment, and the measurement error was obtained.
FIG. 8A shows an image of objects piled up in bulk, and FIG. 8B shows an image of a three-dimensional point cloud obtained in this embodiment.
A three-dimensional point cloud was acquired by the method of this embodiment for one object on the workbench.
The object was moved by 1 mm, 5 mm, and 10 mm, and a three-dimensional point cloud was similarly obtained at the position after each movement.
The 3D point cloud before and after the movement was read by a computer, displayed on the image, and manually superimposed.
The amount of movement between each three-dimensional point cloud was calculated, and the average value of the amount of movement was used as the measurement error.
The measurement error in this environment was 0.4 mm, and it was found that the measurement error was small in this embodiment.

1 ... Projection unit, 2 ... Imaging unit, 3 ... Measurement processing unit, 4 ... Object, 10 ... Three-dimensional measurement system

Claims (4)

It ’s a three-dimensional measurement system.
A projection unit that irradiates an object with a sinusoidal pattern light multiple times while shifting the phase.
An image pickup unit that captures the reflected light reflected by the object, and
A measurement processing unit that generates a phase image from a captured image using the phase shift method,
Have,
The measurement processing unit inputs at least one image before and after the measurement processing unit executes the phase shift method to the CNN trained to generate an image in which the influence of the specular reflection component is reduced.
Three-dimensional measurement system. The measurement processing unit generates a further phase image from one of the phase image and the image obtained by processing the phase image with CNN by using the epipolar constraint condition.
The three-dimensional measurement system according to claim 1. The measurement processing unit
Generate a phase image from the captured image using the phase shift method,
The phase image is input to the CNN to generate a phase image in which the influence of the specular reflection component is reduced.
Further phase images are generated from the phase image with the influence of the specular reflection component reduced by using the epipolar constraint condition.
A three-dimensional point cloud is generated from the further phase image using the principle of triangulation.
The three-dimensional measurement system according to claim 1 or 2. It ’s a three-dimensional measurement method.
Irradiate the object with a sine and cosine pattern light multiple times while shifting the phase.
The reflected light reflected by the object is photographed,
Generate a phase image from the captured image using the phase shift method,
At least one of the images before and after performing the phase shift method is input to the CNN trained to generate an image with reduced effects of specular reflection components.
Three-dimensional measurement method.

Images