JP7544342B2 - Three-dimensional measurement system and three-dimensional measurement method - Google Patents

Description

The present invention relates to a three-dimensional measurement system and a three-dimensional measurement method for an object having a glossy surface.

Three-dimensional measurement technology is used, for example, in factory production lines to recognize the position of a workpiece when the target workpiece is grasped by a robot hand. One known three-dimensional measurement technology is to project a pattern image onto the workpiece from a projector and then capture and analyze the reflected light from the workpiece with a camera.

Patent Document 1 discloses a three-dimensional shape measurement system that includes a projection unit that sequentially projects a sequence of coded pattern images consisting of multiple types of coded pattern images, an imaging unit that sequentially images an object onto which the coded pattern images are projected by the projection unit to obtain a sequence of captured images consisting of multiple captured images, a decoding unit that performs decoding corresponding to interreflection using the coded pattern image sequence and the captured image sequence, and a measurement unit that measures the three-dimensional shape of the object by associating the coded pattern images with the captured images using the results of the decoding in the decoding unit and the geometric relationship between the projection unit and the imaging unit.

JP 2020-020640 A

However, when using active 3D measurement methods such as the phase shift method on objects with a high proportion of specular reflection, such as metals with shiny surfaces, it is known that the measurement accuracy deteriorates or measurement becomes impossible.

Therefore, the present invention aims to provide a system and method capable of performing highly accurate three-dimensional measurements of objects with glossy surfaces.

The three-dimensional measurement system of the present invention comprises:
a projection unit that projects a sinusoidal pattern light onto an object multiple times while shifting the phase of the light;
An imaging unit that captures light reflected by the object;
a measurement processing unit that generates a phase image from the captured image by using a phase shift method;
having
The measurement processing unit inputs at least one of images taken before and after the measurement processing unit performs the phase shift method to a CNN that has been trained to generate an image in which the influence of specular reflection components is reduced.

In the three-dimensional measurement system of the present invention, it is preferable that the measurement processing unit generates a further phase image from either the phase image or an image obtained by processing the phase image using a CNN, using an epipolar constraint condition.

In the three-dimensional measurement system of the present invention, the measurement processing unit is
A phase image is generated from the captured image using a phase shift method;
The phase image is input to a CNN to generate a phase image in which the influence of the specular reflection component is reduced;
generating a further phase image from the phase image in which the influence of the specular reflection component has been reduced, utilizing an epipolar constraint condition;
A three-dimensional point cloud is preferably generated from said further phase images using the principles of triangulation.

In the three-dimensional measurement method of the present invention,
A sinusoidal pattern of light is irradiated onto the target multiple times while shifting the phase.
Photographing the light reflected by the object;
A phase image is generated from the captured image using a phase shift method;
At least one of the images before and after the phase shifting method is applied is input to a CNN trained to generate images with reduced influence of the specular reflection component.

The present invention provides a system and method capable of performing highly accurate three-dimensional measurements of objects with glossy surfaces.

1 is a diagram showing a configuration of a three-dimensional measurement system according to an embodiment of the present invention. FIG. 2 is a diagram showing a flow of a three-dimensional measurement method according to an embodiment of the present invention. 1 shows sinusoidal light patterns that are phase shifted relative to one another. 1 shows a CNN constructed in one embodiment of the present invention. FIG. 13 is a diagram for explaining correction by epipolar constraint. FIG. 11 is a diagram showing the flow of a three-dimensional measuring method according to a modified embodiment of the present invention. In this embodiment, two scenes are shown that are generated by a simulator for generating training data for a CNN. 1A shows an image of a bulk pile of objects, and FIG. 1B shows an image of a three-dimensional point cloud obtained by this embodiment.

FIG. 1 is a diagram showing the configuration of a three-dimensional measurement system according to an embodiment of the present invention.
The three-dimensional measurement system 10 has a projection unit 1 which is a projector, an imaging unit 2 which is a camera, and a measurement processing unit 3 which is a computer.
The projection unit 1 projects a sinusoidal pattern light onto the object 4 multiple times while shifting the phase of the light. The imaging unit 2 captures an image of the light reflected by the object 4.

FIG. 2 is a diagram showing a flow of a three-dimensional measuring method according to an embodiment of the present invention.
In step S1, the measurement processing unit 3 generates a phase image A from a captured image using a phase shift method.
In step S2, the measurement processing unit 3 inputs the phase image A to a CNN (convolutional neural network) to generate a phase image B in which the influence of the specular reflection component is reduced.
In step S3, the measurement processing unit 3 uses the epipolar constraint condition to generate a phase image C from the phase image B. Note that this step S3 is optional and can be omitted.
In step S4, the measurement processing unit 3 generates a three-dimensional point group from the phase image C using the principle of triangulation.

First, the phase shift method will be described.
3(a) to (d) show sinusoidal pattern lights that are phase-shifted from one another.
The projection unit 1 irradiates the target 4 with a plurality of phase-shifted sinusoidal pattern lights as shown in Fig. 3. In the phase shift method, the more the number of pattern lights, the higher the measurement accuracy, but the more the amount of calculation increases. Therefore, in this embodiment, the 4-step phase shift method using four pattern lights is used, but the number of pattern lights can be any number as long as it is three or more.

Secondly, we will explain CNN.
FIG. 4 shows a CNN constructed in one embodiment of the present invention.
In a normal CNN, local information is lost in the process of convolving an image. However, in the three-dimensional measurement of a bulk object, it is necessary to restore information on pixels that are difficult to measure due to the influence of specular reflection components from local image features around them. Therefore, it is necessary to directly transmit local image features output in an upper layer close to the input of the CNN to a lower layer by Skip Connection. Therefore, in this embodiment, a CNN based on Unet that has Skip Connection and can solve regression problems on a pixel-by-pixel basis is used.
In this way, in the measurement processing unit 3, the CNN is trained in advance to generate a phase image B from a phase image A in which the influence of the specular reflection component is reduced.
It is also possible to use a CNN having a two-stage structure of Unets (see FIGS. 6(c) and 6(d) described later). In this case, the influence of the specular reflection component can be reduced with higher accuracy.

Thirdly, the epipolar constraint will be described.
FIG. 5 is a diagram for explaining correction by epipolar constraint.
When an observation point x is photographed by a camera at point _{O_L} , the observation point x is projected onto the camera's projection plane at point _{x_L} . At this time, points _{x_1} , _{x_2} , and _{x_3} between the observation point x and point _{x_L} are projected onto the epipolar line of the projection plane of the projector at point _{O_R} .
Thus, in a projector-camera system, the projector pixel corresponding to any camera pixel must lie on the corresponding epipolar line, and this epipolar line can be calculated in advance using extrinsic and intrinsic parameters.
If there is a point (phase) that deviates from the epipolar line, the measurement processing unit 3 draws a perpendicular line from that point onto the epipolar line, and modifies the point of intersection of the perpendicular line and the epipolar line as the correct point.

In the above-described embodiment, the measurement processing unit 3 executes the steps of the phase shift method, CNN, and epipolar constraint in this order. First, the phase shift method is used to generate a phase image A from a captured image, and the phase image A is input to the CNN to generate a phase image B in which the influence of the specular reflection component is reduced, thereby making it possible to reduce the number of images captured in the real environment.
For example, assume that 100 images are used as input data for CNN. When the captured images are input to CNN, 100 images must be captured. On the other hand, when the phase shift method is first used to generate a phase image from the captured image and input the phase image to CNN, the captured image and simulation data are combined to generate the phase image, so that, for example, 10 images need only be captured (the remaining 90 images use simulation data), and the effort and time required for capturing can be reduced.
However, the order of each step is not limited to that of the above-described embodiment, and various modifications are possible. The following describes such modifications.

6(a) to (d) are diagrams showing the flow of a three-dimensional measuring method according to a modified embodiment of the present invention.
6A, the CNN is executed before the phase shift method. That is, the measurement processing unit 3 inputs the captured image, which is the image before the phase shift method is executed, to the CNN to generate an image in which the influence of the specular reflection component is reduced, and then generates a phase image A using the phase shift method.
In this specification, the expression "generating a phase image from a captured image" includes not only the case where a phase image is generated directly from a captured image, but also the case where a phase image is generated from an image obtained by processing the captured image (for example, an image obtained by processing the captured image in a CNN, as in FIG. 6(a)).

In FIG. 6(b), CNN is executed after epipolar constraint. That is, the measurement processing unit 3 generates a phase image A from the captured image using the phase shift method, generates a phase image B using the epipolar constraint condition, inputs phase image B to the CNN, and generates a phase image C in which the influence of the specular reflection component is reduced.

In FIG. 6(c), CNN is executed twice. That is, the measurement processing unit 3 inputs phase image A to the CNN to generate phase image B in which the influence of the specular reflection component is reduced, and inputs phase image B to the CNN to generate phase image C in which the influence of the specular reflection component is further reduced.

In FIG. 6(d), CNN is executed twice before and after the phase shift method. That is, the measurement processing unit 3 inputs the captured image to the CNN to generate an image in which the influence of the specular reflection component is reduced, generates phase image A using the phase shift method, and inputs phase image A to the CNN to generate phase image B in which the influence of the specular reflection component is reduced.

Hereinafter, an embodiment of the present invention will be described.
In this embodiment, the steps were performed in the order shown in FIG.
The projector and camera used in this embodiment have the following configurations.
Projector: Panasonic PT-VMZ50J
Resolution: 1920 x 1080
Effective luminous flux: 5000lm
Contrast: 3,000,000:1
Camera: Baumer VCxU-124
Lens: VS TECHNOLOGY VS-1614H1N
Resolution: 800 x 600
Field of view: 400 x 300 mm
Working distance: 550 mm
Spatial resolution: 0.50 x 0.50mm/pixel
In addition, the Zhang method was used for the calibration between the projector and the camera. The calibration test results are as follows:

7(a) and (b) show two scenes generated by a simulator for generating training data for a CNN in this embodiment.
Blender was used as the simulator, and the Blender settings were as follows:
Metallic: 1.00
Specular: 0.50
Roughness: 0.30
Diffuse reflection count: 4
Specular reflection count: 4
Number of renders: 1024/pixel

In the first experiment, the reduction of the influence of the specular reflection component by the phase shifting method, CNN, and epipolar constraint was confirmed by simulation. That is, phase image A generated by the phase shifting method, phase image B generated by CNN, and phase image C generated by the epipolar constraint were each converted into a 3D point cloud.
It was visually confirmed that the accuracy of the three-dimensional point cloud converted from phase image B was higher than that of the three-dimensional point cloud converted from phase image A, and that the accuracy of the three-dimensional point cloud converted from phase image C was even higher.
This means that the influence of the specular reflection component can be reduced with each step.

As a second experiment, an evaluation of measurement accuracy error was carried out by simulation.
In the conventional method (a method in which CNN was removed from this example), the average was 7.21 mm, the standard deviation (σ) was 6.39 mm, and 3σ was 19.17 mm.
In this example, the average was 0.21 mm, the standard deviation (σ) was 0.68 mm, and the 3σ was 1.64 mm.
This shows that the measurement accuracy is significantly improved in this embodiment as compared with the conventional method.

In the third experiment, we performed three-dimensional measurements in a real environment and calculated the measurement error.
FIG. 8(a) shows an image of the bulk object, and FIG. 8(b) shows an image of the 3D point cloud obtained by this example.
A three-dimensional point cloud was acquired for one object on a workbench using the method of this embodiment.
The object was moved by 1 mm, 5 mm, and 10 mm, and a three-dimensional point cloud was similarly acquired at each position after the movement.
The three-dimensional point clouds before and after the movement were read into a computer, displayed on the image, and manually overlaid.
The amount of movement between each group of three-dimensional points was calculated, and the average of the amounts of movement was taken as the measurement error.
The measurement error in this environment was 0.4 mm, and it was found that the measurement error was small in this embodiment.

1...Projection unit, 2...Imaging unit, 3...Measurement processing unit, 4...Object, 10...3D measurement system

Claims (2)

1. A three-dimensional measurement system, comprising:
a projection unit that projects a sinusoidal pattern light onto an object multiple times while shifting the phase of the light;
An imaging unit that captures light reflected by the object;
a measurement processing unit that generates a phase image from the captured image by using a phase shift method;
having
The measurement processing unit includes:
A phase image is generated from the captured image using a phase shift method;
The phase image is input to a CNN trained to generate an image with reduced influence of the specular reflection component to generate a phase image with reduced influence of the specular reflection component ;
generating a further phase image from the phase image in which the influence of the specular reflection component has been reduced, utilizing an epipolar constraint condition;
generating a three-dimensional point cloud from said further phase images using the principles of triangulation;
Three-dimensional measuring system. A three-dimensional measurement method, comprising:
A sinusoidal pattern of light is irradiated onto the target multiple times while shifting the phase.
Photographing the light reflected by the object;
A phase image is generated from the captured image using a phase shift method;
The phase image is input to a CNN trained to generate an image with reduced influence of the specular reflection component to generate a phase image with reduced influence of the specular reflection component ;
generating a further phase image from the phase image in which the influence of the specular reflection component has been reduced, utilizing an epipolar constraint condition;
generating a three-dimensional point cloud from said further phase images using the principles of triangulation;
Three-dimensional measurement methods.

Images