JP6702234B2 - Three-dimensional measuring device, three-dimensional measuring method, and program - Google Patents

Description

  The present invention relates to a three-dimensional measurement technique for measuring a three-dimensional position from a captured image of a measurement object, and particularly to a technique for determining whether or not the detection accuracy of the three-dimensional position is sufficient.

  In recent years, a technique for measuring a three-dimensional position of a target (subject) or a distance to the target from a captured image has been widely used. As a method of three-dimensional measurement, there is an active stereo method in which the position of an object is detected by projecting a sine wave pattern or pattern light on which a predetermined spatial coding is performed and using the principle of triangulation.

  In order to obtain the distance or the three-dimensional position from the photographed image by any measurement principle, it is necessary that the object be photographed with sufficient brightness and that the brightness saturation does not occur. If the brightness is not sufficient or the brightness is saturated, accurate measurement cannot be performed. In order to deal with this problem, it is conceivable to shoot a plurality of images with different brightness, but this causes a problem that the processing time becomes long. When photographing is performed under the condition that luminance saturation does not occur, it is necessary to determine whether or not the measurement can be performed with sufficient accuracy in the low luminance region.

  Patent Document 1 discloses that in three-dimensional measurement using the phase shift method, data having a predetermined luminance amplitude or less is not used. Japanese Patent Laid-Open Publication No. 2004-163242 states that the incorrect distance is prevented from being calculated in the low-luminance region by not using the data with low accuracy.

Japanese Patent No. 5352997

  However, when the distance to the area including a plurality of pixels is calculated as the average of the distances of the pixels included in the area, the distance accuracy cannot be obtained only from the luminance amplitude. The distance thus obtained is also influenced by the size of the area and the size of the offset component of the luminance. This is because the larger the area, the more stable the average value, and the larger (brighter) the offset component is, the larger the noise becomes.

  That is, it is not possible to determine whether or not the calculated distance has sufficient accuracy based on only the brightness amplitude in the captured image.

  The present invention has been made in view of the above circumstances, and the present invention determines whether or not the obtained height is reliable in a three-dimensional measuring apparatus that obtains the height of an object to be measured based on a captured image. The aim is to provide possible technology.

A three-dimensional measurement apparatus according to a first aspect of the present invention relates to an image acquisition unit that acquires an image of an object and a plurality of pixels included in an area set in the image based on the image. Height calculating means for obtaining the height of the subject in the pixels and for obtaining the height for the region from the heights of the subject for the plurality of pixels; a variation in subject height for the plurality of pixels; Determining means for determining whether or not the height obtained for the region is reliable, based on the number of pixels of.

  In this aspect, the height calculation means can obtain an average of the heights of the subject of the plurality of pixels as the height of the region.

In this aspect, the determining unit determines that the height of the area is reliable when (standard deviation of subject height of the plurality of pixels)/(number of the plurality of pixels) ^1/2 is equal to or less than a threshold value. It can be determined that it is possible. The standard deviation of the measurement results is generally not calculated unless the measurement is performed multiple times, but if the height of the area is calculated as the average height of the pixels in the area, the height of the area is calculated by the above formula. Can estimate the standard deviation of the measurement results. Therefore, the standard deviation of the height of the area can be obtained from the standard deviation of the height of each pixel in the area, and the certainty of the height of the area can be evaluated.

  In the three-dimensional measuring apparatus according to this aspect, when the determining unit determines that the height of the region is unreliable, the region is enlarged to recalculate the height, or the photographing condition is changed. It is also preferable to further include a control unit that proposes to re-acquire an image or perform these processes. Here, the change of the photographing condition may be any change so that a brighter image can be photographed. Examples of changing the shooting conditions include changing the brightness of illumination, changing the exposure time, and changing the F value of the optical system of the camera.

  In the three-dimensional measuring apparatus according to this aspect, the method of obtaining the height of the subject at each pixel is not particularly limited. Examples of the height calculation method include a phase shift method, a light cutting method, a coded light projection method, and the like. In either method, the brightness value in the captured image affects the calculation accuracy.

The three-dimensional measuring apparatus according to the second aspect of the present invention is
Projection means for projecting light of a sine wave pattern,
Image acquisition means for acquiring an image of the object onto which the light is projected,
Based on a plurality of images captured by changing the phase of light projected on the object, the height of the plurality of pixels included in the area set in the image from the reference plane of the subject in the pixel is increased. Height calculation means for calculating the average height of the plurality of pixels as the height of the area,
If σ/N ^1/2 ≦E _T, where σ is the height variation of the plurality of pixels, N is the number of the plurality of pixels, and E _T is the height tolerance for the region. Determining means for determining that the height for the area is reliable, and otherwise determining that it is not reliable;
Equipped with.

  In this aspect, the height calculating means obtains the phase of the light for each of the plurality of pixels based on the plurality of images, performs phase connection processing and phase connection error correction processing, and then performs the plurality of the plurality of pixels. The determination unit classifies the phase before the correction process into a plurality of classes, and the number of pixels included in a class having the largest number of pixels is the plurality of pixels. It is also preferable to determine that the height for the region is unreliable, even if it is less than a predetermined percentage of the total number of pixels.

  The present invention can be understood as a three-dimensional measuring device including at least a part of the above means. The present invention can also be understood as a three-dimensional measurement method including at least a part of the processing performed by the above means. Further, it can be regarded as a computer program for causing a computer to execute the steps of these methods, or a computer-readable storage medium in which the program is stored non-temporarily. Each of the above configurations and processes can be combined with each other to configure the present invention as long as no technical contradiction occurs.

  According to the present invention, it is possible to determine whether or not the obtained height is reliable in a three-dimensional measuring device that obtains the height of a measurement target based on a captured image.

FIG. 1 is a diagram showing a configuration of a three-dimensional measuring apparatus according to an embodiment. 2A is a diagram showing a pattern of projected light, and FIG. 2B is a diagram showing a measurement principle by the phase shift method. FIG. 3 is a flowchart showing the overall flow of the component floating inspection process, which is one of the board inspections according to the embodiment. FIG. 4 is a flowchart showing the details of the process of obtaining the height of the target area in the board inspection process. 5A is a diagram showing an example of setting the distance detection area, FIG. 5B is a histogram of phase errors before correction of phase connection error, and FIG. 5C is a histogram of phase errors after correction of phase connection error. is there. 6(A) and 6(B) are diagrams showing experimental results for confirming the estimation accuracy of the standard deviation of the area height.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. The three-dimensional measurement system according to the present embodiment projects pattern light onto a measurement target to perform image capturing, and measures the three-dimensional shape of the measurement target using the principle of triangulation.

(First embodiment)
Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. The present embodiment is a three-dimensional measuring device for inspecting a board. The three-dimensional measuring apparatus according to the present embodiment measures a three-dimensional shape of an object to be measured using a principle of triangulation by projecting pattern light on a substrate (an object to be measured) to which components are attached and taking an image. To do. In the present embodiment, the surface height of the board (the distance from the reference plane parallel to the surface on which the board is installed) is measured, and it is inspected whether the parts are properly attached to the board (part floating test). ..

[Constitution]
FIG. 1 is a diagram showing a configuration of a three-dimensional measuring apparatus 100 according to this embodiment. The three-dimensional measuring apparatus 100 is roughly composed of a projection unit 110, a camera 120, and a computing device 130.

  The projection unit 110 projects sine wave pattern light onto the measurement target 150. The projection unit 110 includes a light source such as a halogen lamp or a xenon lamp, a pattern generation element such as a liquid crystal element for forming a pattern on light emitted from the light source, and an optical system such as a microlens. The light projected by the projection unit 110 does not have to be visible light, and may be invisible light such as infrared light.

The pattern of light projected by the projection unit 110 will be described. In the present embodiment, the three-dimensional measurement is performed using the phase shift method, so that the projection unit 110 projects the striped pattern light that changes periodically. FIG. 2A shows an example of a light pattern projected by the projection unit 110. The example shown in FIG. 2A shows pattern light having vertical stripes, that is, pattern light in which the luminance continuously changes in a sinusoidal manner in the horizontal direction and the luminance is constant in the vertical direction. The frequency of the sine wave (the spatial frequency of the pattern light) is not particularly limited. In this embodiment, the vertical stripe pattern light is projected multiple times while shifting the phase. It is necessary to project at least three times to calculate the phase, and considering the measurement accuracy, the larger the number of captured images, the better. In the present embodiment, for ease of measurement, the phase is shifted by 90° (quarter cycle) and projection is performed four times. Hereinafter, the pattern light having the plurality of vertical stripes is referred to as a vertical stripe pattern light set. As described above, the height of the measurement target 150 can be measured by calculating the phase of the change in the brightness generated in the image from the plurality of images captured by projecting each of the vertical stripe pattern light sets on the measurement target 150. ..

  FIG. 2B shows the brightness values I1 to I4 at a certain point in a plurality of (here, four) images captured by changing the phase of the sine wave of the projection light. From these luminance values, the phase shift θ at that point can be obtained (phase restoration). Based on this phase, the height of the point can be obtained by using the principle of triangulation.

  The camera 120 photographs the measuring object 150 onto which the pattern light is projected. The camera 120 takes an image every time the projection unit 110 switches the phase of the pattern light based on a command from the control unit 131. In the present embodiment, the image data captured by the camera 120 is input to the image input unit 133.

  The arithmetic device 130 is a computer including a processor such as a CPU, a memory (storage device) such as RAM or ROM, an interface with an external device, an input device such as a keyboard or a mouse, and an output device such as a display or a speaker. The arithmetic unit 130 has the functions of the control unit 131, the image input unit 133, the height calculation unit 134, the accuracy determination unit 135, the inspection unit 136, the notification unit 137, etc., when the CPU executes the program stored in the memory. provide. Some or all of the functions described above may be realized by a hardware circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Details of each of these functional units will be described with reference to the following flowcharts.

[processing]
[[Overall processing of board inspection]]
FIG. 3 is a flowchart showing the overall flow of the component floating inspection process, which is one of the substrate inspections performed by the three-dimensional measuring apparatus 100 according to this embodiment. The flowchart shown here is an example of processing, and the processing content and processing order may be changed as appropriate. Note that the projection unit 110 and the camera 120 have been calibrated (calibration processing).

  First, in step S102, a sine wave pattern having a different phase for each projection timing is projected from the projection unit 110 onto the measurement object 150, and the camera 120 photographs the measurement object 150 in accordance with the projection of the pattern light. .. The projection unit 110 projects the sine wave pattern light of the designated phase at the designated timing on the measurement object 150 in accordance with the instruction from the control unit 132. The control unit 132 controls the camera 120 so as to capture an image each time the projection unit 110 changes the phase of the pattern light. The image data captured by the camera 120 is input to the image input unit 133 and temporarily stored. The plurality of image data captured in step S102 are collectively referred to as captured image data.

  Next, in step S104, the user sets a plurality of distance detection areas in the captured image via an input unit (not shown). Alternatively, the three-dimensional measuring apparatus 100 automatically sets a plurality of distance detection areas in the captured image using a known technique. It is desirable that the distance detection area is an area in which the heights of the points therein are equal. FIG. 5A shows an example of setting the distance detection area. FIG. 5A shows an image 51 in which a board including the component 52 is photographed. Here, four distance detection areas 53a to 53d are set in the component 52. If the positional relationship between the substrate and the camera at the time of shooting is determined in advance, the distance detection area can be determined in advance.

  As will be described later, one height is calculated for each distance detection area. From the heights of these four regions, it is determined whether the component 52 is horizontally installed at an appropriate position (height), that is, whether the component 52 is properly attached to the board.

  In step S106, the height calculation unit 134 obtains the height of each point (each pixel) included in the area for each of the distance detection areas 53a to 53d, and takes the average of the heights as the height of the area. decide.

  In step S108, the inspection unit 136 determines from the height of each area whether the component 52 is normally attached. Specifically, if the maximum value of the distance difference in each area is within the reference value (allowable variation) and the average of the distances in each area is within the reference (S108-YES), the inspection unit 136. Determines that the component 52 is properly attached (S110). It should be noted that these reference values (allowable variation in height and allowable range of height) are set in advance. The condition for determining whether or not the component is normally attached is not limited to the condition shown here.

  On the other hand, when the maximum value of the distance difference exceeds the reference value in the determination of step S108, it is found that the component 52 is not mounted horizontally, and when the average distance is outside the reference value, it is not mounted at the correct height. I understand. Therefore, in this case (S108-NO), the inspection unit 136 determines that the mounting state of the component 52 is abnormal (S112).

[[Detailed processing of distance calculation processing]]
FIG. 4 is a flowchart showing a detailed flow of the distance calculation process in step S106. FIG. 4 is a process for obtaining the height of one distance detection area. Therefore, the process shown in FIG. 4 is executed for each of the distance detection areas 53a to 53d. Hereinafter, the distance detection area targeted for processing in FIG. 4 is referred to as a target area.

  In step S202, the height calculation unit 134 calculates the phase for each pixel in the target area. The phase φ of each pixel can be converted into the height h by using the difference from the reference phase determined for each pixel by calibration. When obtaining the phase, an operation called phase connection is performed to convert the phase into a continuous quantity over a wide range. Not all phases are correctly connected by phase connection, and a phase connection error may occur. When a phase connection error occurs, 2πi/α (i is an integer, α is a connection magnification) is added to the phase after phase connection as an error.

  FIG. 5B shows a histogram of the phase error after the phase connection. The phase error is the difference between the phase of each pixel and the average phase in the target area. An error due to noise follows a normal distribution (Gaussian distribution), but an error due to a phase connection error takes a discrete value as described above. Therefore, the shape of the histogram is a shape in which α kinds of normal distributions whose centers are shifted by 2πi/α are mixed. Here, since the probability that the phase connection error occurs is lower than the probability that the phase connection error does not occur, the distribution in the vicinity of the phase error 0 becomes the maximum set as shown in FIG. 5B.

  In step S204, the height calculation unit 134 classifies the distribution shown in FIG. 5B into a plurality of classes. By this processing, a plurality of distributions separated by dotted lines in FIG. 5B are obtained.

In step S206, the height calculation unit 134 determines whether or not the number of pixels belonging to the class 54 having the largest number of pixels is equal to or more than a predetermined ratio of the number of pixels of the entire target area. If this determination is YES, the process proceeds to step S208, and the phase connection error is corrected. Specifically, the height calculation unit 134 subtracts 2πi/α from the phase of each class (the integer i is determined for each class). As a result, the histogram of the phase error has one normal distribution 55 having a center of 0 as shown in FIG. On the other hand, if the determination in step S206 is NO, it is not possible to correctly correct the phase connection error. Therefore, it is proposed to proceed to step S220 and change the shooting conditions to perform reshooting.

  In general, the number of phases classified into a class without phase misconnection is sufficiently larger than the number of phases classified into other classes. However, depending on the conditions, there may not be a sufficient difference between the number of elements in the class without connection errors and the number of elements in other classes, and in such a case, the estimation of the class without connection errors may be erroneous. .. In order to prevent such an erroneous determination, the determination in step S206 is introduced. If the condition of step S206 is not satisfied, it is determined that the connection error cannot be corrected. The predetermined ratio in step S206 may be appropriately evaluated by using an experiment, but it is considered that the number of elements in the class having the connection error does not exceed 35% of the total. Therefore, 35% can be used as the predetermined ratio.

  In step S212, the height calculation unit 134 calculates the standard deviation σ of the corrected phase (that is, the standard deviation of the distribution 55 shown in FIG. 5C). Further, in step S214, the height calculation unit 134 acquires the number of pixels N in the target area (that is, the number of elements of the distribution 55).

In step S216, the accuracy determination unit 135 determines whether σ/N ^1/2 is equal to or less than the allowable error E _T in height measurement. The height measurement tolerance E _T is preset by the user and stored in the memory. Since the target area is composed of (estimated) pixels with the same height, if the height of the target area is calculated as the average value of the heights of the pixels in the area, the height of the target area The standard error is σ/N ^1/2 . Therefore, by the determination in step S216, it is possible to determine whether the measurement error of the height of the region is within the allowable error. Note that here, σ/N ^1/2 is set to be equal to or less than the allowable error E _T, but it may be set to be equal to or less than E _T /2 or E _T /3.

  If the determination in step S216 is YES, it can be estimated that the measurement error of the height of the target area (deviation from the true value) is within the allowable error. Therefore, in this case, the process proceeds to step S218, and the height calculation unit 134 determines the average of the heights of the pixels of the target area as the height of the target area.

  On the other hand, if the determination in step S216 is NO, the estimation error of the height of the target area is likely to be larger than the allowable error. Therefore, in this case, the process proceeds to step S220, and the control unit 131 controls the notification unit 137 so as to suggest changing the imaging condition and performing remeasurement. Examples of changing the shooting conditions include changing the brightness of illumination, changing the exposure time, and changing the F value of the optical system of the camera. In step S220, the control unit 131 may automatically change the shooting condition and automatically perform the remeasurement instead of proposing the change of the shooting condition.

[Evaluation results]
6(A) and 6(B) are diagrams showing evaluation results by experiments. FIG. 6(A) shows the evaluation result for a relatively favorable condition region, and FIG. 6(B) shows the evaluation result for a condition condition worse than that of FIG. 6(A).

In this experiment, phase estimation was tried 50 times. A plane area (an example is shown as an area 61 in FIG. 6A and an area 65 in FIG. 6B) for which the average height is obtained is set, and an average phase in the area is set for each experimental data. And the standard deviation was estimated. The standard deviation was calculated for 50 average phases, and the standard deviation of the average phase was compared with the estimated value in each trial. Also, the relationship between the area size (the number of pixels) and the standard deviation was evaluated by changing the area of the area.

  In FIGS. 6A and 6B, a circle 62 indicates the standard deviation of the average phase of 50 times. A vertical line 63 (actually, a plot of 50 points) shows the standard deviation of the average phase estimated from the experimental data (standard deviation of the phase in the area and the number of pixels in the area) of each time. Curve 64 shows the theoretical value of the standard deviation of the average phase.

It can be seen from FIGS. 6A and 6B that the standard deviation can be estimated with an accuracy of about 0.001 radian even when the conditions are relatively poor. Further, it can be seen that when the area size (the number of pixels in the area) increases, the error of the average phase of the area decreases in the order of N ^−1/2 with respect to the area size N.

[Advantageous effects of this embodiment]
Originally, the measurement error cannot be obtained unless the measurement is performed a plurality of times. In the present embodiment, since the height of the area is calculated as the average of the heights of a plurality of pixels in the area, the standard deviation of the height of the area is calculated from the standard deviation of the height of the area and the number of pixels in the area. Can be estimated. Therefore, the height measurement error can be obtained by one measurement (more accurately, one set of pattern light projection and photographing).

  In the board inspection, the luminance amplitude due to the sine wave pattern of the illumination becomes small in the area where the reflectance is low, and the height measurement cannot be performed sufficiently accurately in the pixel unit due to factors such as a decrease in the SN ratio and a phase connection error. On the other hand, in the case of inspecting the component floating in the board inspection, the inspection can be performed as long as the accuracy of variation in the average height of each region is stable rather than each pixel. In the present embodiment, it is possible to determine whether or not the variation in height measurement in area units is within the allowable range. That is, the height can be measured with the required accuracy even in a low amplitude region where the reflectance is low. In addition, it is possible to propose to change the imaging condition and perform remeasurement when sufficient accuracy cannot be obtained.

(Modification)
In the above description, the height of the component is obtained using the phase shift method, but the specific method for obtaining the component height is not limited to the phase shift method. Other available methods include a light cutting method for projecting slit light or spot light and a coded light projection method for projecting spatially coded pattern light. Regardless of the specific method, the average height error in the area can be obtained from the height variation and the area size at each point in the area.

  Further, the three-dimensional measuring apparatus according to the present invention does not necessarily have to be used by incorporating it into the board inspection apparatus or the inspection apparatus. The measurement result of the three-dimensional measurement can be used for any purpose, and the present invention can be understood as a three-dimensional measurement device or a three-dimensional measurement method.

  Further, in the above embodiment, the height of the object surface is obtained, but a position along an arbitrary axis, a position within a certain projection plane, a three-dimensional position, or the like may be a target for measurement. Since the variation of the measurement values of each pixel in the target area follows the normal distribution, the size of the target area should be smaller than the error due to noise when the measurement target is a three-dimensional position. Must be set to.

100: Three-dimensional measurement device 110: Projection unit 120: Camera 130: Arithmetic device 150: Measurement object 131: Control unit 133: Image input unit 134: Three-dimensional position calculation unit 135: Accuracy determination unit 136: Inspection unit 137: Notification Department

Claims (8)

Image acquisition means for acquiring an image of the object,
Based on the image, the height of the subject in each of the plurality of pixels included in the region set in the image is obtained, and the height of the region is calculated from the height of the subject in the plurality of pixels. A height calculation means for obtaining
A determination unit that determines whether or not the height obtained for the region is reliable, based on variations in subject height for the plurality of pixels and the number of the plurality of pixels,
Equipped with
The determination unit determines that the height of the area is reliable when (standard deviation of subject height of the plurality of pixels)/(number of the plurality of pixels) ^1/2 is equal to or less than a threshold value. ,
Three-dimensional measuring device. The height calculation means obtains the average of the heights of the subject of the plurality of pixels as the height of the region,
The three-dimensional measuring device according to claim 1. When the height of the area is determined to be unreliable by the determination means, the area is enlarged to recalculate the height, or the image capturing condition is changed to re-acquire an image, or Further comprising a control means that proposes to perform the processing of
Three-dimensional measurement apparatus according to claim 1 or 2. For each of the pixels, the height of the subject at the pixel is obtained by any one of the phase shift method, the light cutting method, and the coded light projection method.
The three-dimensional measuring device according to any one of claims 1 to 3 . Projection means for projecting light of a sine wave pattern,
Image acquisition means for acquiring an image of the object onto which the light is projected,
Based on a plurality of images captured by changing the phase of light projected onto the object, the height of the plurality of pixels included in the area set in the image from the reference plane of the subject in the pixel is increased. Height calculation means for calculating the average height of the plurality of pixels as the height of the area,
When the standard deviation of the height of the plurality of pixels is σ, the number of the plurality of pixels is N, and the allowable error of the height of the area is E _T , σ/N ^1/2 ≦E _T For example, if the height of the area is determined to be reliable, otherwise it is determined to be unreliable, and
Three-dimensional measuring device equipped with. The height calculation means obtains the phase of the light for each of the plurality of pixels based on the plurality of images, performs phase connection processing and phase connection error correction processing, and then, for each of the plurality of pixels. Is to seek the height of
The determination unit may classify the phase before the correction process into a plurality of classes, and the number of pixels included in the class having the largest number of pixels may be smaller than a predetermined ratio of the total number of the plurality of pixels. , Determining that the height for the region is unreliable,
The three-dimensional measuring device according to claim 5 . An image acquisition step of acquiring an image of the object,
Based on the image, the height of the subject in each of the plurality of pixels included in the region set in the image is obtained, and the height of the region is calculated from the height of the subject in the plurality of pixels. A height calculation step for obtaining
A determination step of determining whether or not the height obtained for the region is reliable, based on variations in subject height for the plurality of pixels and the number of the plurality of pixels;
Only including,
In the determination step, it is determined that the height of the area is reliable when (standard deviation of subject height of the plurality of pixels)/(number of the plurality of pixels) ^1/2 is equal to or less than a threshold value. ,
Three-dimensional measurement method. A program for causing a computer to execute each step of the method according to claim 7 .

Images