WO2024201662A1

WO2024201662A1 - Object detection device

Info

Publication number: WO2024201662A1
Application number: PCT/JP2023/012243
Authority: WO
Inventors: 大雅佐藤
Original assignee: ファナック株式会社
Priority date: 2023-03-27
Filing date: 2023-03-27
Publication date: 2024-10-03

Abstract

The present invention provides an object detection device comprising: a model transformation unit that applies one or more transformation processes to a model image and a model feature to generate one or more transformed model images and one or more transformed model features; a first characteristic calculation unit that calculates a first characteristic related to a characteristic of a feature extraction process by a feature extraction unit, on the basis of a transformed model feature and a third image feature extracted from the transformed model image by the feature extraction unit; and an evaluation index calculation unit that calculates an evaluation index for the feature extraction process on the basis of one or more first characteristics calculated by the first characteristic calculation unit on the basis of one or more third image features and one or more transformed model features, wherein the feature extraction unit has a plurality of feature extraction processes, the first characteristic calculation unit calculates the one or more first characteristics on the basis of the third image feature and the transformed model feature for each of the plurality of feature extraction processes, and the evaluation index calculation unit calculates the evaluation index for determining the feature extraction process to apply to the detection process by the detection unit for each of the plurality of feature extraction processes.

Description

Object detection device

This disclosure relates to an object detection device.

　A robot system is known that detects the position and orientation of an object based on an image captured by a visual sensor and has the robot perform a task on the object. In detection, first, image features that represent specific parts of the object, whose position and orientation is known, are extracted from the captured image by feature extraction processing and registered together with the position and orientation. For objects whose position and orientation is unknown, image features that represent specific parts of the object are similarly extracted from the captured image by feature extraction processing, and the position and orientation of the object is identified based on the amount of change in the position and orientation in the image obtained by comparing it with the registered image features (model features).

Patent documents 1 and 2 describe a device that identifies the position and orientation of an object by performing a matching process using edge points.

JP 2019-211903 A JP 2017-91079 A

The image features used for matching change depending on the feature extraction process applied. For example, even with the same edge feature extraction process, the edge features extracted by a Sobel filter and a Laplacian filter differ. Furthermore, if the imaging conditions change and the appearance of the object changes, the image features output by the feature extraction process will change. There is a demand for an object detection device that can determine a feature extraction process that can stably detect the object even in situations where the imaging conditions change.

One aspect of the present disclosure is an object detection device that includes a feature extraction unit that extracts image features from an image, a model image receiving unit that receives a first image of an object whose position and orientation are known as a model image, a model feature storage unit that stores the first image feature extracted from the model image by the feature extraction unit as a model feature, and a detection unit that identifies the position and orientation of the object by comparing a second image feature extracted by the feature extraction unit from a second image of the object whose position and orientation are unknown with the model feature, the object detection device further including a model conversion unit that applies one or more conversion processes to the model image and the model feature to generate one or more converted model images and one or more converted model features, respectively, and a model conversion unit that converts the converted model image into a converted model image by the feature extraction unit. The object detection device includes a first characteristic calculation unit that calculates a first characteristic related to the characteristics of the feature extraction process by the feature extraction unit based on the extracted third image feature and the conversion model feature, and an evaluation index calculation unit that calculates an evaluation index of the feature extraction process based on one or more first characteristics calculated by the first characteristic calculation unit based on the one or more third image features and the one or more conversion model features, where the feature extraction unit has a plurality of feature extraction processes, the first characteristic calculation unit calculates the one or more first characteristics for each of the plurality of feature extraction processes based on the third image feature and the conversion model feature, and the evaluation index calculation unit calculates the evaluation index for each of the plurality of feature extraction processes to determine the feature extraction process to be applied to the detection process of the detection unit.

These and other objects, features and advantages of the present invention will become more apparent from the detailed description of exemplary embodiments of the present invention illustrated in the accompanying drawings.

FIG. 1 is a diagram illustrating a device configuration of a robot system including an object detection device according to an embodiment. FIG. 2 is a functional block diagram of a robot control device. FIG. 13 is a diagram showing a result of applying a Sobel filter to an image of a captured object. FIG. 13 is a diagram showing a result of applying a Laplacian filter to an image of a captured object. FIG. 11 is a data flow diagram showing the flow of data in the feature extraction process determination process. FIG. 11 illustrates an example of a conversion process. 6B is a flowchart showing the feature extraction process determination process. 6B is a flowchart showing the feature extraction process determination process together with FIG. 6A. FIG. 13 is a diagram illustrating an example of a list of evaluation indexes. FIG. 13 is a diagram illustrating an example of processing by a Shi-Tomasi corner detector.

Next, an embodiment of the present disclosure will be described with reference to the drawings. In the drawings, similar components or functional parts are given similar reference symbols. The scale of these drawings has been appropriately changed to facilitate understanding. Furthermore, the form shown in the drawings is one example for implementing the present invention, and the present invention is not limited to the form shown.

1 is a diagram showing the equipment configuration of a robot system 100 including an object detection device 60 according to one embodiment. As shown in FIG. 1, the robot system 100 includes a robot 10, a robot control device 50 that controls the robot 10, a visual sensor 70, and a teaching device 40 connected to the robot control device 50. The visual sensor 70 is mounted on the tip of the arm of the robot 10. The visual sensor 70 is connected to the robot control device 50 and operates under the control of the robot control device 50. The robot control device 50 is equipped with a function as an object detection device 60 that detects the position and orientation of an object 1 placed on, for example, a workbench 2 based on an image captured by the visual sensor 70. The robot control device 50 can cause the robot 10 to perform a predetermined task on the object 1 based on the detected position and orientation of the object 1.

As an example, the robot 10 is a six-axis vertical articulated robot. Note that various types of robots may be used as the robot 10 depending on the work target, such as a horizontal articulated robot, a parallel link type robot, or a dual-arm robot. The robot 10 can perform the desired work using an end effector attached to the wrist. The end effector is an external device that can be replaced depending on the application, such as a hand, a welding gun, or a tool. Figure 1 shows an example in which a hand 33 is used as an end effector.

The visual sensor 70 may be a camera that captures two-dimensional images such as grayscale images or color images, or a stereo camera or three-dimensional sensor that can obtain distance images or three-dimensional point clouds. In this embodiment, it is assumed that the visual sensor 70 is a camera that captures two-dimensional images. The robot control device 50 holds model data of the object and can execute a detection process that identifies the position and orientation of the object by matching the image of the object in the captured image with the model data (pattern matching). In this embodiment, it is assumed that the visual sensor 70 has been calibrated, and that the robot control device 50 holds calibration data that defines the relative positional relationship between the visual sensor 70 and the robot 10. This makes it possible to convert a position on an image captured by the visual sensor 70 into a position on a coordinate system (such as a robot coordinate system) fixed to the working space.

The robot control device 50 controls the operation of the robot 10 according to an operation program or commands from the teaching device 40. The robot control device 50 may have a hardware configuration as a general computer having a processor, memory (ROM, RAM, non-volatile memory, etc.), a storage device 152 (see FIG. 2), an operation unit, an input/output interface, a network interface, etc.

The teaching device 40 is used as an operation terminal for teaching the robot 10, performing various settings, displaying information, etc. A teaching operation panel, a tablet terminal, etc. may be used as the teaching device 40. The teaching device 40 may have a hardware configuration as a general computer having a processor, memory (ROM, RAM, non-volatile memory, etc.), a storage device, an operation unit, a display unit 41 (see Figure 2), an input/output interface, a network interface, etc. The display unit 41 is equipped with, for example, a liquid crystal display as a display device.

FIG. 2 is a functional block diagram of the robot control device 50. As shown in FIG. 2, the robot control device 50 has a function as an object detection device 60 in addition to a function for controlling the robot 10.

The robot control device 50 includes an operation control unit 151 and a storage device 152. The storage device 152 is, for example, a storage device formed of a non-volatile memory or a hard disk device. The storage device 152 stores a robot program for controlling the robot 10, a program (vision program) for performing image processing such as workpiece detection based on images captured by the visual sensor 70, calibration data, various setting information, etc.

The motion control unit 151 controls the motion of the robot according to the robot program or according to commands from the teaching device 40. The robot control device 50 is equipped with a servo control unit (not shown) that performs servo control on the servo motors of each axis according to commands for each axis generated by the motion control unit 151.

The object detection device 60 has a function of detecting an object from an image captured by the visual sensor 70. In the detection process for detecting the position and orientation of an object, generally, the image features (edge points, etc.) of the object in the input image are extracted by applying feature processing such as a filter to the input image, and the position and orientation of the object in the input image are identified by comparing the extracted image features with pre-prepared model data (image features) of the object. In a system for handling objects such as that shown in FIG. 1, a filter for extracting edge features of the object may be used as a feature extraction process. However, even for the same edge feature extraction process, there are multiple types of filters, such as Sobel filters and Laplacian filters. The object detection device 60 according to this embodiment provides a function for automatically determining a feature extraction process suitable for stably detecting a certain object.

2, the object detection device 60 includes a visual sensor control unit 161, an image acquisition unit 162, a detection unit 163, a feature extraction unit 164, a model image reception unit 165, a matching area reception unit 166, a model feature storage unit 167, a model conversion unit 168, a stability calculation unit (first characteristic calculation unit) 169, a sensitivity calculation unit (second characteristic calculation unit) 170, an evaluation index calculation unit 171, and a display control unit 172. These functional blocks may be realized by the processor of the robot control device 50 executing a program. In this case, the components of the object detection device 60 in FIG. 2 correspond to the processor of the robot control device 50.

The visual sensor control unit 161 controls the operation of the visual sensor 70. For example, the visual sensor control unit 161 can control the visual sensor 70 according to instructions for the visual sensor 70 in an operation program. The image acquisition unit 162 has a function of acquiring image information obtained by the visual sensor 70 capturing an image within its field of view. In this embodiment, the image acquisition unit 162 acquires a two-dimensional image from the visual sensor 70.

The detection unit 163 can perform a detection process to identify the position and orientation of an object by comparing image features (hereinafter, mainly these image features) extracted by the feature extraction unit 164 from an image captured of an object whose position and orientation are unknown (hereinafter, this image is also referred to as a second image) with model features of the object prepared in advance. The model features are, for example, image features extracted by the feature extraction unit 164 from an image captured of an object whose position and orientation are known.

The feature extraction unit 164 has multiple types of feature extraction processes that extract image features from an image. The feature extraction processes have multiple filters that extract edge features, for example. The multiple types of filters include, for example, Sobel filters and Laplacian filters. Both Sobel filters and Laplacian filters are filters for extracting edge features of an object. However, these filters have different properties. The Sobel filter determines whether or not a pixel is an edge point based on the edge strength represented by the first differential value in the x direction (horizontal direction) and the first differential value in the y direction (vertical direction) of the image. On the other hand, the Laplacian filter determines whether or not a pixel is an edge point based on the second differential value of the image. These differences in filtering properties result in different edge features as a result of the filtering process.

Here, examples of the processing results when the Sobel filter and the Laplacian filter are applied to the model image of the same object are shown in Fig. 3A (image GI1) and Fig. 3B (image GI2). The model image used here is image IG11 shown on the left side of Fig. 5. However, in image IG11, the details of the cast surface of the object 90 are omitted. In Fig. 3A (image GI1) and Fig. 3B (image GI2), the edge points extracted by the filter are represented by black dots. As can be seen from Fig. 3A (image GI1) and Fig. 3B (image GI2), the edge features (edge points) extracted by each filter are different. In the processing result by the Laplacian filter in Fig. 3B (image GI2), it can be seen that edge points are output well to the arc-shaped step portion 91 of the object 90. In this case, it can be said that the Laplacian filter is suitable for identifying the position and orientation of the object by matching the arc-shaped step portion 91 as a characteristic part of the object.

In this way, the image features used in the detection process (matching) change depending on the feature extraction process applied. Furthermore, even when the same object is imaged, the way the object is captured changes depending on the object's properties and the imaging conditions (including lighting, the angle at which the object is captured, etc.). Therefore, it is necessary to consider that the output results of filters such as those shown in Figures 3A and 3B also change depending on the object's properties and imaging conditions. As described below, the object detection device 60 according to this embodiment is configured to evaluate the properties of the feature extraction process for a certain object in response to changes in imaging conditions, and to determine the appropriate feature extraction process to apply to that object.

The model image receiving unit 165 receives an image of an object whose position and orientation are known (hereinafter, this image is also referred to as the "first image") as a model image.

The matching area receiving unit 166 provides a function for receiving input specifying an area in the model image to be used for matching with the target object. The matching area may be a part of the model image, or may be the entirety of the model image. For example, the matching area receiving unit 166 receives a user operation for specifying an area in the model image to be used for matching with the target object. For example, the matching area receiving unit 166 may provide a graphical user interface that displays a captured image of the target object on the display unit 41 of the teaching device 40, and receives an operation for specifying a matching area on the captured image using a pointing device or the like.

The model feature storage unit 167 stores, as model features, the image features (hereinafter also referred to as "first image features") that are included in the matching region among the image features extracted from the model image by the feature extraction unit 164.

The model transformation unit 168 provides a function for applying one or more transformation processes to input image data. The transformation processes include one or more of a change in brightness and a projective transformation. Projective transformation includes rotation and scale transformation. The model transformation unit 168 can output a "transformed model image" obtained by applying a transformation process to a model image. The model transformation unit 168 can also output "transformed model features" obtained by applying a transformation process to model features.

The image features obtained by applying feature extraction processing to the transformed model image are referred to as "third image features."

The stability calculation unit 169 calculates a first characteristic related to the characteristics of the feature extraction process based on the third image feature and the transformation model feature extracted by the feature extraction unit 164 from the transformation model image. The third image feature represents an image obtained by applying the feature extraction process to an image (transformed model image) to which transformation (brightness transformation, projective transformation, etc.) by the model transformation unit has been applied to the model image, in other words, an image in which the appearance of the object has changed. Therefore, by comparing the transformation model feature with the third image feature, it is possible to derive information that represents the characteristics of the feature extraction process with respect to changes in the image. The stability calculation unit 169 calculates, as the first characteristic, an index that represents the stability of the feature extraction process with respect to changes in the image. Hereinafter, this index will also be referred to simply as stability.

The sensitivity calculation unit 170 calculates a second characteristic related to the characteristics of the feature extraction process based on the third image feature and the transformation model feature extracted by the feature extraction unit 164 from the transformation model image. The sensitivity calculation unit 170 calculates, as the second characteristic, an index that indicates the sensitivity of the feature extraction process by the feature extraction unit to changes in the image. Hereinafter, this index will also be referred to simply as sensitivity.

The evaluation index calculation unit 171 can calculate an evaluation index for determining a feature extraction process based on one or more stabilities calculated based on the third image feature and the transformation model feature. The evaluation index calculation unit 171 can also calculate an evaluation index for determining a feature extraction process based on one or more stabilities and one or more sensitivities.

The general flow of processing in the object detection device 60 from inputting a model image to calculating an evaluation index for determining the feature extraction processing will be explained with reference to the data flow diagram in Figure 4.

(Step 1) A user inputs a model image 201 obtained by capturing an object whose position and orientation are known.
(Step 2) The user specifies an area on the model image 201 to be used for matching.

(Procedure 3) The object detection device 60 performs the following processes (procedure 3-1) to (procedure 3-4) for each of the feature extraction processes 164A.
(Step 3-1) A feature extraction process is applied to the model image 201.
(Step 3-2) Image features within the matching region in the model image 201 are registered as model features 202 for the selected feature extraction process.
(Step 3-3) The object detection device 60 performs the following processes (a) to (d) for each of the multiple conversion processes 168A.
(a) A transformation process is applied to the model image 201 to obtain a transformed model image 203.
(b) A feature extraction process is applied to the transformed model image 203 to obtain a third image feature 204 .
(c) A transformation process is applied to the model features 202 to obtain transformed model features 205 .
(d) Stability and sensitivity are calculated (symbol 169A) from the image features (third image features 204) extracted from the transformed model image 203 and the transformed model features (transformed model features 205), and stored as the stability and sensitivity of the selected transformation process.
(Step 3-4) Next, an evaluation index for determining a feature extraction process from a plurality of sets of stability and sensitivity corresponding to a plurality of conversion processes is calculated (reference numeral 171A).
(Step 4) The object detection device 60 determines the feature extraction process to be applied to the detection process based on the evaluation index calculated for each feature extraction process in the above step 3. For example, the evaluation index calculation unit 171 may determine the feature extraction process having the highest evaluation index as the feature extraction process to be used for the detection process.

An example of the conversion process by the model conversion unit 168 is shown in FIG. 5. In FIG. 5, a converted image IG12 is generated by applying a projective transformation (including rotation and scale transformation) to a model image IG11 obtained by capturing an image of an object 90.

A specific example of the calculation of stability by the stability calculation unit 169 will be described below. The stability calculation unit 169 calculates stability by the following process. Here, it is assumed that the image features are edge points.
(Step k1) Determine the correspondence between the transformation model feature and the edge points included in each of the third image features extracted from the transformation model image.
(Step k2) Calculate the stability of the model feature based on the proportion of edge points included in the transformed model feature whose difference from the corresponding edge points included in the third image feature is smaller than a specific value.

In the above step k1, for example, the correspondence between the edge points included in the transformation model feature and the third image feature may be determined according to the following rule.
(Rule) Define edge points in the transformation model feature and the third image feature as points consisting of two variables, namely, their position and the direction of the brightness gradient. Then, determine an edge point in the third image feature whose distance from an edge point in the transformation model feature (distance in a space defined by the two variables (position, brightness gradient)) is within a predetermined threshold as the edge point corresponding to the edge point of the transformation model feature.

In the above step k2, the stability calculation unit 169 calculates the stability as follows, for example. Let the total number of edge points appearing in the transformation model feature be _Htotal . Let the total number of edge points appearing in the third image feature, the distance between the edge points and the corresponding points in the transformation model feature being smaller than a specific value, be _G1 . In this case, the stability calculation unit 169 may calculate the stability as follows.
(Stability)=G ₁ /H _total

From the above, stability can be said to be the proportion of similarity between the corresponding image feature of the third image feature and the transformation model feature. In other words, stability is an index that indicates how many edge points are extracted at the target position (the position of the corresponding point in the transformation model feature) in the third image feature. In this way, the stability calculation unit 169 provides an index that indicates the stability of the image feature against image changes (transformation) for the image feature extracted by the feature extraction process to be evaluated.

A specific example of the calculation of sensitivity by the sensitivity calculation unit 170 will be described below. The sensitivity calculation unit 170 calculates the sensitivity by the following process. Here, it is assumed that the image features are edge points.
(Step m1) Determine the correspondence between the transformation model feature and the edge points included in each of the third image features extracted from the transformation model image.
(Step m2) Calculate the sensitivity of the model feature based on the proportion of image features whose difference from the corresponding edge points included in the transformation model feature is smaller than a specific value among edge points included in the third image feature.

The determination of corresponding points in step m1 above is the same as in step k1 above.

In step m2, the sensitivity calculation unit 170 calculates the sensitivity as follows, for example. The total number of edge points appearing in the third image feature is set to _Gtotal . Among them, the total number of edge points whose distance to the corresponding point of the transformation model feature is smaller than a specific value is set to _G1 . In this case, the sensitivity calculation unit may calculate the sensitivity as follows.
(Sensitivity) = G ₁ /G _total

From the above, we can say that sensitivity is an index that indicates how few third image features are similar to the transformed model features. In other words, sensitivity is an index that indicates the degree to which the image features extracted by the feature extraction process being evaluated react sensitively only to the model features, even if there is a change in the image (transformation).

For example, as the number of edge points (G _total ) appearing in the third image feature increases, the likelihood that edge points will appear at the same positions of the edge points of the transformation model feature increases, tending to increase stability, but tending to decrease sensitivity.

FIG. 6A-FIG. 6B are flowcharts showing the process for determining the feature extraction process (feature extraction process determination process) executed in the object detection device 60. This process is executed under the control of the processor.

First, the model image receiving unit 165 receives an input of a model image acquired by the visual sensor 70 capturing an image of an object whose position and orientation are known (step S1). Next, the matching area receiving unit 166 receives an operation by the user to specify a matching area on the model image that corresponds to an area from which image features are to be extracted (step S2).

The series of processes from steps S4 to S16 are repeatedly executed for each feature extraction process (loop 1 of step S3). The model feature storage unit 167 applies the feature extraction process to the model image (step S4). Then, the model feature storage unit 167 saves the image features in the matching area in the model image as model features for the feature extraction process selected in loop 1 (step S5).

The series of processes from steps S7 to S15 are repeated a predetermined number of times while changing the brightness conversion parameters (loop 2 of step S6). Also, the series of processes from steps S8 to S15 are repeated a predetermined number of times while switching the projection transformation parameters (loop 3 of step S7).

In step S8, the model transformation unit 168 applies a process to change the brightness of the model image. Furthermore, the model transformation unit 168 applies a projective transformation to the model image whose brightness has been changed (step S9). The projective transformation includes, for example, rotation and scale transformation of the model image.

Next, the feature extraction unit 164 applies the selected feature extraction process to the model image (transformed model image) to which the brightness change and projective transformation have been applied (step S10).

As shown in FIG. 6B, the model transformation unit 168 then applies projective transformation to the model features resulting from the selected feature extraction process (step S11). Next, the stability calculation unit 169 calculates the above-mentioned stability from the image features (third image features) extracted from the transformed model image and the model features (transformed model features) to which projective transformation has been applied (step S12). The stability calculation unit 169 then stores the calculated stability as one of the stabilities of the selected feature extraction process.

Next, in step S14, the sensitivity calculation unit 170 calculates the above-mentioned sensitivity from the image feature (third image feature) extracted from the transformation model image and the transformation model feature. Then, the sensitivity calculation unit 170 stores the calculated sensitivity as one of the sensitivities for the selected feature extraction process (step S15).

Loop processing by loop 2 and loop 3 is executed for multiple brightness change parameters and projection transformation parameters, and the number of stabilities and sensitivities each is generated equal to the number of brightness change parameters multiplied by the number of projection transformation parameters. When loop processing by loop 2 ends, the process proceeds to step S16. In step S16, the evaluation index calculation unit 171 calculates an evaluation index for the currently selected feature extraction process from the pair of stabilities and the pair of sensitivities generated for that feature extraction process.

For example, suppose that n stabilities ST(i) and sensitivities SE(i) have been calculated for each selected feature extraction process (i ranges from 1 to n). In this case, the evaluation index calculation unit 171 may use the average, weighted harmonic mean, or other statistics for the stabilities ST(1) to ST(n) and sensitivities SE(1) to SE(n) as the evaluation index for the selected feature extraction process.

When the loop processing by loop 1 is completed, the evaluation index calculation unit 171 determines, for example, the feature extraction process with the highest evaluation index among the evaluation indexes calculated for each of the multiple feature extraction processes as the feature extraction process to be used for the detection process (step S17).

In the above-described embodiment, by applying multiple types of transformations (brightness transformation, projective transformation) to calculate multiple stability and sensitivity values and then calculating the evaluation index based on these, it becomes possible to select an appropriate feature extraction process and apply it to detection even in situations where the appearance and quality of the target object change due to various environmental factors.

The feature extraction process determination process is configured to calculate an evaluation index for determining the feature extraction process based on the stability of image features against changes in the image. By applying the feature extraction process determined in this way to the detection process, stable detection becomes possible. As the detection process becomes more stable, it also becomes easier to adjust the detection parameters (detection score, etc.).

The feature extraction process determination process is configured to calculate an evaluation index for determining the feature extraction process based on the sensitivity of image features to changes in the image. By applying the feature extraction process determined in this way to the detection process, it becomes possible to stabilize detection. In this case, it is also possible to shorten the detection process time.

The feature extraction process determination process is configured to be able to calculate an evaluation index for determining the feature extraction process using a single image (model image). This configuration has the advantage of reducing the user's workload required for the feature extraction process determination process.

In the embodiment described above, the evaluation index calculation unit 171 is configured to calculate an evaluation index for determining a feature extraction process based on stability and sensitivity, but the evaluation index calculation unit 171 may calculate an evaluation index based on stability. Even in this case, it is possible to determine a feature extraction process that can stably detect changes in the image. Alternatively, the evaluation index calculation unit 171 may calculate an evaluation index based on sensitivity. Even in this case, it is possible to determine a feature extraction process that can provide stable detection.

In the above embodiment, an example was shown in which, for a certain feature extraction process, when one or more pairs of stability and sensitivity are obtained by one or more conversion processes, their statistical quantity such as a weighted harmonic mean is used as the evaluation index for that feature extraction process. As another operation example, the evaluation index calculation unit 171 may calculate an evaluation index for each of one or more conversion processes (i.e., for each pair of one or more third image features and one or more conversion model features), and determine the minimum value of the one or more calculated evaluation indexes as the evaluation index for the feature extraction process. By calculating an evaluation index in this way for each feature extraction process and determining the feature extraction process to be used for detection, it is possible to determine a feature extraction process that can perform stable detection even in cases where the object is poorly reflected.

The display control unit 172 may operate to display the evaluation index for each feature extraction process calculated by the evaluation index calculation unit 171. FIG. 7 shows an example in which a list 250 showing the evaluation indexes of each feature extraction process (filter) is displayed on the display unit 41 by the display control unit 172. By displaying the evaluation indexes in this way, the user can recognize which feature extraction process is suitable. By referring to this list 250, the user can also select the feature extraction process to apply to detection. The list 250 may also function as a user interface that accepts a user operation to select a feature extraction process to be used for detection from multiple feature extraction processes.

In the above embodiment, a filter that performs edge detection has been given as an example of feature extraction processing, but the above-mentioned content for determining a feature extraction processing can be applied to various types of feature extraction processing. For example, take a corner detector that detects corners within an image. Examples of corner detectors include the Harris corner detector, the Shi-Tomasi corner detector, and the FAST (Features from Accelerated Segment Test) corner detector.

The Harris corner detector and Shi-Tomasi corner detector detect points that have a large amount of change (E in the following equation (1)) when the image position is shifted as corners. In the following equation (1), I(x,y) is the brightness of the pixel in the image, I(x+u,y+v) is the brightness of the pixel in the shifted image, and w(x,y) is the window function.

The Harris corner detector and the Shi-Tomasi corner detector determine whether a corner exists from the eigenvalues of M in the following equations (2) and (3), which are simplified versions of the above equations, but the method of determination is different.

Figure 8 shows an example of the results of processing an image of an object using the Shi-Tomasi corner detector. Image IG20 shown in Figure 8 shows the results of applying the Shi-Tomasi corner detector to an image containing an object 190 (part of the object is marked with the reference symbol 190). In image IG20, the detected corners CP are represented by black dots (only a portion is marked with the reference symbol CP).

The FAST corner detector classifies whether a candidate point is actually a corner by the difference in brightness values of a 16-pixel circle. If the difference is consecutively greater or smaller than a threshold, it is considered a corner. The FAST corner detector is said to be able to operate at high speed.

Due to the differences in the detection methods used by each corner detector, it is believed that, as with the edge detector, the extraction of image features by the corner detector will also be affected by changes in how the object appears. Therefore, for corner detectors as well, it is possible to calculate the stability and sensitivity described above, obtain an evaluation index, and determine a corner detector that will provide stable detection. Therefore, the various processes for determining the feature extraction process in the above-mentioned embodiments can also be applied to corner detectors.

As described above, according to this embodiment, it is possible to determine a feature extraction process that can perform stable detection even in situations where the imaging conditions change.

In the above embodiment, a configuration example is shown in which the object detection device is realized as a function of the robot control device, but the object detection device can also be realized as an independent device separate from the robot control device. For example, the object detection device can be configured as an information processing device such as a personal computer connected to the robot control device.

The functional blocks of the robot control device shown in Figure 2 may be realized by the processor of the robot control device executing various software stored in a storage device, or may be realized by a hardware-based configuration such as an ASIC (Application Specific Integrated Circuit).

The programs that execute various processes such as the feature extraction process and determination process in the above-mentioned embodiments can be recorded on various computer-readable recording media (for example, semiconductor memories such as ROM, EEPROM, and flash memory, magnetic recording media, and optical disks such as CD-ROM and DVD-ROM).

Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible to these embodiments without departing from the gist of the present disclosure, or without departing from the spirit of the present disclosure derived from the contents described in the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-mentioned embodiments, the order of each operation and the order of each process are shown as examples, and are not limited to these. The same applies when numerical values or formulas are used to explain the above-mentioned embodiments.

The following additional notes are provided regarding the above embodiment and modifications.
(Appendix 1)
A feature extraction unit (164) for extracting image features from an image;
a model image receiving unit (165) that receives, as a model image, a first image obtained by capturing an object whose position and orientation are known;
a model feature storage unit (167) that stores the first image feature extracted from the model image by the feature extraction unit as a model feature;
a detection unit (163) that identifies a position and orientation of the object by comparing a second image feature extracted by the feature extraction unit from a second image capturing the object, the position and orientation of which are unknown, with the model feature,
The object detection device (60) further comprises:
a model transformation unit (168) that applies one or more transformation processes to the model image and the model feature to generate one or more transformed model images and one or more transformed model features, respectively;
a first characteristic calculation unit (169) that calculates a first characteristic related to a characteristic of the feature extraction process by the feature extraction unit based on a third image characteristic extracted from the transformation model image by the feature extraction unit and the transformation model characteristic;
an evaluation index calculation unit (171) that calculates an evaluation index of the feature extraction process based on one or more first characteristics calculated by the first characteristic calculation unit based on the one or more third image features and the one or more transformation model features,
The feature extraction unit (164) has a plurality of feature extraction processes,
the first characteristic calculation unit (169) calculates the one or more first characteristics for each of the plurality of feature extraction processes based on the third image feature and the transformation model feature;
The object detection device (60), wherein the evaluation index calculation unit (171) calculates the evaluation index for determining which feature extraction process to apply to the detection process of the detection unit for each of the multiple feature extraction processes.
(Appendix 2)
A matching area receiving unit (166) that receives a part or the whole of the object shown in the model image as a matching area,
The object detection device (60) described in Appendix 1, wherein the model feature storage unit (167) stores, as the model feature, the image feature included in the matching area among the first image features.
(Appendix 3)
The object detection device (60) described in Appendix 1 or 2, wherein the evaluation index calculation unit (171) determines the feature extraction process to be used by the feature extraction unit in the detection process by the detection unit (163) based on the evaluation index calculated for each of the multiple feature extraction processes.
(Appendix 4)
The object detection device (60) described in Appendix 3, wherein the evaluation index calculation unit (171) determines the feature extraction process having the highest evaluation index among the evaluation indexes calculated for each of the multiple feature extraction processes as the feature extraction process to be used by the feature extraction unit (164) in the detection process by the detection unit (163).
(Appendix 5)
The object detection device (60) according to any one of appendixes 1 to 4, wherein the one or more conversion processes by the model conversion unit (168) include a process of converting brightness of the model image.
(Appendix 6)
The object detection device (60) according to any one of appendices 1 to 5, wherein the one or more transformation processes by the model transformation unit (168) include performing a projective transformation on each of the model image and the model feature.
(Appendix 7)
An object detection device (60) according to any one of appendixes 1 to 6, wherein the first characteristic calculated by the first characteristic calculation unit (169) represents stability of the feature extraction process against changes in an image.
(Appendix 8)
The first characteristic calculation unit (169)
determining a correspondence relationship between the transformation model feature and the image feature included in each of the third image features;
An object detection device (60) as described in Appendix 7, which calculates the first characteristic based on the proportion of image features included in the transformation model features whose difference from corresponding image features included in the third image features is smaller than a specific value.
(Appendix 9)
a second characteristic calculation unit (170) that calculates a second characteristic representing sensitivity of the feature extraction process by the feature extraction unit (164) to a change in an image based on the third image characteristic and the transformation model characteristic,
The object detection device (60) described in any one of Appendices 1 to 8, wherein the evaluation index calculation unit (171) calculates the evaluation index for each of the multiple feature extraction processes based on the one or more first characteristics and one or more second characteristics calculated by the second characteristic calculation unit (170) based on the one or more third image features and the one or more transformation model features.
(Appendix 10)
The second characteristic calculation unit (170)
determining a correspondence relationship between the transformation model feature and the image feature included in each of the third image features;
An object detection device (60) as described in Appendix 9, which calculates the second characteristic based on the proportion of image features among the image features included in the third image features whose difference from the corresponding image features included in the transformation model features is smaller than a specific value.
(Appendix 11)
The object detection device (60) described in Appendix 9 or 10, wherein the evaluation index calculation unit (171) calculates a weighted harmonic mean of the one or more first characteristics and the one or more second characteristics as the evaluation index for each of the multiple feature extraction processes.
(Appendix 12)
The evaluation index calculation unit (171) calculates, for each of the plurality of feature extraction processes,
calculating an index for determining the feature extraction process for each pair of the one or more third image features and the one or more transformation model features;
An object detection device (60) according to any one of appendixes 1 to 11, wherein the minimum value of the indices calculated for each of the sets is used as an evaluation index for the feature extraction process.
(Appendix 13)
The object detection device (60) according to any one of appendix 1 to 12, further comprising a display control unit (172) that displays the evaluation index calculated for each of the multiple feature extraction processes on a display device.

10 Robot 33 Hand 40 Teaching device 41 Display unit 50 Robot control device 60 Object detection device 70 Visual sensor 100 Robot system 151 Operation control unit 161 Visual sensor control unit 162 Image acquisition unit 163 Detection unit 164 Feature extraction unit 165 Model image reception unit 166 Matching area reception unit 167 Model feature storage unit 168 Model conversion unit 169 Stability calculation unit 170 Sensitivity calculation unit 171 Evaluation index calculation unit 172 Display control unit 201 Model image 202 Model feature 203 Transformed model image 204 Third image feature 205 Transformed model feature

Claims

a feature extraction unit that extracts image features from an image;
a model image receiving unit that receives, as a model image, a first image obtained by capturing an object whose position and orientation are known;
a model feature storage unit that stores the first image feature extracted from the model image by the feature extraction unit as a model feature;
a detection unit that identifies a position and orientation of the object by comparing a second image feature extracted by the feature extraction unit from a second image capturing the object, the position and orientation of which are unknown, with the model feature,
The object detection device further comprises:
a model transformation unit that applies one or more transformation processes to the model image and the model feature to generate one or more transformed model images and one or more transformed model features, respectively;
a first characteristic calculation unit that calculates a first characteristic related to a characteristic of a feature extraction process performed by the feature extraction unit, based on a third image feature extracted from the transformation model image by the feature extraction unit and the transformation model feature;
an evaluation index calculation unit that calculates an evaluation index of the feature extraction process based on one or more first characteristics calculated by the first characteristic calculation unit based on the one or more third image features and the one or more transformation model features,
The feature extraction unit has a plurality of feature extraction processes,
the first characteristic calculation unit calculates the one or more first characteristics based on the third image feature and the transformation model feature for each of the plurality of feature extraction processes;
The object detection device, wherein the evaluation index calculation unit calculates the evaluation index for each of the plurality of feature extraction processes to determine which feature extraction process to apply to the detection process of the detection unit.
a matching area receiving unit that receives a part or the whole of the object shown in the model image as a matching area;
The object detection device according to claim 1 , wherein the model feature storage unit stores, as the model feature, an image feature included in the matching region among the first image features.
The object detection device according to claim 1 or 2, wherein the evaluation index calculation unit determines the feature extraction process to be used by the feature extraction unit in the detection process by the detection unit based on the evaluation index calculated for each of the plurality of feature extraction processes.
The object detection device according to claim 3, wherein the evaluation index calculation unit determines the feature extraction process having the highest evaluation index among the evaluation indexes calculated for each of the plurality of feature extraction processes as the feature extraction process to be used by the feature extraction unit in the detection process by the detection unit.
The object detection device according to any one of claims 1 to 4, wherein the one or more conversion processes performed by the model conversion unit include a process of converting the brightness of the model image.
The object detection device according to any one of claims 1 to 5, wherein the one or more transformation processes performed by the model transformation unit include performing a projective transformation on each of the model image and the model feature.
The object detection device according to any one of claims 1 to 6, wherein the first characteristic calculated by the first characteristic calculation unit represents the stability of the feature extraction process against changes in the image.
The first characteristic calculation unit is
determining a correspondence relationship between the transformation model feature and the image feature included in each of the third image features;
The object detection device according to claim 7 , wherein the first characteristic is calculated based on a proportion of image features included in the transformation model features whose difference from corresponding image features included in the third image features is smaller than a specific value.
a second characteristic calculation unit that calculates a second characteristic representing sensitivity of the feature extraction process by the feature extraction unit to a change in an image based on the third image feature and the transformation model feature,
9. The object detection device according to claim 1, wherein the evaluation index calculation unit calculates the evaluation index for each of the plurality of feature extraction processes based on the one or more first characteristics and one or more second characteristics calculated by the second characteristic calculation unit based on the one or more third image features and the one or more transformation model features.
The second characteristic calculation unit,
determining a correspondence relationship between the transformation model feature and the image feature included in each of the third image features;
The object detection device according to claim 9 , wherein the second characteristic is calculated based on a proportion of image features among the image features included in the third image features whose difference from the corresponding image features included in the transformation model features is smaller than a specific value.
The object detection device according to claim 9 or 10, wherein the evaluation index calculation unit calculates a weighted harmonic average of the one or more first characteristics and the one or more second characteristics as the evaluation index for each of the plurality of feature extraction processes.
The evaluation index calculation unit, for each of the plurality of feature extraction processes,
calculating an index for determining the feature extraction process for each pair of the one or more third image features and the one or more transformation model features;
The object detection device according to claim 1 , wherein a minimum value of the indices calculated for the respective sets is set as an evaluation index for the feature extraction process.
The object detection device according to any one of claims 1 to 12, further comprising a display control unit that displays the evaluation index calculated for each of the plurality of feature extraction processes on a display device.