JP2024111594A - Working Equipment - Google Patents

Abstract

To provide a work device which can improve a success rate of work for a workpiece.SOLUTION: A pickup device 1 includes: an imaging device 9; a pickup section 10 which picks up a workpiece 20; a positioning mechanism PM which positions the imaging device 9 and the pickup section 10 relatively to the workpiece 20; and a control device 100 which controls the imaging device 9, the pickup section 10 and the positioning mechanism PM. The control device 100 acquires a prediction value S indicating easiness of work for the workpiece 20 and a prediction attitude U of the pickup section 10 in which the prediction value S is predicted to be a determination value T or more on the basis of image data. When all prediction values S are less than the determination value T, the control device 100 moves the pickup section 10 to the prediction attitude U corresponding to any one workpiece 20, and repeats acquisition of image data by causing the imaging device 9 to capture a workpiece again and acquisition of the prediction value S and the prediction attitude U in the attitude.SELECTED DRAWING: Figure 1

Description

The present invention relates to a work device, and, for example, to a technology that can detect the optimal approach direction to a workpiece when grasping a pile of workpieces.

Conventionally, a picking device has been proposed that includes a three-dimensional imaging device that takes three-dimensional images of a group of workpieces, a robot arm that can grasp the workpieces, and a control device that controls the operation of the robot arm (Patent Document 1).

JP 2015-213973 A

The device in Patent Document 1 uses a three-dimensional imaging device to determine the approach direction to the workpiece. However, if the workpieces are randomly stacked in a container, when an image of a desired workpiece is taken with an imaging device fixed above the workpiece, it is blocked by other works nearby. This causes a problem that the image necessary for posture detection is missing, reducing the success rate of picking up the workpiece.

In addition, when taking an image of the workpiece using an imaging device fixed above the workpiece, the imaging device must be positioned so as not to interfere with the gripping device. This results in a long shooting distance, which can cause the problem of low resolution in the captured image. When the image resolution is low, it is difficult to find feature points in the image and to recognize the workpiece itself. This leads to a lower success rate in picking up the workpiece.

The object of the present invention is to provide a work device that can improve the success rate of work on a workpiece.

The working apparatus 1 of the present invention includes an imaging device 9 that captures an image of a workpiece 20 and outputs image data;
A working unit 10 that performs a specified operation on the workpiece 20;
a positioning mechanism PM for positioning the imaging device 9, the working unit 10, and the workpiece 20 relative to each other;
a control device 100 that controls the imaging device 9, the working unit 10, and the positioning mechanism PM,
The control device 100 includes:
an approach direction etc. determining means Ap for obtaining a predicted value indicating the ease of work on the workpiece 20 and a predicted attitude of the working unit 10 when the predicted value is predicted to be equal to or greater than the judgment value based on image data output from the imaging device 9, determining the workpiece 20 to be worked on based on the obtained predicted value, and determining the imaging direction of the imaging device 9 when the predicted value is equal to or greater than the judgment value as the approach direction to the workpiece 20 determined to be the work object;
a control unit (114) that controls the working unit (10) so as to perform work on the workpiece (20) determined as a work target from the determined approach direction,
If the predicted value is less than the judgment value, the control device 100 moves the working unit 10 to the predicted posture, and again repeats the process of obtaining the predicted value and predicted posture in the posture to which the working unit 10 is moved for the image data captured and output by the imaging device 9.
The predetermined operations are variously determined depending on the use of the workpiece, and examples of the predetermined operations include picking up the workpiece, assembling, welding, applying grease, and inspecting the appearance.
The judgment value is a value that is arbitrarily determined by design or the like, and is determined by obtaining an appropriate value through, for example, either one or both of a test and a simulation.

According to this configuration, the approach direction etc. determining means Ap of the control device 100 obtains, based on the image data, a predicted value indicating the ease of working on the workpiece 20 and a predicted posture of the working unit 10 when this predicted value is predicted to be equal to or greater than a judgment value. The approach direction etc. determining means Ap determines the workpiece 20 to be worked on based on the predicted value. For example, the workpiece 20 with the largest obtained predicted value is selected as the workpiece. If the predicted value is equal to or greater than the judgment value, the imaging direction at the time of image data acquisition is determined as the approach direction to the workpiece 20. The control unit 114 controls the working unit 10 to work on the workpiece 20 from the determined approach direction.

If all the predicted values are less than the judgment value, the control device 100 moves the working unit to a predicted posture corresponding to one of the workpieces 20, again captures an image of the workpiece 20 using the imaging device 9 to obtain image data, and repeats the process of obtaining the predicted value and predicted posture at that posture.
In this way, the search time for an approach direction with a high possibility of performing a specified task can be shortened compared to a case where the working unit 10 is moved randomly to search for a posture where the predicted value is equal to or greater than the judgment value. Since the work is performed on the workpiece 20 from the approach direction with a high possibility of performing the task, the success rate of the task on the workpiece can be improved compared to the above-mentioned conventional technology.

The approach direction etc. determining means Ap may obtain the predicted value and the predicted attitude by applying a prediction model generated by machine learning to the image data.
The prediction model includes a machine learning result that has been trained to detect at least one target work position from image data outputted by capturing an image of the workpiece 20. The machine learning result is a learning result that uses, as teacher data, a correct answer value indicating the ease of a work motion calculated based on the image data of the learning target and a work position that is specified in advance on the image data of the learning target.
In this case, by applying the prediction model, it is possible to finely adjust the target posture of the working unit 10 as necessary, and thus the success rate of the work on the workpiece 20 can be further improved.

The positioning mechanism PM has a stage 33 to which the imaging device 9 and the working unit 10 are fixed, and the imaging device 9 and the working unit 10 may be positioned so that the optical axis of the imaging device 9 is parallel to the approach direction of the working unit 10. In this case, after the workpiece 20 is imaged by the imaging device 9, there is no need to correct any misalignment with the optical axis direction of the imaging device 9 when moving the working unit 10, thereby shortening the working time of the workpiece 20 in the working unit 10 and improving the success rate of the work.

The positioning mechanism PM may include a base 32, a stage 33 to which the imaging device 9 and the working unit 10 are fixed, a first positioning mechanism 8 that positions the stage 33 with two degrees of freedom of rotation relative to the base 32, and a second positioning mechanism 47 that positions the base 32. In this case, the imaging device 9 can be fixed to the stage 33 at the tip side of the first positioning mechanism 8 having two degrees of freedom of rotation, so that the imaging device 9 can be positioned near the workpiece 20. In this case, image data with higher resolution than that of the conventional technology in which the imaging device is fixed above the workpiece can be obtained. In addition, the ease of work on the workpiece 20 is determined using image data obtained by the imaging device 9 configured to be able to change the shooting direction, and the work is performed by approaching from the same direction as the shooting direction. This improves the success rate of work on the workpiece 20 in the working device 1.

After the predicted value becomes equal to or greater than the judgment value, the control device 10 may repeatedly change the posture of the working unit 10 to the predicted posture a number of times, and among the multiple image data captured by the imaging device 9 in each posture and output, determine the shooting direction in the posture in which the predicted value is maximized as the approach direction to the workpiece 20. By determining the shooting direction in the posture in which the predicted value is maximized as the approach direction to the workpiece 20 among the multiple image data output in this manner, the success rate of the work on the workpiece 20 is further improved.

The imaging device 9 and the stage 33 may be positioned so that the optical axis of the imaging device 9 coincides with the central axis of the stage 33. The working unit 10 can approach the workpiece 20 from the direction in which the workpiece 20 is photographed without changing the posture of the working unit 10. This improves the success rate of working on the workpiece 20 and shortens the time required to change posture, compared to a configuration in which the optical axis of the imaging device 9 and the central axis of the stage 33 do not coincide.

The first positioning mechanism may be an angle adjustment device 8 including the base 32 and the stage 33, two or more sets of link mechanisms 34 that connect the stage 33 to the base 32 in a manner that allows the position to be changed, and an actuator 31 that drives the link mechanisms 34. In this case, the mass of the moving parts is smaller and high-speed positioning is possible compared to a two-degree-of-freedom mechanism in which two rotational positioning mechanisms are arranged in series. Therefore, the search time for the optimal approach direction can be shortened.

In the working device of the present invention, if all predicted values are less than the judgment value, the control device moves the working unit to a predicted posture corresponding to one of the workpieces, captures an image of the workpiece again with the imaging device to obtain image data, and repeats the process of obtaining the predicted value and predicted posture at that posture. Therefore, the success rate of work on the workpiece can be improved compared to the conventional technology described above.

1 is a diagram showing a configuration of a working device according to a first embodiment of the present invention; 13 is a front view of a simplified model of the working device in which two link mechanisms of the angle adjustment device are omitted. FIG. FIG. 13 is a diagram showing one link mechanism of the angle adjustment device as a straight line. FIG. 2 is a diagram showing an imaging device and a pickup unit fixed to a stage of the angle adjustment device. FIG. 4 is a diagram showing a state in which an imaging device and a pickup unit are mounted on the angle adjustment device shown in FIG. 3 . FIG. 2 is a block diagram of a control system for the working device. FIG. 13 is a diagram for explaining how image data used for learning is collected. FIG. 11 is a diagram for explaining post-shooting processing during learning. FIG. 13 is a diagram showing an example of a state in which a correct answer value indicating the ease of a pick-up operation is maximized. 4 is a flowchart for explaining a series of operations of the operating device. 11B is a flowchart showing the pickup process of FIG. 11A; FIG. 2 is a typical configuration diagram of the control device. FIG. 13 is a front view of a pickup unit according to a modified example. 13 is a flowchart showing a learning procedure. 10 is a flowchart illustrating another series of operations in the operating device. FIG. 11 is a diagram showing a configuration of a working device according to a second embodiment of the present invention. FIG. 13 is a diagram showing a configuration of a working device according to a third embodiment of the present invention. FIG. 13 is a diagram showing a configuration of a working device according to a fourth embodiment of the present invention. FIG. 13 is a diagram showing a configuration of a working device according to a fifth embodiment of the present invention.

[First embodiment]
A working device according to an embodiment of the present invention will be described with reference to FIGS.
<Overall configuration of working device>
1 , a pickup device 1, which is an example of a working device, picks up a workpiece 20 that is stacked in an arbitrary orientation on a workpiece setting table 14 or inside a container 13. The pickup device 1 includes an imaging device 9, a pickup unit 10 as a working unit, a positioning mechanism PM, and a control device 100.

The imaging device 9 captures an image of at least one workpiece 20 and outputs image data. The pick-up unit 10 approaches the workpiece 20 from the approach direction and picks up the workpiece 20 as a specified task. The multiple workpieces 20 to be picked up are often stacked in a messy manner inside the workpiece container 13. The positioning mechanism PM positions the imaging device 9 and the pick-up unit 10 relative to the workpieces 20. The control device 100 controls the imaging device 9, the pick-up unit 10, and the positioning mechanism PM.

<Positioning mechanism>
The positioning mechanism PM has an angle adjustment device 8, which is a first positioning mechanism, and a second positioning mechanism 47. The angle adjustment device (also called a "link actuator") 8 has a parallel link mechanism 30 shown in FIG. 2 that supports the image pickup device 9 and the pickup unit 10 so that the posture can be changed, and an actuator 31 for posture control that drives the parallel link mechanism 30. The second positioning mechanism 47 shown in FIG. 1 adjusts the relative position of the angle adjustment device 8 with respect to the workpiece 20. The second positioning mechanism 47 has a linear motion unit 4 and a rotation mechanism 7. A structure 2 is fixed to a floor or the like, which is the ground, and a linear motion unit 4 is installed on the structure 2. The rotation mechanism 7 is attached to the output section 6 of the linear motion unit 4.

<Linear motion unit>
An XYZ stage that moves in three orthogonal axial directions is applied to the linear motion unit 4. The linear motion unit 4 includes first, second and third electric actuators 4X, 4Y and 4Z that respectively correspond to the mutually orthogonal X-axis, Y-axis and Z-axis. The first electric actuator 4X has a guide portion 4Xa that extends in the X-axis direction, which is the left-right direction in FIG. 1, on the upper part of the structure 2, an X-axis table 4Xb that slides along the guide portion 4Xa, a motor 5X that is a drive source, and a conversion mechanism such as a ball screw (not shown) that converts the rotation of the motor 5X into a linear reciprocating motion.

The second electric actuator 4Y is supported by the X-axis table 4Xb. The second electric actuator 4Y has a guide portion 4Ya extending along the Y-axis direction, which is the depth direction in FIG. 1, a Y-axis table 4Yb sliding along the guide portion 4Ya, a motor 5Y as a drive source, and a conversion mechanism such as a ball screw (not shown) that converts the rotation of the motor 5Y into a linear reciprocating motion.

The third electric actuator 4Z advances and retreats in the Z-axis direction (up and down in this example) perpendicular to the X-axis and Y-axis directions. The third electric actuator 4Z is supported by the Y-axis table 4Yb. The third electric actuator 4Z has a guide portion 4Za extending along the Z-axis direction, a Z-axis table 4Zb sliding along the guide portion 4Za, a motor 5Z as a drive source, and a conversion mechanism such as a ball screw (not shown) that converts the rotation of the motor 5Z into a linear reciprocating motion. The X, Y, and Z-axis tables 4Xb, 4Yb, and 4Zb can move in the X, Y, and Z-axis directions by operating the corresponding conversion mechanisms with each motor 5X, 5Y, and 5Z. A rotation mechanism 7 is attached to the Z-axis table 4Zb, which serves as the output unit 6.

<Rotation mechanism>
A rotation mechanism 7 is supported on the Z-axis table 4Zb via a bracket or the like. The rotation mechanism 7 is a rotation actuator that rotates around a rotation axis in the Z-axis direction, and a base 32 (FIG. 2) of an angle adjustment device 8 (described later) is attached to the output part of the rotation actuator. Thus, a second positioning mechanism 47 having the linear motion unit 4 and the rotation mechanism 7 positions the base 32 (FIG. 2). In the pickup device 1 of this example, the rotation axis of the rotation actuator and the central axis QA (FIG. 2) of the base 32 are arranged to be coaxial.

<Angle adjustment device>
The angle adjustment device 8 can be rotated around a rotation axis in the Z-axis direction by the rotation mechanism 7. As shown in Fig. 2, the angle adjustment device 8 includes a parallel link mechanism 30 and an actuator 31. The parallel link mechanism 30 includes a base 32, a stage 33 to which the imaging device 9 and the pickup unit 10 in Fig. 1 are fixed, and three sets of link mechanisms 34 that connect the stage 33 to the base 32 in Fig. 3. The actuator 31 is a drive source for attitude control that drives the link mechanisms 34. Fig. 2 shows one representative set of the three sets of link mechanisms 34 shown in Fig. 3.

The positioning mechanism PM shown in Fig. 1 is configured to be able to change the position of the base 32 in Fig. 2. The angle adjustment device 8 positions the stage 33 with two rotational degrees of freedom relative to the base 32. In the angle adjustment device 8, the actuator 31 can change the angle α of the end link member 35 on the base end side relative to the base 32.
As shown in Fig. 3, the parallel link mechanism 30 connects a stage 33 to a base 32 via three sets of link mechanisms 34 in a manner that allows the position of the stage 33 to be changed. The imaging device 9 and the pickup unit 10 shown in Fig. 1 are attached to the stage 33. In this embodiment, the parallel link mechanism 30 having the three sets of links 34 shown in Fig. 3 has been shown, but the number of link mechanisms 34 may be two, four or more.

As shown in Fig. 2, each link mechanism 34 has a base end link member 35, a tip end link member 36, and a central link member 37. The link mechanism 34 is a four-joint link mechanism consisting of four revolute pairs. As shown in Fig. 3, the base end and tip end link members 35, 36 are L-shaped.
2, one end of the end link member 35 on the base end side is rotatably connected to the base 32. One end of the end link member 36 on the tip end side is rotatably connected to the stage 33. The other ends of the end link members 35, 36 are rotatably connected to both ends of the central link member 37.

The parallel link mechanism 30 has a structure that combines two spherical link mechanisms. The central axes of the rotation pairs between the end link members 35, 36 on the base end side and the end link member 37 on the tip end side may have a certain cross angle γ or may be parallel.

Figure 4 shows a set of link mechanisms 34 represented by straight lines. The three sets of link mechanisms 34 can be shown by models of geometrically identical shapes. The geometrically identical shapes refer to a geometric model in which each link member 35, 36, 37 is represented by straight lines, i.e., a model represented by each rotation pair and straight lines connecting these rotation pairs, in which the base end portion and the tip end portion are symmetrical with respect to the center of the central link member 37 regardless of the posture. The parallel link mechanism 30 of this embodiment is a rotationally symmetric type, and the positional relationship between the base end side base 32 and the base end side end link member 35 and the tip end side stage 33 and the tip end side end link member 36 is rotationally symmetrical with respect to the center line C of the central link member 37. The center of each central link member 37 is located on a common orbital circle D.

The base 32, stage 33, and three link mechanisms 34 constitute a two-degree-of-freedom rotation mechanism. The rotation angle φ of the stage 33 relative to the base 32 is the angle between a reference line passing through the intersection of the central axis QA of the base 32 and a line projecting the central axis QB of the stage 33 on a plane perpendicular to the central axis QA of the base 32. The perpendicular angle at which the central axis QB of the stage 33 is inclined relative to the central axis QA of the base 32 is called the bend angle θ. This two-degree-of-freedom rotation mechanism is compact, yet allows the stage 33 to have a wide range of motion relative to the base 32.

For example, if the central axes QA and QB of the base 32 and stage 33 are defined as straight lines passing through the spherical link centers PA and PB on the base end side and the tip side and intersecting at right angles with the central axes of the base 32 and stage 33 and the respective rotation pairs of the end link members 35 and 36 on the base end side and tip side, the maximum bend angle θ _max between the central axis QA of the base 32 on the base end side and the central axis QB of the stage 33 on the tip end side can be set to approximately 90°. Note that the maximum bend angle θ _max may be 90° or more. The rotation angle φ of the stage 33 relative to the base 32 can be set in the range of 0° to 360°.

The position of the stage 33 relative to the base 32 is changed around the intersection O between the central axis QA of the base 32 on the base end side and the central axis QB of the stage 33 on the tip end side, which serves as the center of rotation. Even if the position of the stage 33 relative to the base 32 changes, the distance L between the spherical link centers PA and PB on the base end side and tip end side does not change.

The bending angle θ can be adjusted only by operating the link mechanism 34, and does not require the operation of multiple joints as in a multi-joint robot. For this reason, the parallel link mechanism 30 can operate more quickly than a multi-joint robot. Therefore, if the parallel link mechanism 30 in FIG. 3 is used to collect image data required for machine learning, which will be described later, a large amount of image data can be collected in a short time compared to a multi-joint robot.

The actuator 31 for posture control shown in FIG. 2 is a rotary actuator equipped with a motor, a reduction mechanism, and a rotation angle sensor. The actuator 31 is installed on the surface of the base end member 40 of the base 32, coaxially with the rotation shaft 42 of the end link member 35 on the base end side. The output shaft of the reduction mechanism is connected to the rotation shaft 42. The actuator 31 is fixed to the base end member 40. As shown in FIG. 3, if three actuators 31 for changing the angles α1 to α3 of the end link member 35 on the base end side relative to the base 32 are provided in three link mechanisms 34, respectively, the positioning rigidity of the angle adjustment device 8 is improved. If the actuators 31 for posture control are provided in at least two of the three link mechanisms 34, the posture of the stage 33 relative to the base 32 can be determined.

<Imaging device and pickup unit>
As shown in Fig. 5, the imaging device 9 is disposed so that its optical axis 9a indicating the imaging direction coincides with the central axis QB of the stage 33. Suction pads 10A and 10B are disposed on both sides of the imaging device 9 on the stage 33. The suction pads 10A and 10B constitute the pickup unit 10 that picks up the workpiece 20 in Fig. 1. The imaging device 9 and the pickup unit 10 are disposed so that the optical axis 9a of the imaging device 9 in Fig. 5 is parallel to the approach direction A1 of the pickup unit 10.

In this embodiment, a vacuum suction type pickup unit 10 is used as the pickup unit 10. The pickup unit 10 has two suction pads 10A and 10B. The suction pads 10A and 10B are adjacent to the imaging device 9 and are arranged so that the two suction pads 10A and 10B are axially symmetrical with respect to the optical axis 9a. The pickup unit 10 is attached to a position where the suction pads 10A and 10B are not within the observation field of the imaging device 9. As shown in FIG. 6, a distance sensor 11 is arranged on the stage 33 to detect the distance to the workpiece 20 (FIG. 1) determined as the pickup target (work target). The distance sensor 11 is installed on the stage 33 close to the imaging device 9 and the pickup unit 10.

<Control device>
7, the control device 100 includes an image data acquisition unit 111, an approach direction etc. determination means Ap, and a control unit 114. The image data acquisition unit 111 acquires image data from the imaging device 9. The approach direction etc. determination means Ap acquires a predicted value S indicating the ease of work on the work and a predicted attitude U of the pickup unit 10 when the predicted value S is predicted to be equal to or greater than a judgment value T based on the image data, and determines the work 20 to be the work target based on the acquired predicted value S, and determines the imaging direction of the imaging device 9 when the predicted value S is equal to or greater than the judgment value T as the approach direction to the work 20 (FIG. 1) determined to be the work target.

The approach direction etc. determining means Ap includes a memory unit 115, a detection unit 112, and a learning unit 113. The memory unit 115 stores the acquired image data, as well as teacher data and machine learning results described below. The detection unit 112 processes the acquired image data. The detection unit 112 acquires the predicted value S and the predicted attitude U by applying a prediction model M generated by machine learning to the image data.

Specifically, the detection unit 112 processes the image data acquired by the machine learning results read from the memory unit 115, and outputs the target workpiece, its pick-up position, a predicted value S indicating the ease of the pick-up operation, and a predicted posture U for which the predicted value S is likely to be equal to or greater than the judgment value T. When there are multiple workpieces, the detection unit 112 detects workpieces that can be picked up from the image data, then obtains the predicted value S and predicted posture U for each of the detected workpieces, and outputs the grip position, predicted value S, and predicted posture U of the workpiece with the largest predicted value S.

The learning unit 113 performs machine learning using the image data stored in the memory unit 115, a pick-up position which is a target position specified in advance for each image, and a correct answer value SG which indicates the ease of the pick-up operation as teacher data.
The control unit 114 controls the linear motion unit 4 , the rotation mechanism 7 , the angle adjustment device 8 , and the pickup unit 10 .
The control device 100 acquires image data from the imaging device 9 using the image data acquisition unit 111, processes the acquired image data using the detection unit 112, and outputs a control signal that controls the operation of the positioning mechanism PM and the pickup unit 10 via the control unit 114 based on the processing results of the detection unit 112.

<About predicted value S and correct answer value SG>
In this embodiment, the numerical value indicating the ease of the pick-up operation output by the detection unit 112 that has read the learning result for the input image data of the work is called the predicted value S. The correct answer of the predicted value S given as teacher data during learning is called the correct value SG. Both the predicted value S and the correct value SG take values in the range of "0 to 100", for example, and the easier the pick-up operation is, the larger the value takes.

The predicted attitude U is the attitude of the pickup unit 10 that is predicted to result in a predicted value S of the target workpiece being equal to or greater than the judgment value T, and may be given, for example, as a combination of command values for each axis of the positioning mechanism PM. The correct attitude UG is the attitude given as teacher data for the command values to the positioning mechanism PM that moves the pickup unit 10 in the approach direction most suitable for picking up the workpiece, based on the image data of the workpiece input during learning. In this embodiment, a suction-type pickup unit 10 is used, so the normal direction A2 of the surface of the workpiece 20 in Figure 8 is set as the optimal approach direction.

<About workpiece detection>
The detection unit 112 in Fig. 7 detects the pick-up position by applying, for example, a neural network. The detection unit 112 performs numerical calculations of the neural network using the learning results stored in the storage unit 115. Image data from the imaging device 9 is input to the neural network. The neural network outputs the results of numerical calculations to the output layer after passing the input image data through a filter process called an intermediate layer.
The output layer of the neural network outputs the coordinates (x, y) of the pick-up position of the pick-upable workpiece 20 shown in Figure 9, as well as a predicted value S indicating the ease of pick-up operation at the detected pick-up position, and a predicted attitude U that is estimated to be highly likely to be greater than or equal to the judgment value T, calculated from the input image data.

<Collection of learning data>
A method of collecting data used for learning will be described. In this embodiment, as shown in FIG. 8, a rectangular parallelepiped workpiece 20 is placed at the bottom of the container 13. At this time, the normal direction A2 of the surface of the workpiece 20 is assumed to be known in advance. FIG. 14 shows a flowchart of the learning procedure. The control device 100 in FIG. 7 controls the positioning mechanism PM (FIG. 14, step a1), moves the imaging device 9 to various positions (A), (B), and (C) around the workpiece 20 as shown in FIG. 8 (FIG. 14, step a2), photographs the workpiece 20 (FIG. 14, step a3), and stores the output image data in the storage unit 115 in FIG. 7 (FIG. 14, steps a4 and a5). The control device 100 also stores the rotation angle φ and bending angle θ of the angle adjustment device 8, and the correct attitude UG as a better command value for moving from that attitude to the normal direction A2, which is the optimal approach direction of the workpiece 20 in FIG. 8, in the storage unit 115 in FIG. 7 in association with the image data to be stored.

After capturing the learning images, the annotation work performed by the worker on each image data stored in the memory unit 115 will be described. Figure 9 shows one of the image data stored in the memory unit 115 of Figure 7 displayed on the monitor 110 connected to the control device 100. The worker then uses an input device 109 such as a mouse or keyboard to set a pick-up position PP on the surface of the workpiece 20 in the image, as shown in Figure 9. If the coordinates set here are (x, y), these data are stored in the memory unit 115 of Figure 7 as indicating the pick-up position PP (Figure 14, step a5).

As shown in Figure 10, an image captured from a direction in which the optical axis 9a indicating the shooting direction of the imaging device 9 coincides with the normal A2 to the surface of the workpiece 20 is defined as an image that is easy to pick up, and a maximum value of 100 is set as the correct answer value SG indicating the ease of the pick-up operation for the pick-up position PP (Figure 9) of this image.
The operator sets a point indicating the pick-up position PP using an input device 109 (FIG. 7) such as a mouse or a keyboard. The correct answer value SG indicating the ease of the pick-up operation is calculated by the following formula (1) using, for example, the angle ε between the normal A2 to the surface of the target workpiece 20 and the approach direction of the angle adjustment device.

SG=100×cos(ε/2)…(1)
The correct answer value SG has a maximum value of 100 when ε=0, and decreases as ε increases. ε is a value in the range from 0 to π. SG has a minimum value of 0 when ε=π.
The coordinates (x, y) set in the above work and the correct value SG indicating the ease of the pick-up operation are stored in the storage unit 115 of Fig. 7. In this embodiment, the correct value is However, it may be determined according to the shape of the workpiece or the structure and function of the pickup unit. The above explanation is an example of picking up a rectangular workpiece by a suction type pickup unit. This is what was done.

<Learning process>
The learning unit 113 performs learning by the backpropagation method using the image data collected by the above work, image data obtained by changing the work position in the image and the like using computer graphics CG based on the image data and increasing the number of n, coordinates (x, y), a correct value SG indicating the ease of the pick-up operation, and a correct attitude UG at which the correct value SG is the maximum value "100", and stores the result in the storage unit 115. In this embodiment, when the optical axis 9a of the imaging device 9 in FIG. 8 coincides with the normal line A2 of the work 20, the "prediction value" indicating the ease of the pick-up operation becomes the maximum value "100". The detection unit 112 in FIG. 7 performs numerical calculations of the neural network using this learning result.

<A series of operations of the pickup device>
An example is shown in which a series of operations from detection of a target pick-up position to pick-up operation is executed using a neural network after learning. Figures 11A and 11B are flowcharts for explaining a series of operations of a pick-up device. Figure 12 is a representative configuration diagram of a control device 100 that executes the processing of the flowchart. In Figure 12, memory 102 corresponds to storage unit 115 in Figure 7. Moreover, processor 101 in Figure 12 functions as image data acquisition unit 111, detection unit 112, learning unit 113, and control unit 114 in Figure 7 by reading and executing a program stored in memory 102.

The control device 100 controls the positioning mechanism PM to move the imaging device 9 to the vicinity of the workpiece (FIG. 11A, step S1). Next, the control device 100 controls the positioning mechanism PM to orient the optical axis of the imaging device 9 toward the workpiece (FIG. 11A, step S2), and captures an image from a certain shooting direction (FIG. 11A, step S3).

The control device 100 acquires image data from the imaging device 9, inputs it to a neural network, detects the coordinates (x, y) of the pick-up position of a pickable workpiece, and calculates a predicted value S and a predicted attitude U that indicate the ease of the pick-up operation (FIG. 11A, step S4). If the workpiece is not found and the predicted value cannot be calculated, or if the predicted value is not within a predetermined range (FIG. 11A, step S5: NO), the control device 100 transmits an error signal and ends the processing of this flowchart.

When a workpiece is found and a predicted value S is calculated (FIG. 11A, step S5: YES), the control device 100 compares the predicted value S, which indicates the ease of the pick-up operation obtained from the output layer, with a judgment value T (FIG. 11A, step S6). The presence or absence of a workpiece in step S5 may be determined using conventional image processing after step S3, but if the system is trained so that the predicted value S is less than a threshold value when no workpiece is present, separate processing is not required.

When the predicted value S of the output image data is smaller than the judgment value T (FIG. 11A, step S6: NO), the posture is changed to the predicted posture U (FIG. 11A, step S8), and the processing of steps S3 to S6 is repeated. In this way, when the predicted value S is less than the judgment value T, the control device 100 moves the pickup unit 10 to the predicted posture, and again repeats the process of obtaining the predicted value S and predicted posture U for the posture to which the pickup unit 10 is moved for the image data captured and output by the imaging device 9.

On the other hand, when the predicted value S is equal to or greater than the judgment value T (FIG. 11A, step S6: YES), the movement of the positioning mechanism PM is stopped, and the relative distance D to the pick-up position is measured using the distance sensor 11 (FIG. 6) or the like (FIG. 11B, step S7). At this time, the state before pick-up is photographed by the imaging device 9 (FIG. 11B, step S9). In addition, the position and optical axis direction of the imaging device 9 are stored in the memory unit 115.

Next, the control device 100 calculates the movement distance of each axis of the linear motion unit 4 required for picking up based on the relative distance D and the position and optical axis direction of the imaging device 9, and moves the pickup unit 10, which is an end effector, by the distance D parallel to the optical axis of the imaging device 9 (FIG. 11B, step S10). Then, the control device 100 controls the pickup unit 10 to perform a pick-up operation on the workpiece 20 (FIG. 11B, step S11). In this way, the control unit 114 of the control device 100 controls the pickup unit 10 to perform work on the workpiece determined as the work target from the determined approach direction. After the pick-up operation, the control device 100 retracts the pickup unit 10 by the distance D in a direction parallel to and opposite to the direction of entry (FIG. 11B, step S12). At this time, the control device 100 photographs the state after the pick-up operation using the imaging device 9 (FIG. 11B, step S13).

Next, the control device 100 determines whether the workpiece 20 has been picked up. For example, the control device 100 detects mismatches by comparing image data output from the imaging device 9 before and after the pick-up operation. The control device 100 can determine that the pick-up was successful if a mismatch corresponding to the workpiece to be picked up is detected, and can determine that the pick-up was unsuccessful if no change is detected (FIG. 11B, step S14). The process for determining whether the pick-up was successful is not limited to this method. As long as it is possible to determine whether the pick-up was successful, it may be determined using other sensors, such as a proximity sensor.

Thereafter, the control device 100 stores the above-mentioned determination result and the detected coordinates (x, y) in the storage unit 115 of the control device 100 together with the image before the pick-up operation (FIG. 11B, step S15).
If the pickup is successful (FIG. 11A, step S16: YES), the control device 100 ends the processing of this flowchart (FIG. 11A, step S17), and if the pickup process is not successful (FIG. 11A, step S16: NO), the control device 100 repeats the processing from step S2 onwards.

<Additional learning>
If necessary, additional learning may be performed using the above pick-up operation results. The control device 100 performs machine learning by adding to the teacher data an image of the object captured during the actual pick-up operation, a pick-up position detected in the image, and a correct answer value SG indicating the ease of the pick-up operation determined according to the pick-up operation result. The control device 100 then causes the subsequent workpiece pick-up operation to be performed based on the machine learning result updated by the additional learning.

In this case, if the pickup is successful, the correct answer value SG, which indicates the ease of pickup, is set to 100, and if it is unsuccessful, a new value obtained by multiplying the correct answer value SG by a coefficient α (where α < 1) is set to the correct answer value SG. The control device 100 adds the correct answer value SG and the coordinates (x, y) of the pickup position updated in this way to the teacher data, performs learning, and updates the learning result and stores it in the memory unit 115.

The coefficient α may be set to 0.3, for example, when a failure is detected in step S14 in Fig. 11B, and may be set to 0.5 when an image is taken during the approach operation and it is detected that the pickup unit 10 in Fig. 7 and the workpiece 20 (Fig. 8) have come into contact with each other and the posture of the workpiece has changed. Note that the coefficient α does not need to be limited to these values, and may be set according to the cause of the failure of the pickup operation.
The detected positions may be fine-tuned by a process similar to the annotation process described above. Furthermore, learning may be performed using the fine-tuned positions as teaching data, and the learning results stored in the storage unit 115 may be updated.
Additional learning enables rapid response to changes in the work environment, etc.

As shown in the flowchart of FIG. 15, even after the predicted value S is changed to a posture that is equal to or greater than the judgment value T (step S6: YES), the predicted value S may be changed to a predicted posture U that is equal to or greater than the judgment value T by repeating N times, and the posture in which the predicted value S is maximum among the predicted values S that are equal to or greater than the judgment value T may be set as the approach direction (steps S18 to S26). That is, after the predicted value S becomes equal to or greater than the judgment value T, the control device 100 in FIG. 7 repeatedly changes the posture of the pickup unit 10 to the predicted posture multiple times, and among the multiple image data captured and output by the imaging device 9 in each posture, determines the shooting direction in the posture that maximizes the predicted value S as the approach direction to the workpiece. This can improve the success rate of the pickup. Here, the pickup process in FIG. 15 is the process shown in S7 to S15 in FIG. 11B.

The control unit 114 in FIG. 7 may omit the learning function and use the learning results created by another device of the same type. By omitting the learning function, the calculation function required for machine learning and the storage function for learning data are unnecessary, making it possible to use parts with lower functionality. This allows the price of the control unit 114 to be reduced compared to a control unit with a learning function.

<Action and effect>
According to the pickup device 1 described above, the control device 100 acquires, based on image data, a predicted value S indicating the ease of working on the workpiece, and a predicted attitude U of the pickup unit 10 at which this predicted value S is predicted to be equal to or greater than a judgment value T. For example, the control device 100 selects the workpiece for which the acquired predicted value S is the largest as the work target. When the predicted value S is equal to or greater than the judgment value T, the control device 100 determines the imaging direction at the time of acquiring the image data as the approach direction to the workpiece. The control unit 114 controls the pickup unit 10 so that work is performed on the workpiece from the determined approach direction.

If all the predicted values S are less than the judgment value T, the control device 100 moves the pickup unit 10 to the predicted posture U corresponding to one of the workpieces, again captures an image of the workpiece using the imaging device 9 to obtain image data, and repeats the process of obtaining the predicted value S and predicted posture U at that posture.
This makes it possible to shorten the search time for an approach direction with a higher probability of picking up a workpiece, compared to a case where the pickup unit 10 is moved randomly to search for a posture where the predicted value is equal to or greater than the judgment value. Since the workpiece is worked on from the approach direction with a higher probability of picking up a workpiece, the success rate of the work on the workpiece can be improved, compared to the conventional technology described above.

The approach direction etc. determining means Ap of the control device 100 obtains the predicted value S and predicted attitude U by applying a prediction model generated by machine learning to the image data. In this case, the target attitude of the pickup unit 10 can be fine-tuned as necessary. This can further improve the success rate of work on the workpiece.

The positioning mechanism PM has a stage 33 (FIG. 6) to which the imaging device 9 and the pickup unit 10 are fixed, and the imaging device 9 and the pickup unit 10 are positioned so that the optical axis 9a of the imaging device 9 is parallel to the approach direction A1 (FIG. 5) of the pickup unit 10. In this case, after the workpiece is imaged by the imaging device 9, there is no need to correct any misalignment with the optical axis direction of the imaging device 9 when moving the pickup unit 10, thereby shortening the workpiece operation time in the pickup unit 10 and improving the success rate of the operation.

The positioning mechanism PM in FIG. 1 includes the base 32 in FIG. 6 and the stage 33 to which the imaging device 9 and the pickup unit 10 are fixed, and has an angle adjustment device 8, which is a first positioning mechanism that positions the stage 33 with two degrees of freedom of rotation relative to the base 32, and a second positioning mechanism 47 in FIG. 1 that positions the base 32. In this case, as shown in FIG. 6, the imaging device 9 can be fixed to the stage 33 on the tip side of the angle adjustment device 8 having two degrees of freedom of rotation, so that the imaging device 9 can be positioned near the workpiece. In this case, image data with higher resolution than that of the conventional technology in which the imaging device is fixed above the workpiece can be obtained. In addition, the ease of work on the workpiece is determined using image data obtained by the imaging device 9 configured to be able to change the shooting direction, and the work is performed by approaching from the same direction as the shooting direction. This improves the success rate of work on the workpiece in the pickup device 1 in FIG. 1.

The imaging device 9 and stage 33 are positioned so that the optical axis 9a of the imaging device 9 in FIG. 6 coincides with the central axis QB of the stage 33. In this case, the pickup unit 10 can approach the workpiece from the direction in which the workpiece was photographed without changing the posture of the pickup unit 10. This improves the success rate of the workpiece operation and shortens the time required to change posture, compared to a configuration in which the optical axis 9a of the imaging device 9 and the central axis QB of the stage 33 do not coincide.

The first positioning mechanism is an angle adjustment device 8 that includes a base 32, a stage 33, two or more link mechanisms 34 that connect the stage 33 to the base 32 so that the position can be changed, and an actuator 31 that drives the link mechanisms 34. In this case, the mass of the moving parts is smaller and high-speed positioning is possible compared to a two-degree-of-freedom mechanism in which two rotational positioning mechanisms are arranged in series. Therefore, the search time for the optimal approach direction can be shortened.

<Other embodiments>
In the following description, parts corresponding to matters previously described in each embodiment are given the same reference numerals, and duplicated description is omitted. When only a part of the configuration is described, the other parts of the configuration are the same as the previously described embodiment unless otherwise specified. The same configuration has the same effect. It is possible to combine not only the parts specifically described in each embodiment, but also partially combine the embodiments, provided that there is no particular problem with the combination.

<Modifications of the Pickup Unit>
In the first embodiment, an example in which the pickup unit 10 in Fig. 5 is provided with the suction pads 10A and 10B is shown, but the pickup unit 10 may be configured to pick up an object by other means than suction. For example, the pickup unit 10 may be configured to grip an object by other hand mechanisms in a similar manner.
A pickup unit 200 of a modified example shown in Fig. 13 is attached to the stage 33 together with the imaging device 9. The pickup unit 200 includes a gripper base 201 and grippers 202 and 203. The grippers 202 and 203 are in an open state as shown in Fig. 13 when approaching a workpiece, and are in a closed state when picking up the workpiece. In the closed state, the workpiece 20 (see Fig. 1) is gripped between the grippers 202 and 203 when the pickup is successful.
In this manner, the pick-up operation may be performed by gripping the workpiece with the gripping parts 202 and 203. Although an example in which the gripping part has two fingers has been shown, the gripping part may have three or more fingers.

[Second embodiment: vertically articulated type]
As shown in Fig. 16, a vertical articulated type articulated mechanism PMa may be used as the positioning mechanism. The articulated mechanism PMa includes a base unit 48 and a plurality of arms 49a to 49f (first to sixth in this example) having a plurality of joints. The arms 49a to 49f are sequentially connected to the base unit 48. The base unit 48 is installed on the floor or the like, which is the ground. A pickup unit 10, which is an end effector, and an imaging device 9 are attached to the tip of the sixth arm 49f on the tip side.

The first arm 49a on the base end side is rotatably mounted on a rotary drive source 50 provided on the base unit 48 around the Z-axis direction. A drive source 51 such as a motor is provided at each joint. The base unit 48 is provided with a controller CR, which is a control device. The imaging device 9, the pickup unit 10, the rotary drive source 50, and each drive source 51 are controlled by the controller CR. The pickup device 1A equipped with this multi-joint mechanism PMa has a slower positioning speed than the first embodiment equipped with a parallel link mechanism, but achieves the same effects.

[Third embodiment: horizontal multi-joint type]
As shown in FIG. 17, a horizontal articulated type articulated mechanism PMb may be applied as the positioning mechanism. A first arm 49a on the base end side is provided rotatably around the Z-axis direction via a rotation mechanism 53 on a rotary drive source 50 provided on a base unit 48. A second arm 49b is provided rotatably around the Z-axis direction via a rotation mechanism 53 on the tip of the first arm 49a. A third arm 49c on the tip side is configured to be movable in the Z-axis direction relative to the second arm 49b by a linear motion mechanism 52. A pickup unit 10 as an end effector and an imaging device 9 are attached to the tip of the third arm 49c via a rotation mechanism 53. The pickup device 1B equipped with this articulated mechanism PMb also exhibits the same effects as the first embodiment equipped with a parallel link mechanism, although the positioning speed is inferior.

[Fourth embodiment: vertical articulated type + link actuator]
18, the positioning mechanism may be a mechanism in which the link actuator 8 is connected via a vertical articulated type articulated mechanism PMa. In this example, a base 32 of the link actuator 8 is attached to the tip of the third arm on the tip side.
[Fifth embodiment: horizontal multi-joint type + link actuator]
19, the positioning mechanism may be a mechanism in which the link actuator 8 is connected via a horizontal articulated type articulated mechanism PMb. In this example, the base 32 of the link actuator 8 is attached to the tip of the third arm 49c on the tip side via a rotation mechanism 53.
According to the fourth and fifth embodiments, the pickup device as a whole can be made compact, thereby reducing the working space required, and other effects similar to those of the first embodiment can be achieved.

The distance sensor 11 in FIG. 6 may be provided integrally with the imaging device 9, or may detect the distance to the workpiece 20 (FIG. 1) by utilizing the focusing function of the imaging device 9.
In the first embodiment, the two suction pads 10A, 10B are arranged adjacent to both sides of the imaging device 9, but the suction pads 10A, 10B may be arranged in appropriate positions depending on the shape of the workpiece to be picked up.

In the first embodiment, two suction pads are used, but three or more suction pads may be used depending on the weight and shape of the workpiece. If a workpiece can be picked up with one suction pad, there is no need to install multiple suction pads and one suction pad will suffice. As the pickup unit 10 that picks up the workpiece, any gripping mechanism such as a clamping mechanism or a multi-fingered hand other than a suction pad can be used.

In the first embodiment, an example of using a neural network for workpiece detection was shown, but a general supervised learning method such as a support vector machine, an unsupervised learning method such as an autoencoder, k-means, or principal component analysis may also be used, or a machine learning method that combines multiple learning methods may also be used.
7, in addition to the backpropagation method, a general supervised learning method such as a support vector machine may be used, or an unsupervised learning method such as an autoencoder, k-means method, or principal component analysis may be used. Also, a machine learning method that combines a plurality of learning methods may be used.

The control device 100 may obtain, based on the image data, a predicted value S indicating the ease of work on the workpiece without performing machine learning, and a predicted attitude U of the pickup unit 10 at which the predicted value S is predicted to be equal to or greater than a judgment value T according to a determined judgment criterion. The determined judgment criterion is determined by, for example, obtaining an appropriate judgment criterion through either one or both of a test and a simulation.
In addition to picking up a workpiece, the specified tasks may include, for example, assembly, welding, greasing, and visual inspection.

Although the embodiments of the present invention have been described above, the embodiments disclosed herein are illustrative in all respects and are not limiting. The scope of the present invention is indicated by the claims rather than the above description, and is intended to include all modifications within the meaning and scope of the claims.

1...Pickup device (working device), 8...Angle adjustment device (first positioning mechanism), 9...Imaging device, 10...Pickup unit (working unit), 20...Work, 31...Actuator, 32...Base, 33...Stage, 34...Link mechanism, 47...Second positioning mechanism, 100...Control device, 114...Control unit, Ap...Means for determining approach direction, etc., PM...Positioning mechanism,

Claims (7)

An imaging device that captures an image of a workpiece and outputs image data;
A working unit that performs a predetermined operation on the workpiece;
a positioning mechanism for positioning the imaging device, the working unit, and the workpiece relative to each other;
a control device that controls the imaging device, the working unit, and the positioning mechanism,
The control device includes:
an approach direction etc. determining means for obtaining a predicted value indicating the ease of working on the workpiece and a predicted posture of the working part when this predicted value is predicted to be equal to or greater than a judgment value based on image data output from the imaging device, determining the workpiece to be worked on based on the obtained predicted value, and determining the imaging direction by the imaging device when the predicted value is equal to or greater than the judgment value as the approach direction to the workpiece determined to be the workpiece;
a control unit that controls the working unit so as to perform work on the workpiece determined as a work target from the determined approach direction,
When the predicted value is less than a judgment value, the control device moves the working unit to the predicted posture, and again repeats the process of obtaining the predicted value and the predicted posture in the posture to which the working unit is moved for image data captured and output by the imaging device. The working device according to claim 1, wherein the approach direction etc. determining means obtains the predicted value and the predicted attitude by applying a prediction model generated by machine learning to the image data. In the working device according to claim 1 or 2, the positioning mechanism has a stage to which the imaging device and the working unit are fixed, and the imaging device and the working unit are arranged so that the optical axis of the imaging device and the approach direction of the working unit are parallel. In the working device according to claim 1 or 2, the positioning mechanism includes a base and a stage to which the imaging device and the working unit are fixed, a first positioning mechanism that positions the stage with two degrees of freedom of rotation relative to the base, and a second positioning mechanism that positions the base. In the working device according to claim 4, the control device repeatedly changes the posture of the working unit to the predicted posture multiple times after the predicted value becomes equal to or greater than a judgment value, and determines, as the approach direction to the workpiece, the photographing direction in the posture in which the predicted value is maximized among the multiple image data captured by the imaging device and output in each posture. The working device according to claim 4, wherein the imaging device and the stage are arranged so that the optical axis of the imaging device and the central axis of the stage coincide with each other. In the working device according to claim 4, the first positioning mechanism is an angle adjustment device that includes the base and the stage, two or more sets of link mechanisms that connect the stage to the base in a manner that allows the stage to change its position, and an actuator that drives the link mechanisms.

Images