JP2015136763A - Control device, robot system, robot and robot control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a control device capable of specifying whether or not an end effector of a robot is positioned in a first region, and further to provide a robot system, a robot, a robot control method and the like.SOLUTION: A control device 100 includes: a captured image reception section 110 receiving a captured image obtained by a first region being imaged by an image section; and a processing section 130 activating a robot 300 provided with an end effector 310 operating in the first region. Further, the processing section 130 determines whether or not the end effector 310 is in the first region on the basis of information of a marker 50 imaged in the captured image.

Description

  The present invention relates to a control device, a robot system, a robot, a robot control method, and the like.

  In recent years, industrial robots have been increasingly introduced to mechanize and automate work performed by humans at production sites. However, precise positioning is a prerequisite for positioning the robot, which is a barrier to robot introduction.

  Here, visual servoing is one of the methods for positioning the robot. Visual servo is a technique for controlling a robot based on a difference between a reference image (goal image, target image) and a captured image (current image). Certain types of visual servos are useful in that they do not require precision in calibration, and are attracting attention as a technology that lowers the barrier for robot introduction.

  As a technique related to this visual servo, there is a conventional technique described in Patent Document 1, for example.

JP 2003-305675 A

  In visual servoing, an object detection process is performed on a captured image, for example, a detection target such as a work target or a robot end effector is detected, and the robot is controlled based on the detection result. Therefore, when visual servoing is performed, it is necessary that the detection target is reflected in the captured image. That is, assuming that the range in real space (imaging range) reflected in the captured image is the robot work range, the detection target must be positioned within the work range in order to perform visual servo normally.

  However, even if the detection target is reflected in the captured image, it may be determined that the detection target is not reflected, or the detection target is not reflected in the captured image. It may be mistakenly determined that In such a case, the robot cannot be controlled normally.

  According to one aspect of the present invention, a captured image receiving unit that receives a captured image obtained by imaging a first region by an imaging unit, a processing unit that operates a robot including an end effector that operates in the first region, The processing unit is related to a control device that determines whether or not the end effector is in the first region based on information on a marker reflected in the captured image.

  In one aspect of the present invention, it is determined whether or not the end effector of the robot is in the first region based on whether or not the marker is recognized in the captured image.

  Therefore, it is possible to specify whether or not the end effector of the robot is located in the first region.

  Moreover, in one aspect of the present invention, the captured image reception unit receives the captured image obtained by capturing at least one marker in the second area, and the processing unit receives the captured image. And performing recognition processing of the marker arranged in the second region, and determining whether or not the end effector is located in the first region based on the result of the recognition processing. Visual servo or inspection processing may be performed.

  This makes it possible to perform visual servoing or inspection processing based on the result of the marker recognition processing.

  In one embodiment of the present invention, the processing unit performs the recognition process of the marker so that at least a part of the area of the marker is shielded by the work object of the robot or the end effector. A marker shielding state that is a state may be detected, and based on the detected marker shielding state, it may be determined whether or not the end effector is located in the first region.

  Accordingly, it is possible to determine whether or not the end effector is located in the first region based on the marker arrangement and the marker shielding state.

  In one aspect of the present invention, when the processing unit determines that at least a partial region of the marker is shielded by the work object or the end effector, the end effector is the first effector. And determining that at least a part of the area of the marker is not shielded by the work object or the end effector, the end effector is outside the first area. You may determine that it is located in.

  Accordingly, it is possible to determine whether or not the end effector is positioned in the first region based on the presence or absence of the marker shielding state by the work object or the end effector.

  In the aspect of the invention, the second region may be a region located behind the first region from the position of the imaging unit toward the imaging direction of the captured image.

  Thereby, for example, it is possible to place a marker on the background of the work range of the robot.

  In the aspect of the invention, the captured image reception unit may be configured such that the marker is not arranged in the first area, and the marker is arranged in the second area outside the first area. And when the processing unit determines that at least a partial area of the marker is shielded by the work object or the end effector, the end When it is determined that the effector is located outside the first region, and it is determined that at least a partial region of the marker is not shielded by the work object or the end effector, the end effector is the first effector. It may be determined that it is located within one region.

  Thereby, when performing visual servo and inspection processing, while preventing the marker placed in the second area from affecting the object detection processing, based on the presence or absence of marker shielding state by the work object or the end effector, It becomes possible to determine whether or not the end effector is positioned within the first region.

  In the aspect of the invention, the processing unit may set the captured image as a region of interest corresponding to the first region, and perform the visual servo or the inspection process in the region of interest. Good.

  This makes it possible to perform visual servoing or inspection processing based on an image of a region where no marker is shown in the captured image.

  In one aspect of the present invention, the captured image reception unit may receive the captured image obtained by imaging a plurality of markers arranged in the second region.

  Thereby, it becomes possible to determine in more detail whether or not the end effector is located in the first region than when only one marker is arranged.

  In one aspect of the present invention, the captured image reception unit may receive the captured image obtained by capturing a plurality of different markers in the second region.

  Thereby, it becomes possible to specify the region shown in the captured image, etc., according to the recognized marker type.

  In the aspect of the invention, the processing unit may specify the imaging direction of the captured image based on the recognition processing result of the plurality of different markers in the captured image.

  This makes it possible to adjust the orientation of the imaging unit.

  In the aspect of the invention, the processing unit may specify the position and orientation of the work target of the robot or the end effector of the robot based on the recognition processing result of the plurality of markers in the captured image. Processing may be performed.

  This makes it possible to compare the actual position and orientation of the work object or end effector with the target position and orientation of the work object or end effector.

  In the aspect of the invention, the processing unit may perform a process of controlling the robot so that the captured image matches or approaches a reference image as the visual servo.

  As a result, the position and orientation of the robot can be set to the target position and orientation.

  Another aspect of the present invention includes a robot including an end effector that operates in a first area, a marker, and an imaging unit that images the first area. The present invention relates to a robot system that determines whether or not the end effector is in the first region based on information of the marker reflected in a captured image.

  Further, another aspect of the present invention includes an end effector that operates in the first area, and whether the end effector is in the first area based on marker information reflected in a captured image captured by the imaging unit. It relates to a robot that determines whether or not.

  Another aspect of the present invention is a robot control method for operating a robot including an end effector that operates in the first region, based on imaging a marker and marker information reflected in the captured image. Determining whether or not the end effector is in the first region.

  According to some aspects of the present invention, it is possible to provide a control device, a robot system, a robot, a robot control method, and the like that can specify whether or not an end effector of a robot is located in the first region. it can.

  In addition, according to some aspects of the present invention, even if the end effector of the robot is not detected or erroneously detected as a result of performing the object detection process on the captured image, the end effector is positioned in the first region. It is possible to provide a control device, a robot system, a robot, a robot control method, and the like that can specify whether or not to do so.

The system configuration example of this embodiment. FIGS. 2A to 2D are explanatory diagrams of processing when one marker is arranged on the background. FIGS. 3A to 3F are explanatory diagrams of markers. The flowchart explaining the flow of a process of this embodiment. FIGS. 5A to 5F are explanatory diagrams of processing when a plurality of markers are arranged in the background. The flowchart explaining the flow of a process of 1st Example. FIG. 7A and FIG. 7B are explanatory diagrams of marker placement positions. FIG. 8A to FIG. 8C are explanatory diagrams of processing when a marker is arranged only outside the work range. The flowchart explaining the flow of a process of 2nd Example. 10A and 10B are explanatory diagrams in the case where a plurality of different markers are arranged. FIG. 11A to FIG. 11D are explanatory diagrams of a process for specifying the position and orientation of the detection target. The flowchart explaining the flow of a process of a 3rd Example. FIG. 13A and FIG. 13B are configuration examples of a robot system and a robot. 1 is a configuration example of a robot system that controls a robot via a network. Flow chart of position-based visual servo. 16A and 16B are explanatory diagrams of a reference image and a captured image. Flow chart of feature-based visual servo.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Method of this Embodiment In order to perform visual servo normally, the detection target must be located within the work range (first region) of the robot in real space. When the detection target is located outside the work range, it is necessary to restart the visual servo after that, for example, by moving the robot so that the detection target is located within the work range. In order to perform such control, it is necessary to determine whether or not the detection target is located within the robot work range.

  Therefore, in the present embodiment, a marker is arranged on the background (second region) of the detection object, and it is determined whether or not the detection object is within the work range based on the detection result of the arranged marker.

  Here, the work range is a range in the real space where the detection target needs to be positioned within the range when the robot performs work by visual servo. The work range is a specific example of the first area. As described above, when visual servoing is performed, it is necessary for the detection target to be reflected in the captured image, so the work range is often the same as the imaging range. The imaging range is a range in real space that appears in the captured image captured by the imaging unit. For example, the imaging range is a range WA shown in FIG. In the case of FIG. 2A, the work range is the same as the imaging range. However, as shown in FIGS. 7A and 7B described later, the work range is not necessarily the same as the imaging range. Note that the detection target in the present embodiment mainly refers to a robot work target or a robot end effector.

  Next, a configuration example of the control device 100 of this embodiment is shown in FIG. The control device 100 according to the present embodiment is a process for operating a robot including a captured image receiving unit 110 that receives a captured image obtained by imaging the first region by the imaging unit 200 and an end effector that operates in the first region. Part 130.

  Then, the processing unit 130 determines whether or not the end effector 310 is in the first region based on the marker information shown in the captured image.

  Specifically, the captured image reception unit 110 receives a captured image obtained by imaging at least one marker 50 arranged in the second region. For example, the captured image reception unit 110 acquires a captured image obtained by capturing and imaging at least one marker 50 on the background of the work target or the end effector 310 of the robot 300.

  Further, the processing unit 130 performs visual servoing or inspection processing based on the captured image. At this time, the processing unit 130 performs recognition processing of the marker 50 arranged in the background (second region), and the work target or the end effector 310 is located within the work range based on the result of the recognition processing. Whether to determine whether or not. The work target is, for example, a work or a pallet.

  Here, a specific example will be described with reference to FIGS. FIG. 2A shows a state where the end effector EE (gripping unit) of the robot grips the work object OB and puts the grasped work object OB into the hole HL by visual servoing. Note that AM is a robot arm.

  This visual servoing is performed based on captured images (IM1 to IM3) as shown in FIGS. 2B to 2D captured by the imaging unit CM. In these captured images, a state within the imaging range WA (working range) in FIG. A marker MK is arranged on the background BG of the work object OB and the end effector EE, and the angle of view of the imaging unit CM is adjusted so that the entire marker MK is reflected in the captured image when there is no shielding object. ing.

  In this case, since the imaging range WA reflected in the captured image is set as the work range, if the detection target is reflected in the captured image, the detection target is located within the work range, and the detection target Can be determined that the detection target is located outside the work range. However, in the present embodiment, as described above, it is determined whether or not the detection target is located within the work range based on the result of the marker MK recognition process.

  Specifically, as in the captured image IM1 in FIG. 2B and the captured image IM2 in FIG. 2C, a part of the marker MK is shielded by the work object OB, and the marker MK is correctly recognized. If not, it is determined that the work object OB is located within the work range.

  On the other hand, when the marker MK is not shielded by the work object OB and can be correctly recognized as in the captured image IM3 of FIG. 2D, the work object OB is positioned outside the work range. Judge that.

  As described above, according to the present embodiment, it is possible to specify whether or not the detection target (work target or robot end effector) is positioned within the work range.

  Here, the captured image reception unit 110 may be a communication unit (interface unit) that communicates with the imaging unit 200 via a network including at least one of wired and wireless, or may be the imaging unit 200 itself. .

  The function of the processing unit 130 can be realized by hardware such as various processors (CPU and the like), ASIC (gate array and the like), a program, and the like.

  And the control apparatus 100 is not limited to the structure of FIG. 1, Various deformation | transformation implementations, such as abbreviate | omitting some of these components and adding another component, are possible. Furthermore, the control apparatus 100 of this embodiment is included in the robot 300 as shown in FIG. 13B described later, and may be configured integrally with the robot 300. Furthermore, as illustrated in FIG. 14 described later, the function of the control device 100 may be realized by a server 500 and a terminal device 330 (control unit) included in each robot 300.

  The captured image refers to an image obtained by capturing with the image capturing unit 200. The captured image may be an image stored in an external storage unit or an image acquired via a network.

  The background of the work object or the background of the end effector 310 of the robot 300 is, for example, a wall, a plate, a column, a stand, a screen, and the back of the work object or the end effector 310 when viewed from the position of the imaging unit 200 A display (display unit) or the like, which is BG in FIG. The background is not limited to this as long as the marker 50 can be arranged. When the background is a display unit such as a screen or a display, the display unit displays a marker. However, the background of the work object or the background of the end effector 310 needs to be an area within the imaging range so that the marker 50 appears in the captured image.

  Here, the background of the work object or the background of the end effector 310 is a specific example of the second region. That is, when the second area is the background of the work object or the background of the end effector 310, the second area is the first area from the position of the imaging unit 200 toward the imaging direction of the captured image. It can be said that this is an area located behind.

  Here, the marker 50 is a tangible object that can be used as a mark, a character, a figure, a symbol, a pattern, a three-dimensional shape, a combination thereof, or a combination of these and a color, and can be fixed to an object. is there. For example, stickers, stickers, labels, and the like. It does not matter what the shape, color, pattern, etc. of the marker 50 is, but it is easy to distinguish the marker from other regions for stable detection of the marker, for example, black for a white region Images and stickers made of are desirable. Specific examples include markers MK1 to MK4 as shown in FIGS. 3A to 3D, markers MK5 to MK8 such as map symbols shown in FIG. 3E, and FIG. Examples thereof include markers MK11 to MK13 such as letters and numbers shown. Furthermore, the marker 50 may be a QR (Quick Response) code, an AR (Augmented Reality) marker, or the like. As an exception, the marker 50 may be an image that can be displayed on the display unit.

  As described above, since a marker that is easier to detect than the detection target is used as a marker, the marker can be recognized at a recognition rate higher than the detection target recognition rate (detection rate), and the detection target is positioned within the work range. It is possible to more reliably determine whether or not to do so.

  Then, the processing unit 130 performs a process of controlling the robot 300 as a visual servo so that the captured image matches or approaches the reference image.

  Thereby, the position and orientation of the robot 300 can be set to the target position and orientation. That is, if an image that can be captured with the robot 300 in the target posture is set as a reference image, and the robot 300 is controlled so that a captured image that matches or is close to the reference image is controlled, the robot 300 takes the target posture. It becomes possible to control. The visual servo will be described in more detail later.

  As a comparative example of the present embodiment, as described above, whether or not the detection target is within the work range is determined based on whether or not the detection target itself has been detected in the object detection process for the captured image. A method is also conceivable. In contrast to such a method, the method of the present embodiment is superior in the points described below.

  Here, the results of the object detection process will be described before describing the advantages of the present embodiment over the comparative example.

  In a visual servo whose operation is unstable, the detection target may be detached during the servo from the angle of view of the imaging unit that images the detection target. As described above, the reason why the visual servo is not successful is highly likely that the object detection process has failed.

  Here, the results of the object detection process can be roughly divided into the following four cases. The first case is a case where the detection object is reflected in the captured image, and a result that the detection object has been detected correctly is obtained. In this first case, the visual servo can be continued without any problem thereafter.

  Next, in contrast to the first case, the second case is a case where the detection target is not reflected in the captured image, and the result that the detection target cannot be detected is obtained. In the second case, visual servoing cannot be performed as it is, but visual servoing can be resumed by moving the robot to a state in which the detection target appears in the captured image.

  In the first case and the second case described above, although there is a difference in whether or not the detection target can be detected, the object detection process itself is normally performed, and a correct result is obtained. The following two cases (third case and fourth case) are problematic.

  In the third case, although the detection object is reflected in the captured image, it is erroneously determined that the detection object is not reflected, and as a result, the detection object cannot be detected. This is the case.

  On the other hand, the fourth case is the reverse of the third case, and it is erroneously determined that the detection target is reflected even though the detection target is not reflected in the captured image, and the original detection target is detected. This is a case where a result that a non-object is detected as a detection target is obtained.

  That is, in both the third case and the fourth case, it is wrong to determine whether or not the detection target is reflected in the captured image, and the object detection process itself is not normally performed. Therefore, visual servoing cannot be normally performed based on the result of the object detection process obtained in these cases.

  In summary, based on the result obtained by the object detection process, the visual servo can be continued normally thereafter only in the first case.

  However, as described above, in the second case, there is no problem in the object detection process itself, and therefore, it is possible to take measures such as restarting the visual servo after moving the robot to a state in which the detection target appears in the captured image. it can. On the other hand, in the third case and the fourth case, since there is a problem in the object detection process itself, the visual servo cannot be normally continued unless the method of the object detection process is changed.

  That is, even if the detection object cannot be detected correctly, if it can be determined whether this time is the second case, the third case, or the fourth case, how the robot is controlled thereafter. It can be determined whether it should be.

  However, in the comparative example described above, based on the detection result of the detection object, it is determined whether or not the detection object is located within the work range. It is not possible to correctly determine which one belongs to. On the other hand, in the present embodiment, it is possible to correctly determine which of the four cases the result of the current object detection process belongs, and to perform processing suitable for each case. The present embodiment is superior to the comparative example in this respect.

  Specifically, the processing flow of this embodiment will be described with reference to the flowchart of FIG.

  First, as shown in FIGS. 2A to 2D described above, at least one marker MK is arranged on the background BG of the work object OB (or the background BG of the end effector EE of the robot) and imaged. The captured images (IM1 to IM3) obtained in this way are acquired (S101).

  Next, object detection processing is performed on the acquired captured image (S102), and recognition processing of the marker MK arranged on the background BG is performed based on the captured image (S103).

  In step S102, it is determined whether or not the detection target has been detected (S104). If it is determined that the detection target has been detected, a determination process is performed to determine whether or not the detection target (work target or end effector) is located within the work range based on the result of the recognition process in step S103. It performs (S105).

  If it is determined in step S105 that the detection target is located within the work range, that is, in the case of the first case described above, since the object is detected normally, the visual is determined based on the result of the determination process. Servo or inspection processing is performed (S106).

  Then, it is determined whether the visual servo has converged or whether the inspection process has been completed (S107). If it is determined that the visual servo has converged or the inspection process has been completed, the process ends. . On the other hand, if it is determined that the visual servo has not converged or the inspection process has not been completed, the process returns to step S101 and the process is repeated.

  On the other hand, if it is determined in step S105 that the detection target is not located within the work range, it can be determined that the fourth case in which erroneous detection has occurred in step S102. In this case, since the object cannot be detected normally, it is determined that an error has occurred, and the process ends. Alternatively, in this case, the object detection processing method may be changed, and the process may be repeated by returning to step S101 again.

  Even when it is determined in step S104 that the detection target has not been detected, whether or not the detection target is positioned within the work range based on the recognition processing result in step S103, as in step S105. The determination process is performed (S108).

  If it is determined in step S108 that the detection target is located within the work range, it can be determined that the third case in which a detection error has occurred in step S102. Also in this case, as in the fourth case, since the object detection has not been performed normally, the processing is terminated assuming that an error has occurred. As described above, the error detection processing performed in the third case and the fourth case cannot be performed in the comparative example, and such error detection processing can be performed because of the advantage of this embodiment. is there.

  On the other hand, if it is determined in step S108 that the detection target is not located within the work range, that is, in the second case described above, the object detection process itself is normal, and thus the detection target After the robot is controlled to be positioned within the work range (S109), the process returns to step S101 and the process is repeated. In the comparative example, since it is not possible to determine that it is the second case, the process of step S109 cannot be performed only in the second case. Therefore, the advantage of this embodiment is that the process of step S109 can be performed only in the second case. The above is the processing flow of this embodiment and the notable advantages.

  Next, details of the processing in steps S105 and S108 in the flowchart of FIG. 4 will be described.

  The processing unit 130 detects a marker shielding state in which at least a part of the marker 50 is shielded by the work object or the end effector 310 as a result of the recognition process of the marker 50, and the detected marker shielding is performed. Based on the state, it is determined whether or not the work object or the end effector 310 is located within the work range.

  For example, when the marker shielding state is detected as in the examples of FIGS. 2B and 2C described above, it is determined that the work object OB (detection object) is located within the work range. As in the example of FIG. 2D, when the marker shielding state is not detected, it is determined that the work object OB is located outside the work range.

  However, the content of the determination process for determining whether or not the detection target is located within the work range is not limited to this, and the specific content of the process varies depending on the arrangement of the markers 50.

  In the first embodiment described below, as shown in FIG. 5A described later, a plurality of markers (MK11 to MK53) are arranged in an array on the background BG of the object to be detected within the work range WA. When it is detected that any of the plurality of markers is in the marker shielding state as shown in FIGS. 5B to 5D, it is determined that the detection target is located within the work range WA. As shown in FIG. 5E described later, when the marker shielding state is not detected in any of the plurality of markers, it is determined that the detection target is located outside the work range WA. 5A to 5D are examples in which the detection target is only the work target OB, and the occurrence of the marker shielding state by the end effector EE is not considered. The same applies to FIGS. 7A, 8A, and 8B described below.

  On the other hand, in the second embodiment described below, as shown in FIG. 7A described later, no marker is arranged in the work area WA (and its background), and the background BG outside the work area WA is arranged. Place multiple markers. Contrary to the first embodiment, as shown in FIG. 8A described later, when the marker shielding state is not detected in any of the plurality of markers, the detection target is positioned within the work range WA. Then, when it is detected that any one of the plurality of markers is in the marker shielding state as shown in FIG. 8B described later, it is determined that the detection target is located outside the work range WA.

  Here, the marker shielding state is a state in which the entire region or a partial region of the marker 50 is shielded by the work object or the end effector 310 when the marker 50 is viewed from the imaging unit 200. The marker occlusion state can also be said to be a state where the marker 50 cannot be recognized in the acquired captured image due to the occurrence of occlusion by the work object or the end effector 310 when capturing the captured image. For example, FIG. 2B and FIG. 2C described above are examples of the marker shielding state, and FIG. 2D is an example not of the marker shielding state.

  Further, at least a part of the region of the marker 50 is, for example, when at least one marker MK is arranged as shown in FIG. 2A described above, at least of the entire region of the marker MK. This is a partial area, and may be the entire area of the marker MK. And when several markers (MK11-MK53) are arrange | positioned like FIG. 5 (A) mentioned later, the whole area | region of one part marker, or one part of some markers are arranged. Refers to an area.

  Accordingly, it is possible to determine whether or not the detection target is located within the work range based on the marker arrangement and the marker shielding state.

  Further, in the third embodiment, as shown in FIGS. 10A and 10B described later, a plurality of different markers (MK11 to MKij) are arranged on the background BG, and the detection target is within the working range. In addition to determining whether or not the detection target is located, the position and orientation of the detection target are specified based on the marker in the marker shielding state among the plurality of arranged markers.

2. First Example Next, a first example will be described with reference to FIGS. 5A to 5F and the flowchart of FIG.

  First, in the first example, the captured image reception unit 110 receives a captured image obtained by imaging a plurality of markers 50 arranged in the background (second region) (S201).

  For example, as shown in FIG. 5A, a plurality of markers (MK11 to MK53) are arranged in an array on the background BG of the detection object within the work range WA. Then, the state in which the robot is controlled in that state is imaged by the imaging unit CM, and captured images IM1 to IM4 as shown in FIGS. 5B to 5E are acquired.

  Here, when only one marker MK is arranged as shown in FIG. 2A, for example, as shown in FIG. 5F, the work object OB is shown in the upper right corner of the captured image IM5. At this time, although the work object OB is located in the work range WA, the marker shielding state does not occur. For this reason, it is determined that the work object OB is not located within the work range WA.

  On the other hand, in the case of this example, since the marker MK51 is arranged at the upper right end of the background BG of the work range WA, even when the work object OB is at the position shown in FIG. It can be correctly determined that a marker shielding state occurs in the marker MK51 and the work object OB is located within the work range WA.

  As described above, in this example, only the work object OB is considered as the detection object. Therefore, it does not consider that the marker shielding state is generated by the end effector EE.

  Thereby, as shown in FIG. 2A, it is possible to determine more precisely whether or not the detection target is located within the work range than when only one marker is arranged.

  Next, the processing unit 130 performs marker recognition processing based on the acquired captured image (S202).

  When the processing unit 130 determines that at least a part of the area of the marker 50 is shielded by the detection target (work target, or in some cases, the end effector 310), that is, the marker that cannot be recognized. If it is determined that there is an object, it is determined that the detection object (work object, or in some cases, the end effector 310) is located within the work range (first region) (S203).

  For example, in the captured image IM1 in FIG. 5B, the marker shielding state is generated in the entire area of the markers MK31 to MK33 and a part of the markers MK41 to MK43 by the work object OB. Similarly, in the captured image IM2 of FIG. 5C, a marker shielding state is generated in the entire area of the markers MK51 to MK53 and a partial area of the markers MK41 to MK43 by the work object OB. Furthermore, in the captured image IM3 in FIG. 5D, a marker shielding state is generated in a partial region of the markers MK51 to MK53 due to the work object OB. Therefore, when a captured image as shown in FIGS. 5B to 5D is obtained, it is determined that the detection target is located within the work range WA.

  Therefore, in these cases, it is considered that visual servo can be normally performed. Therefore, visual servo is performed based on the captured image (S205), and it is determined whether the visual servo has converged (S206). ). If it is determined that the visual servo has converged, the process is terminated. On the other hand, if it is determined that the visual servo has not converged, the process returns to step S201 and the process is repeated.

  On the other hand, when the processing unit 130 determines that at least a part of the region of the marker 50 is not shielded by the detection target (work target, or in some cases, the end effector 310), that is, it cannot be recognized. If it is determined that there is no marker, it is determined that the detection target (work target, or possibly the end effector 310) is located outside the work range (first region) (S203).

  For example, in the captured image IM4 of FIG. 5E, the marker shielding state by the work object OB does not occur in any of the markers MK11 to MK53. Therefore, in this case, it can be determined that the detection target is located outside the work range.

  Therefore, the robot is controlled and moved so that the detection target is located within the work range (S204), the process returns to step S201, and the process is repeated.

  Accordingly, it is possible to determine whether or not the detection target is located within the work range based on the presence or absence of the marker shielding state by the detection target. As described above, since the marker is easier to recognize than the detection object, it is possible to more easily determine whether or not the detection object is located within the work range.

  Further, in FIGS. 5C and 5D described above, only the rightmost marker in the captured image is in the marker occlusion state. Therefore, when the robot is controlled as it is, the detection target is out of the work range. It is also possible to determine that there is a high possibility of exiting.

3. Second Embodiment Next, a second embodiment will be described with reference to FIGS. 7A and 7B, FIGS. 8A to 8C, and the flowchart of FIG. .

  As a premise, in this example, the work range is set as a range different from the imaging range. For example, as shown in FIG. 7A, when the imaging range captured by the imaging unit CM is PA, a work range WA is set in the imaging range PA. In the captured image IM shown in FIG. 7B captured by the imaging unit CM, the entire captured image IM corresponds to the imaging range PA, and the region ROI set therein corresponds to the work range WA. As will be described later, this region ROI is set as a region of interest in the captured image.

  Under the above premise, the captured image receiving unit 110 is an image obtained by imaging with the marker 50 placed outside the work range, with the marker 50 not placed in the work range (first area). An image is received (S301).

  For example, in this example, as shown in FIG. 7A, a marker is not arranged in the work area WA (and its background), but a plurality of markers MK are arranged in the background BG outside the work area WA. Then, imaging is performed by the imaging unit CM, and a captured image IM as illustrated in FIG. 7B is acquired. That is, no marker is arranged around the position where the work is performed by the visual servo, the marker is not reflected near the center of the captured image IM, and the marker is reflected near the end of the captured image IM.

  Thus, the processing unit 130 can perform the following determination process after performing the marker recognition process (S302).

  That is, when the processing unit 130 determines that at least a part of the area of the marker 50 is shielded by the detection target (work target, or in some cases, the end effector 310), that is, the marker that cannot be recognized. If it is determined that there is an object, it is determined that the detection object (work object, or in some cases, the end effector 310) is located outside the work range (first region) (S303).

  For example, in the captured image IM1 as shown in FIG. 8A, a marker shielding state occurs at the marker arranged at the position P1 due to the work object OB. Therefore, it is determined that the detection target is located outside the work range.

  Therefore, the robot is controlled and moved so that the detection target is positioned within the work range (S304), the process returns to step S301, and the process is repeated.

  On the other hand, when the processing unit 130 determines that at least a part of the region of the marker 50 is not shielded by the detection target (work target or the end effector 310 in some cases), that is, it cannot be recognized. If it is determined that there is no marker, it is determined that the detection target (work target, or in some cases, the end effector 310) is located within the work range (first region) (S303).

  For example, in the captured image IM2 as shown in FIG. 8B, the marker occlusion state does not occur in any marker due to the work object OB. Therefore, it is determined that the detection target is located within the work range. As described above, also in this example, only the work object OB is considered as the detection object, and the occurrence of the marker shielding state by the end effector EE is not considered.

  Therefore, in the case of FIG. 8B, since it is considered that visual servo can be normally performed, visual servo is performed based on the captured image (S305), and whether or not the visual servo has converged is determined. Determination is made (S306). If it is determined that the visual servo has converged, the process is terminated. On the other hand, if it is determined that the visual servo has not converged, the process returns to step S301 and the process is repeated.

  In step S305 described above, the processing unit 130 sets a region corresponding to the work range (first region) as a region of interest as a captured image, and performs visual servoing or inspection processing in the region of interest.

  For example, when the captured image IM2 as shown in FIG. 8B is acquired, visual servoing is performed based on an image obtained by cutting out the attention area ROI as shown in FIG. 8C.

  This makes it possible to perform visual servoing or inspection processing based on an image of a region where no marker is shown in the captured image.

  Conversely, when the entire captured image is viewed, the marker can be detected, so the visual servo does not go well, and it detects that the detection target is about to come out of the imaging range. be able to.

  As described above, in this example, when visual servoing or inspection processing is performed, it is possible to prevent a marker placed on the background from interfering with object detection processing. That is, in the object detection process, it is possible to prevent the detection target from being erroneously detected or the visual servo from not converging. In addition to this, it is possible to determine whether or not the detection object is located within the work range based on the presence or absence of the marker shielding state by the detection object.

4). Third Example In the third example, for example, markers (MK11 to MKij (i and j are integers of 2 or more)) having different patterns and colors are arranged in advance on the background BG shown in FIG. Keep it. In FIG. 10A, for convenience of illustration, each marker is indicated by a symbol. However, in actuality, markers with different patterns or colors as shown in FIGS. 3A to 3D described above are actually used. It is assumed that it is arranged.

  As shown in FIG. 10B, the captured image receiving unit 110 (CM) is obtained by arranging a plurality of different markers 50 (MK11 to MKij) in the background BG (second region). The captured image is received. In FIG. 10B, the details of the markers are omitted for the sake of illustration, but in reality, markers (MK11 to MKij) as shown in FIG. 10A are arranged on the background BG. It shall be.

  Thus, the processing unit 130 performs a marker recognition process, and it is possible to specify an area shown in the captured image based on the recognized marker type.

  In other words, the processing unit 130 can specify the imaging direction of the captured image based on the result of recognition processing of a plurality of different markers 50 in the captured image. Specifically, the processing unit 130 captures the captured image (the region reflected in the captured image, the captured image based on the type of the marker recognized in the marker recognition process, and the arrangement position of each marker determined and stored in advance. Range) can be specified.

  For example, in FIG. 10B, when the marker shown at the upper left corner of the captured image is MK11, the region reflected in the captured image can be specified as PA1, and the imaging direction can be specified as DR1. Further, when the marker shown at the upper left corner of the captured image is MK94, the area reflected in the captured image is PA2, and the imaging direction can be specified as DR2.

  Here, the imaging direction means a direction in which the camera captures an image, for example, the direction of the optical axis passing through the center point of the imaging lens of the camera.

  Thereby, it is possible to adjust the orientation (imaging direction) of the imaging unit 200 and the like.

  Further, the processing unit 130 can perform a process of specifying the position and orientation of the work object of the robot 300 or the end effector 310 of the robot 300 based on the recognition processing result of the plurality of markers 50 in the captured image. Note that the position and orientation refers to the position and orientation.

  For example, when a plurality of different markers (MK11 to MKij) are arranged on the background BG as shown in FIG. 10A, as a result of performing the marker recognition process, as shown in FIG. When only the marker in AR1 can be recognized, it can be specified that the detection object OB is located at a position corresponding to the area where the marker that could not be recognized is arranged.

  The region where the marker cannot be recognized in the captured image varies depending on the position and orientation of the detection target. For example, when the detection object OB is tilted as shown in FIG. 11B, only the markers arranged in the areas AR2 and AR3 can be recognized. Therefore, based on the type of marker that could not be recognized, it can be specified that the detection object OB is inclined and positioned as shown in FIG.

  Further, the region where the marker cannot be recognized in the captured image also varies depending on the position in the depth direction with respect to the imaging unit 200. Specifically, the region where the marker cannot be detected becomes wider when the detection target is located in front of the imaging unit 200. For example, when the detection object OB is positioned at the front as shown in FIG. 11C, only the marker arranged in the area AR4 can be recognized, and conversely, the detection object OB is as shown in FIG. 11D. If it is located in the back, only the marker placed in the area AR5 can be recognized. Therefore, the position of the detection target in the depth direction can also be specified.

  This makes it possible to compare the actual position and orientation of the detection object with the target position and orientation of the detection object. Then, for example, using this comparison result, it is possible to move the robot, check whether the result of the visual servo is as expected, or the like. That is, if you know in advance which of the multiple markers placed in the background the visual servo will converge when it can be recognized, whether the visual servo has converged based on the captured image It becomes possible to determine.

  Specifically, the flow of processing when the visual servo result inspection processing is performed will be described with reference to the flowchart of FIG.

  First, the processing unit 130 performs visual servoing (S401), and determines whether the visual servo has converged (S402). If it is determined that the visual servo has not converged, the process is repeated until the visual servo has converged.

  On the other hand, when it is determined that the visual servo has converged, the captured image reception unit 110 captures images by arranging a plurality of different markers 50 (MK11 to MKij) on the background BG, as shown in FIG. The captured image obtained in this way is acquired (S403).

  Then, the processing unit 130 performs a marker recognition process (S404). As a result of the marker recognition process, the marker placement position where the marker occlusion state occurs when the marker placement position that could not be recognized converges correctly in the visual servo. In step S405, an inspection process is performed to determine whether or not they match.

  In the inspection process, when it is determined that the marker arrangement position that could not be recognized matches the marker arrangement position where the marker occlusion state occurs when the visual servo converges correctly, the process ends. . On the other hand, if it is determined that the arrangement positions do not match, the process returns to step S401 to perform visual servo again. The above is the flow of a series of processes.

  In this example, as shown in FIG. 10A, the case where a plurality of different markers are arranged in the array has been described. However, when one marker can detect which region of the marker is shielded. Alternatively, the position and orientation of the detection target may be specified based on the marker shielding state of one marker.

  Further, the imaging unit (camera) 200 used in the above-described embodiment includes an imaging element such as a charge-coupled device (CCD) and an optical system. The imaging unit 200 is arranged at an angle such that the detection target (work target or the end effector 310 of the robot 300) in the visual servo is within the angle of view of the imaging unit 200, for example, on a ceiling or a work table. The Then, the imaging unit 200 outputs information of the captured image to the control device 100 or the like. However, in the present embodiment, the information of the captured image is output as it is to the control device 100, but the present invention is not limited to this. For example, the imaging unit 200 can include a device (processor) used for image processing or the like.

5. Robot System and Robot Next, FIGS. 13A and 13B show a configuration example of the robot system and the robot 300 to which the control device 100 of the present embodiment is applied. In both cases of FIG. 13A and FIG. 13B, the robot 300 includes an end effector 310. Further, the robot system includes a robot 300, a marker 50, and an imaging unit 200.

  The end effector 310 is a component attached to an end point of an arm in order to grip, lift, lift, attract, or process a workpiece (work object). The end effector 310 may be, for example, a hand (gripping unit), a hook, a suction cup, or the like. Further, a plurality of end effectors may be provided for one arm. The arm is a part of the robot 300 and refers to a movable part including one or more joints.

  For example, in the robot of FIG. 13A, the robot main body 300 (robot) and the control device 100 are configured separately. In this case, some or all of the functions of the control device 100 are realized by, for example, a PC (Personal Computer).

  Further, the robot according to the present embodiment is not limited to the configuration shown in FIG. 13A, and the robot body 300 and the control device 100 may be configured integrally as shown in FIG. 13B. That is, the robot 300 may include the control device 100. Specifically, as illustrated in FIG. 13B, the robot 300 includes a robot body (having an arm and an end effector 310) and a base unit portion that supports the robot body, and the control device 100 is included in the base unit portion. May be stored. The robot 300 in FIG. 13B has a structure in which wheels and the like are provided in the base unit portion so that the entire robot can move. Although FIG. 13A shows an example of a single arm type, the robot 300 may be a multi-arm type robot such as a double arm type as shown in FIG. 13B. The robot 300 may be moved manually, or may be moved by providing a motor for driving wheels and controlling the motor by the control device 100. Moreover, the control apparatus 100 is not necessarily provided in the base unit part provided under the robot 300 as shown in FIG.

  As illustrated in FIG. 14, the function of the control device 100 may be realized by a server 500 that is connected to the robot 300 via a network 400 including at least one of wired and wireless.

  Alternatively, in the present embodiment, a part of the processing of the control device of the present invention may be performed by the control device on the server 500 side. In this case, the processing is realized by distributed processing with the control device provided on the robot 300 side. The control device on the robot 300 side is realized by a terminal device 330 (control unit) installed in the robot 300, for example.

  In this case, the control device on the server 500 side performs a process assigned to the control device of the server 500 among the processes in the control device of the present invention. On the other hand, the control device provided in the robot 300 performs the process assigned to the control device of the robot 300 among the processes of the control device of the present invention. Each process of the control apparatus of the present invention may be a process assigned to the server 500 side or a process assigned to the robot 300 side.

  Thereby, for example, the server 500 having a higher processing capability than the terminal device 330 can perform processing with a large processing amount. Further, for example, the server 500 can collectively control the operations of the robots 300, and for example, it is easy to cause a plurality of robots 300 to perform cooperative operations.

  In recent years, the production of a small number of various types of parts has been increasing. And when changing the kind of components to manufacture, it is necessary to change the operation | movement which a robot performs. With the configuration as shown in FIG. 14, it is possible to change the operation performed by the robot 300 in a batch without the server 500 performing the teaching operation on each robot of the plurality of robots 300 again.

  Furthermore, with the configuration as shown in FIG. 14, it is possible to significantly reduce the trouble of performing software update of the control device 100 as compared with the case where one control device 100 is provided for each robot 300. become.

6). Visual Servo Here, the outline of visual servo, the flow of position-based visual servo, and the flow of feature-based visual servo will be described.

  The visual servo is a kind of servo system that tracks a target by measuring a change in the position of the target as visual information and using it as feedback information. Visual servo is roughly classified into position-based visual servo and feature-based visual servo according to input information (control amount) to the servo system. In the position-based visual servo, the position information and posture information of the object are input information to the servo system, and in the feature-based visual servo, the feature amount of the image is input information to the servo system. In addition, there is a method that hybridizes the position base and the feature base. The visual servo handled in the present invention covers all these methods.

  These visual servos are common in that the input information to the servo system is obtained based on the reference image and the captured image.

6.1 Flow of Position-Based Visual Servo First, the flow of position-based visual servo is shown in the flowchart of FIG. In the position-based visual servo, first, a reference image is set (S1). Here, the reference image is also referred to as a target image or a goal image, and is an image serving as a visual servo control target, which is an image representing the target state of the robot 300. That is, the reference image is an image that represents the target position or target posture of the robot 300 or an image that represents a state where the robot 300 is located at the target position. Also, the reference image must be prepared in advance and stored in the storage unit.

  Next, a work space is imaged by the imaging unit 200, and a captured image is acquired (S2). The captured image represents the current state of the work space. When the robot 300 and the workpiece are reflected in the captured image, the captured image represents the current state of the robot 300 and the workpiece. Although a processing delay occurs depending on the performance of the imaging unit 200, it is assumed here that the captured image shows the current state even when the processing delay occurs.

  For example, FIG. 16A shows a specific example of the reference image RIM, and FIG. 16B shows a specific example of the captured image PIM. In the captured image PIM, the robot RB points the arm AM and the hand HD (or end point EP) upward, but in the reference image RIM, the robot RB bends the arm AM and brings the hand HD closer to the work WK. . Therefore, in this specific example, the robot RB is controlled so that the arm AM of the robot RB is bent and the hand HD is brought close to the work WK.

  Next, a control command is generated (S3). For example, the control command generation is performed by using homography, which is one of coordinate transformations, based on the reference image and the captured image. In this case, a homography matrix is obtained, and a speed command is generated as a control signal for the robot 300 from the homography matrix.

  Here, the control signal (control command) refers to a signal including information for controlling the robot 300. For example, the control signal includes a speed command. The speed command refers to a command method for giving a moving speed or a rotating speed of an end point or the like of the arm of the robot 300 as information for controlling each part of the robot 300.

  Then, based on the generated control signal, it is determined whether or not the control amount (here, the position and posture of the robot 300) has converged to the target value (S4). For example, in the case of using homography, when the velocity vector represented by the velocity command is 0, it can be considered that the position or orientation that is the controlled variable has reached the target state. If the speed vector is not 0, it is determined that the control amount has not converged to the target value. Here, convergence means approaching the target value as much as possible, or reaching the target value.

  If it is determined that the control amount has converged to the target value, the visual servo is terminated. On the other hand, if it is determined that the control amount has not converged to the target value, the processing unit 130 sends a control command to the robot 300 (S5).

  In the position-based visual servo, the above processing is repeated until the control amount converges to the target value.

6.2 Flow of Feature-Based Visual Servo Next, the flow of feature-based visual servo is shown in the flowchart of FIG. As with the position-based visual servo, a reference image is first set (S10), and then the work space is imaged by the imaging unit 200 to obtain a captured image (S11). Note that when feature-based visual servoing is used, it is desirable to perform feature extraction of the reference image and calculate the feature amount when setting the reference image. Alternatively, reference image information obtained by extracting features of the reference image may be stored in the storage unit. The feature extraction of the image is performed using, for example, corner detection or a Gaussian filter.

  Next, the feature of the captured image is extracted (S12). Note that the feature extraction of the reference image is preferably performed at the time of setting the reference image, but may be performed in this step. In the feature extraction, an image feature amount is obtained as input information (control amount) to the visual servo system.

  Then, based on the feature amount of the image, it is compared whether or not the reference image matches the captured image (S13). If it is determined that the images match (S14), the visual servo is terminated. On the other hand, if it is determined that the images do not match (S14), a control command is generated (S15), and the control command is sent to the robot 300 (S16).

  In the feature-based visual servo, the above processing is repeated until the control amount converges to the target value.

  Note that a part or most of the processing of the control device, the robot system, the robot, and the like of the present embodiment may be realized by a program. In this case, a control device, a robot system, a robot, and the like according to the present embodiment are realized by a processor such as a CPU executing a program. Specifically, a program stored in the information storage medium is read, and a processor such as a CPU executes the read program. Here, the information storage medium (computer-readable medium) stores programs, data, and the like, and functions as an optical disk (DVD, CD, etc.), HDD (hard disk drive), or memory (card type). It can be realized by memory, ROM, etc. A processor such as a CPU performs various processes according to the present embodiment based on a program (data) stored in the information storage medium. That is, in the information storage medium, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit) Is memorized.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. Further, the configurations and operations of the control device, the robot system, and the robot are not limited to those described in the present embodiment, and various modifications can be made.

50 markers, 100 control device, 110 captured image reception unit,
130 processing unit, 200 imaging unit, 300 robot, 310 end effector,
330 terminal devices, 400 networks, 500 servers

Claims (15)

A captured image receiving unit that receives a captured image obtained by imaging the first region by the imaging unit;
A processing unit for operating a robot including an end effector operating in the first region;
Including
The processor is
A control device that determines whether or not the end effector is in the first region based on information of a marker reflected in the captured image. In claim 1,
The captured image reception unit
Receiving the captured image obtained by imaging at least one of the markers arranged in the second region;
The processor is
Based on the captured image, recognition processing of the marker arranged in the second area is performed, and based on the result of the recognition processing, whether or not the end effector is positioned in the first area. A control device that performs visual servoing or inspection processing by determining. In claim 2,
The processor is
By performing the recognition process of the marker, a marker shielding state in which at least a part of the marker is shielded by the robot work object or the end effector is detected, and the detected marker A control device that determines whether or not the end effector is located in the first region based on a shielding state. In claim 3,
The processor is
If it is determined that at least a partial area of the marker is shielded by the work object or the end effector, it is determined that the end effector is located within the first area;
When it is determined that at least a part of the area of the marker is not shielded by the work object or the end effector, it is determined that the end effector is located outside the first area. Control device. In any of claims 2 to 4,
The second region is
The control device, wherein the control device is a region located behind the first region from the position of the imaging unit toward the imaging direction of the captured image. In claim 3,
The captured image reception unit
The marker is non-arranged in the first region, and the captured image obtained by imaging by placing the marker in the second region outside the first region is received,
The processor is
If it is determined that at least a partial area of the marker is shielded by the work object or the end effector, it is determined that the end effector is located outside the first area;
When it is determined that at least a part of the area of the marker is not shielded by the work object or the end effector, it is determined that the end effector is located in the first area. Control device. In any one of Claims 2 thru | or 6.
The processor is
A control apparatus, wherein an area corresponding to the first area is set as an attention area in the captured image, and the visual servo or the inspection process is performed in the attention area. In any one of Claims 2 thru | or 7,
The captured image reception unit
A control apparatus that receives the captured image obtained by imaging a plurality of markers arranged in an array in the second region. In any of claims 2 to 8,
The captured image reception unit
A control apparatus that receives the captured image obtained by imaging a plurality of different markers arranged in the second region. In claim 9,
The processor is
A control device that identifies an imaging direction of the captured image based on a result of the recognition processing of the plurality of different markers in the captured image. In any one of Claims 8 thru | or 10.
The processor is
A control device that performs a process of specifying a position and orientation of a work object of the robot or the end effector of the robot based on a result of the recognition process of the plurality of markers in the captured image. In any of claims 2 to 11,
The processor is
A control apparatus that performs processing for controlling the robot so that the captured image matches or approaches a reference image as the visual servo. A robot comprising an end effector operating in a first region;
Markers,
An imaging unit for imaging the first region;
Including
The robot is
A robot system that determines whether or not the end effector is in the first region based on information of the marker reflected in a captured image captured by the imaging unit. Including an end effector operating in a first region;
A robot that determines whether or not the end effector is in the first region based on information of a marker reflected in a captured image captured by an imaging unit. A robot control method for operating a robot having an end effector operating in a first region,
Imaging the marker;
Determining whether the end effector is in the first region based on information of a marker reflected in a captured image;
A robot control method including:

Images