CN112489133A

CN112489133A - Calibration method, device and equipment of hand-eye system

Info

Publication number: CN112489133A
Application number: CN202011287100.9A
Authority: CN
Inventors: 郭俊峰; 刘伟峰; 单纪昌; 刘旭
Original assignee: Beijing Jingdong Qianshi Technology Co Ltd
Current assignee: Beijing Jingdong Qianshi Technology Co Ltd
Priority date: 2020-11-17
Filing date: 2020-11-17
Publication date: 2021-03-12
Anticipated expiration: 2040-11-17
Also published as: CN112489133B

Abstract

The embodiment of the application provides a calibration method, a calibration device and calibration equipment for a hand-eye system, wherein the hand-eye system comprises a mechanical arm and a camera, the mechanical arm comprises N shafts, a calibration plate is arranged at the tail end of the K-th shaft of the mechanical arm, and the calibration method comprises the following steps: and controlling the tail end of the Nth shaft of the mechanical arm to run to a preset inspection pose, shooting the calibration board through a camera to obtain a first calibration board image, acquiring first calibration information according to the first calibration board image, and if the error between the first calibration information and second calibration information stored in a preset storage space is larger than a preset threshold value, performing first calibration processing on the hand-eye system. Above-mentioned in-process, through setting up the calibration board at the K axle is terminal, avoided the operation of dismantling repeatedly when the calibration, improved calibration efficiency.

Description

Calibration method, device and equipment of hand-eye system

Technical Field

The embodiment of the application relates to the technical field of computer vision, in particular to a calibration method, a calibration device and calibration equipment for a hand-eye system.

Background

The hand-eye system is widely applied to various scenes such as logistics, storage, sorting, unstacking and the like. The hand-eye system is a robot vision system consisting of a camera and a mechanical arm. The end of the robotic arm is provided with an end effector (e.g., a gripper or suction cup) for moving or picking objects. The camera can provide visual information for guiding the motion track of the tail end of the mechanical arm.

In practical application, the hand-eye system needs to be calibrated by using the calibration plate to determine the transformation relationship between the mechanical arm coordinate system and the camera coordinate system. Taking a six-axis robot as an example, an end effector is mounted at the end of the sixth axis of the robot. The existing hand-eye calibration process is as follows: and disassembling the end effector arranged at the tail end of the sixth shaft of the mechanical arm, and arranging the calibration plate at the tail end of the sixth shaft of the mechanical arm. Then, after calibration is completed, the calibration plate is removed and the end effector is remounted to the end of the sixth axis of the robotic arm.

In the process of implementing the present application, the inventor finds that at least the following problems exist in the prior art: the operation flow of the hand-eye calibration mode is complex, and the efficiency is low.

Disclosure of Invention

The embodiment of the application provides a calibration method, a calibration device and calibration equipment of a hand-eye system, which are used for improving the calibration efficiency of the hand-eye system.

In a first aspect, an embodiment of the present application provides a calibration method for a hand-eye system, where the hand-eye system includes a mechanical arm and a camera, the mechanical arm includes N axes, a calibration plate is disposed at a K-th axis end of the mechanical arm, and K is a natural number less than N, and the method includes:

controlling the tail end of the Nth shaft of the mechanical arm to run to a preset inspection pose;

shooting the calibration plate through the camera to obtain a first calibration plate image, and acquiring first calibration information according to the first calibration plate image;

and if the error between the first calibration information and the second calibration information stored in the preset storage space is larger than a preset threshold value, performing first calibration processing on the hand-eye system.

In a possible implementation, obtaining first calibration information according to the first calibration plate image includes:

acquiring the pose of the end of the K axis according to the pose of the end of the N axis;

and acquiring the first calibration information according to the pose of the end of the K-th axis and the first calibration plate image.

In a possible implementation, the first calibration process is performed on the hand-eye system, and includes:

determining a plurality of calibration poses corresponding to the end of the Kth axis;

controlling the tail end of the K-th shaft to run to each calibration pose, and shooting the calibration plate through the camera to obtain a second calibration plate image when the tail end of the K-th shaft runs to each calibration pose;

and obtaining a calibration result according to the plurality of calibration poses and the second calibration plate image acquired when the end of the K-th axis runs to each calibration pose.

In a possible implementation manner, determining a plurality of calibration poses corresponding to the K-th axis end includes:

and determining a plurality of calibration poses corresponding to the end of the K axis according to the visual field range of the camera and the current pose of the end of the K axis.

In a possible implementation manner, determining a plurality of calibration poses corresponding to the K-th axis end according to the visual field range of the camera and the current pose of the K-th axis end includes:

acquiring the current pose of the tail end of the Nth axis, and determining the current pose of the tail end of the K-th axis according to the current pose of the tail end of the Nth axis;

and randomly adjusting the current pose of the end of the K-th axis in the visual field range of the camera to generate a plurality of calibration poses corresponding to the end of the K-th axis.

In a possible embodiment, controlling the K-th axis end to run to each of the calibration poses includes:

determining a target pose of the tail end of the Nth axis according to each calibration pose corresponding to the tail end of the Kth axis;

and controlling the end of the Nth axis to run to the target pose.

In a possible embodiment, before controlling the nth axis end of the mechanical arm to move to the preset inspection position, the method further includes:

after the hand-eye system is subjected to second calibration processing, controlling the tail end of the Nth axis to run to the preset inspection pose;

shooting the calibration plate through the camera to obtain a third calibration plate image, and acquiring second calibration information according to the third calibration plate image;

and storing the second calibration information to the preset storage space.

In a possible embodiment, the robot arm comprises six axes, the calibration plate being arranged at the end of the fourth axis of the robot arm.

In a second aspect, an embodiment of the present application provides a calibration device for a hand-eye system, the hand-eye system includes a mechanical arm and a camera, the mechanical arm includes N axles, a calibration plate is disposed at a K-th axle end of the mechanical arm, K is a natural number less than N, the device includes:

the inspection module is used for controlling the tail end of the Nth shaft of the mechanical arm to run to a preset inspection pose;

the inspection module is further used for shooting the calibration plate through the camera to obtain a first calibration plate image and obtaining first calibration information according to the first calibration plate image;

and the calibration module is used for performing first calibration processing on the hand-eye system if the error between the first calibration information and second calibration information stored in a preset storage space is greater than a preset threshold value.

In a possible implementation, the verification module is specifically configured to:

In a possible implementation manner, the calibration module is specifically configured to:

and controlling the end of the Nth axis to run to the target pose.

In one possible embodiment, the verification module is further configured to:

and storing the second calibration information to the preset storage space.

In a third aspect, an embodiment of the present application provides an electronic device, including: a memory for storing a computer program and a processor for executing the computer program to perform the method according to any of the first aspect.

In a fourth aspect, the present application provides a computer-readable storage medium including a computer program, which when executed by a processor implements the method according to any one of the first aspect.

The embodiment of the application provides a calibration method, a calibration device and calibration equipment for a hand-eye system, wherein the hand-eye system comprises a mechanical arm and a camera, the mechanical arm comprises N shafts, a calibration plate is arranged at the tail end of the K-th shaft of the mechanical arm, and the calibration method comprises the following steps: and controlling the tail end of the Nth shaft of the mechanical arm to run to a preset inspection pose, shooting the calibration board through a camera to obtain a first calibration board image, acquiring first calibration information according to the first calibration board image, and if the error between the first calibration information and second calibration information stored in a preset storage space is larger than a preset threshold value, performing first calibration processing on the hand-eye system. Above-mentioned in-process, through setting up the calibration board at the K axle is terminal, avoided the operation of dismantling repeatedly when the calibration, improved calibration efficiency. Furthermore, the calibration plate can be installed on the mechanical arm for a long time, so that calibration information of the hand-eye system can be automatically checked conveniently, whether the error is increased due to long-term operation of the calibration result is determined, and whether the hand-eye calibration is carried out again is quickly determined. In addition, through setting up the calibration plate at the K axle is terminal, can also reduce the probability that calibration plate and arm body bump.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

FIG. 1 is a schematic diagram of an architecture of a hand-eye system according to an embodiment of the present application;

FIG. 2 is a schematic diagram of an operation process of hand-eye calibration in the prior art;

FIG. 3 is a schematic diagram of a hand-eye system provided by an embodiment of the present application;

fig. 4 is a schematic flowchart of a calibration method for a hand-eye system according to an embodiment of the present application;

fig. 5 is a schematic flowchart of a calibration method for a hand-eye system according to another embodiment of the present application;

FIG. 6 is a schematic flow chart illustrating a calibration process for an eye system according to another embodiment of the present application;

fig. 7 is a schematic structural diagram of a calibration apparatus of a hand-eye system according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

With the above figures, there are shown specific embodiments of the present application, which will be described in more detail below. These drawings and written description are not intended to limit the scope of the inventive concepts in any manner, but rather to illustrate the inventive concepts to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

First, terms related to embodiments of the present application will be explained:

calibrating the hands and eyes: and calibrating the external parameter of the camera relative to the calibration plate when the tail end of the mechanical arm is in different poses, and calculating the external parameter of the camera relative to the tail end of the mechanical arm according to the external parameter of the camera relative to the calibration plate and the pose of the tail end of the mechanical arm.

An end effector: the end effector of the robotic arm is typically a hand grip or suction cup.

Positive kinematic equation: the pose of the tail end of the mechanical arm is calculated according to the axis angle of the mechanical arm, and the poses of the K-th axis and the N-th axis are calculated respectively in the embodiment.

Inverse kinematics equation: the pose calculation method is an equation for calculating the axis angle of the mechanical arm according to the pose of the tail end of the mechanical arm, and in this embodiment, the pose calculation is based on the K-th axis and the N-th axis respectively.

The hand-eye system is a robot vision system consisting of a camera and a mechanical arm. The end of the robotic arm is provided with an end effector (e.g., a gripper or suction cup) for moving or picking objects. The camera can provide visual information for guiding the motion track of the tail end of the mechanical arm.

In some hand-eye systems, the camera may be mounted at the end of the arm and follow the end of the arm. This scene may also be referred to as an Eye-in-Hand (Eye-in-Hand) scene. In this scenario, the camera coordinate system is always changed relative to the robot arm coordinate system, while the pose of the camera relative to the robot arm tip is constant.

In other hand-eye systems, the camera may be mounted in a fixed position outside the mechanical arm. This scene may also be referred to as an Eye-to-Hand scene. In this scenario, the camera coordinate system is constant with respect to the robot arm coordinate system, while the pose of the camera with respect to the robot arm tip always changes.

The embodiment of the application is suitable for Eye-to-Hand (Eye-to-Hand) scenes. The architecture of the hand-eye system is described below in connection with fig. 1.

Fig. 1 is a schematic diagram of an architecture of a hand-eye system in an embodiment of the present application. As shown in fig. 1, the hand-eye system includes: a robotic arm, a camera, and a controller. The robot arm may also be referred to as a robot arm. The robotic arm is typically a multi-axis robotic arm, such as: triaxial arm, four-axis arm, five arms, six arms etc.. The more the number of axes of the robot arm is, the more the degree of freedom thereof is, and the more flexible the operation can be performed. Fig. 1 illustrates a six-axis robot arm as an example. Referring to fig. 1, the sixth axis of the robot arm is terminated by an end effector (schematically illustrated as a circle in fig. 1). The end effector may be a hand grip, suction cup, or other actuator.

In the hand-eye system shown in fig. 1, the robot arm corresponds to a hand of the robot, and the camera corresponds to an eye of the robot. The mechanical arm is in communication connection with the controller. The camera is in communication with the controller. After the camera acquires the image including the target object, the image is sent to the controller. And the controller calculates and obtains the position information of the target object in the mechanical arm coordinate system by utilizing the conversion relation between the camera coordinate system and the mechanical arm coordinate system. Furthermore, the controller can calculate the motion track of the tail end of the mechanical arm according to the position information of the target object in the coordinate system of the mechanical arm, and control the tail end of the mechanical arm to reach a specific position, so that the end picking device can grab the target object.

In practical application, the hand-eye system needs to be calibrated by using the calibration plate. The purpose of hand-eye calibration is to obtain the transformation relation between the camera coordinate system and the mechanical arm coordinate system. In the prior art, when the hand-eye calibration is performed, a calibration plate is usually installed at the end of the sixth axis of the mechanical arm. The principle of its calibration is as follows.

Assuming that a transformation relation between a mechanical arm coordinate system and a camera coordinate system is a matrix X, a transformation relation between the mechanical arm coordinate system and a mechanical arm tail end mechanism is a matrix A, a transformation relation between the mechanical arm tail end mechanism and a calibration plate is a matrix Z, and a transformation relation between a camera and the calibration plate is a matrix B. The mechanical arm controls the calibration plate to reach n calibration positions, and the poses of the tail end of the mechanical arm are A respectively at the n positions₁，...，A_nThe pose of the calibration plate relative to the camera is B₁，...，B_n. Wherein A is_i、B_iX and Z are known as fixed values. From this the following equation can be derived:

from the above equation, by eliminating Z, one can derive:

and combining and solving the equations to obtain X, so as to obtain the transformation relation between the mechanical arm coordinate system and the camera coordinate system.

Although the existing hand-eye calibration mode can realize hand-eye calibration, the operation process is complex and the efficiency is low. The operation flow of the conventional hand-eye calibration is described below with reference to fig. 2.

Fig. 2 is a schematic operation flow diagram of hand-eye calibration in the prior art. As shown in fig. 2, during normal operation of the hand-eye system, it is necessary to install an end effector at the end of the sixth axis of the robot arm. When hand eye calibration is needed, the end picking device arranged at the tail end of the sixth shaft of the mechanical arm needs to be detached, and the calibration plate is arranged at the tail end of the sixth shaft of the mechanical arm. Then, after calibration is completed, the calibration plate is removed and the end effector is remounted to the end of the sixth axis of the robotic arm. Therefore, the hand-eye calibration process needs manual disassembly and assembly for many times, the operation flow is complex, and the calibration efficiency is low. Furthermore, in some application scenarios, when calibration information of the hand-eye system needs to be quickly checked, the above installation and disassembly processes need to be repeated to check whether the calibration information is accurate, and the checking efficiency is low.

In addition, when the calibration plate is installed at the end of the sixth shaft of the mechanical arm, the gesture at the end of the sixth shaft is flexible and changeable, so that the calibration plate and the mechanical arm body are easy to collide, and therefore the size of the calibration plate and the motion space of the mechanical arm are limited.

In order to solve at least one of the above-mentioned technical problems, the embodiments of the present application improve the mounting position of the calibration plate, which is fixedly mounted on a different shaft from the end effector. The hand-eye system of the present embodiment includes a robot arm and a camera. Wherein the camera is arranged in a fixed position outside the robot arm. The arm includes N axle, and the Nth axle end of arm is provided with the end effector, and the Kth axle end of arm is provided with the calibration board. K is a natural number less than N. Fig. 3 is a schematic diagram of a hand-eye system according to an embodiment of the present disclosure. As shown in fig. 3, taking a six-axis robot arm as an example, the sixth axis of the robot arm is provided with an end effector (indicated by a circle in fig. 3), and the calibration plate may be provided on any other axis than the sixth axis.

Alternatively, referring to fig. 3, the calibration plate may be provided at the end of the fourth shaft of the robot arm. Because the terminal gesture of fourth shaft of arm is comparatively stable, will calibrate the board setting at the fourth shaft end, can reduce the probability that calibrates board and arm body bump.

In this embodiment, because calibration plate and end effector are installed on the disalignment, and the two mutually noninterfere, when hand eye calibration, need not to dismantle a series of operations such as end effector, installation calibration plate, dismantlement calibration plate, installation end effector, improve hand eye calibration efficiency. Furthermore, in the embodiment of the application, the calibration board can be installed on the mechanical arm for a long time, so that calibration information of the hand-eye system can be conveniently and automatically checked, whether the error is increased due to long-term operation is determined, and whether the hand-eye calibration needs to be carried out again is quickly determined.

The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

Fig. 4 is a schematic flowchart of a calibration method of a hand-eye system according to an embodiment of the present application. The method of this embodiment may be performed by a controller and may be used to calibrate the hand-eye system shown in fig. 3. As shown in fig. 4, the method of this embodiment may include:

s401: and controlling the tail end of the Nth shaft of the mechanical arm to run to a preset inspection pose.

S402: shooting a calibration plate through a camera to obtain a first calibration plate image, and acquiring first calibration information according to the first calibration plate image;

s403: and if the error between the first calibration information and the second calibration information stored in the preset storage space is larger than a preset threshold value, performing first calibration processing on the hand-eye system.

The embodiment can be applied to the scene of automatically checking the calibration information of the hand-eye system to determine whether the calibration processing needs to be carried out again on the hand-eye system.

The preset inspection pose refers to a pose which is designated in advance and used for inspecting calibration information of the hand-eye system.

Specifically, before the embodiment is executed, the hand-eye system is calibrated. And after calibration is finished, controlling the tail end of the Nth shaft of the mechanical arm to run to the preset inspection pose, then shooting the calibration plate through a camera to obtain a calibration plate image, and acquiring second calibration information according to the calibration plate image. And storing the second calibration information to a preset storage space. The second calibration information indicates position information of the calibration plate in the camera coordinate system when the tail end of the Nth axis of the mechanical arm runs to the preset inspection pose after the current calibration is completed.

In this embodiment, when calibration information of the hand-eye system needs to be checked, the end of the nth axis of the mechanical arm can be controlled to run to the preset checking pose, the calibration plate is shot by the camera to obtain a first calibration plate image, and the first calibration information is obtained according to the first calibration plate image. The first calibration information indicates the position information of the calibration plate in the camera coordinate system under the current scene.

Furthermore, whether recalibration is needed or not can be determined by comparing the first calibration information with second calibration information stored in a preset storage space. Specifically, if the error between the first calibration information and the second calibration information is greater than the preset threshold, it indicates that the calibration error of the hand-eye system is increased due to long-term operation, and therefore, the hand-eye system is calibrated again. If the error between the first calibration information and the second calibration information is smaller than or equal to the preset threshold, it indicates that the error between the current calibration information of the hand-eye system and the calibration information after the last calibration is completed is not large, and the calibration process does not need to be performed again under the condition.

It should be noted that, the implementation process of the calibration process is not limited in this embodiment, and a possible calibration processing manner may refer to the detailed description of the following embodiments.

Optionally, the preset inspection poses in the present embodiment may include one or more inspection poses. When the calibration board comprises a plurality of inspection poses, aiming at each inspection pose, when the tail end of the Nth shaft of the mechanical arm is controlled to run to the inspection pose, the camera shoots the calibration board to obtain a first calibration board image, and first calibration information is obtained according to the first calibration board image. Therefore, after traversing the plurality of inspection poses, the first calibration information corresponding to the plurality of inspection poses respectively can be obtained.

Correspondingly, second calibration information corresponding to the plurality of detection poses is stored in the preset storage space. When the comparison is performed, the first calibration information and the second calibration information corresponding to each inspection pose can be compared to determine an error value corresponding to the inspection pose. And finally, summing or averaging a plurality of error values corresponding to a plurality of checking poses, and if the calculation result is greater than a preset threshold value, performing recalibration processing on the hand-eye system.

Optionally, the preset inspection poses include five inspection poses, four of the five inspection poses correspond to four vertices of the square, and the other inspection pose corresponds to a center point of the square. Therefore, the uniformity of the distribution of the inspection poses is ensured, and the accuracy of the inspection result is further ensured.

The calibration method for the hand-eye system provided by the embodiment includes that the hand-eye system includes a mechanical arm and a camera, the mechanical arm includes N axes, and a calibration plate is disposed at the end of the K-th axis of the mechanical arm, and the calibration method includes: and controlling the tail end of the Nth shaft of the mechanical arm to run to a preset inspection pose, shooting the calibration board through a camera to obtain a first calibration board image, acquiring first calibration information according to the first calibration board image, and if the error between the first calibration information and second calibration information stored in a preset storage space is larger than a preset threshold value, performing first calibration processing on the hand-eye system. Above-mentioned in-process, through setting up the calibration board at the K axle is terminal, avoided the operation of dismantling repeatedly when the calibration, improved calibration efficiency. Furthermore, the calibration plate can be installed on the mechanical arm for a long time, so that calibration information of the hand-eye system can be automatically checked conveniently, whether the error is increased due to long-term operation of the calibration result is determined, and whether the hand-eye calibration is carried out again is quickly determined. In addition, through setting up the calibration plate at the K axle is terminal, can also reduce the probability that calibration plate and arm body bump.

On the basis of the above embodiment, the calibration process of the hand-eye system is described in detail below with reference to a specific example.

In this embodiment, it is assumed that five inspection poses are determined after a certain calibration process is completed. Illustratively, the four corners and the exact center of a square are determined, and a total of five points are taken as five inspection poses corresponding to the end of the Nth axis. The camera is turned on to ensure that the calibration plates are all in the camera field of view when the end of the nth axis is run into these five inspection poses.

Controlling the tail end of the Nth shaft to move to a first inspection pose, and calculating the shaft angle of the mechanical arm through an inverse kinematics equation according to the pose of the tail end of the Nth shaft; and calculating the pose of the end of the K-th axis through a positive kinematics equation according to the axis angle of the mechanical arm. And shooting the calibration plate through a camera to obtain a calibration plate image, and determining second calibration information corresponding to the first inspection pose according to the calibration plate image and the pose of the K-th shaft tail end.

Controlling the tail end of the Nth shaft to move to a second inspection pose, and calculating the shaft angle of the mechanical arm through an inverse kinematics equation according to the pose of the tail end of the Nth shaft; and calculating the pose of the end of the K-th axis through a positive kinematics equation according to the axis angle of the mechanical arm. And shooting the calibration plate through a camera to obtain a calibration plate image, and determining second calibration information corresponding to a second inspection pose according to the calibration plate image and the pose of the K-th shaft tail end.

Controlling the tail end of the Nth axis to run to a third inspection pose, and determining second calibration information corresponding to the third inspection pose by adopting the similar process; controlling the tail end of the Nth axis to run to a fourth inspection pose, and determining second calibration information corresponding to the fourth inspection pose by adopting the similar process; and controlling the tail end of the Nth axis to run to a fifth inspection pose, and determining second calibration information corresponding to the fifth inspection pose by adopting the similar process.

And storing the second calibration information corresponding to the five inspection poses into a preset storage space.

When the calibration information of the hand-eye system needs to be checked, for example, when the hand-eye system runs for a long time or the current calibration information of the hand-eye system is suspected to be inaccurate according to the running condition of the hand-eye system, the hand-eye system may be automatically checked and calibrated by using the flow shown in fig. 5.

Fig. 5 is a schematic flowchart of a calibration method for a hand-eye system according to another embodiment of the present application. As shown in fig. 5, the method of the present embodiment includes:

s501: and acquiring five preset checking poses.

The five checking poses are corresponding to four corners and the center of the square.

S502: one non-checked checking position is taken out from the five checking positions.

S503: and controlling the tail end of the Nth axis to run to the checking pose.

S504: and acquiring the pose of the end of the K-th axis, shooting the calibration plate through a camera to obtain a calibration plate image, and acquiring first calibration information corresponding to the checking pose according to the calibration plate image and the pose of the end of the K-th axis.

Specifically, the pose of the end of the K-th axis may be acquired as follows: calculating an axis angle of the mechanical arm through an inverse kinematics equation according to the pose of the tail end of the Nth axis; and calculating the pose of the end of the K-th axis through a positive kinematics equation according to the axis angle of the mechanical arm.

S505: and comparing the first calibration information corresponding to the checking pose with second calibration information corresponding to the checking pose stored in a preset storage space, and determining an error value corresponding to the checking pose.

S506: and judging whether the five checking poses are traversed or not. If so, go to step S507, otherwise, go back to step S502.

S507: and obtaining the inspection error of the hand-eye system according to the error values corresponding to the five inspection poses.

For example, the sum of the error values corresponding to the five inspection poses may be used as the inspection error, and the average of the error values corresponding to the five inspection poses may also be used as the inspection error.

S508: and judging whether the detection error is larger than a preset threshold value or not.

If yes, go to S509. If not, the calibration process is not needed to be carried out again, and the last calibration result can be continuously used.

S509: and carrying out calibration processing on the hand eye system.

Through the embodiment, the calibration error of the hand-eye system can be automatically checked, and the hand-eye system is automatically calibrated when the calibration error is larger, so that the calibration efficiency of the hand-eye system is improved.

Based on any of the above embodiments, the calibration process of the hand-eye system is described below with reference to fig. 6.

Fig. 6 is a schematic flowchart of a calibration process performed on an ocular system according to another embodiment of the present application. As shown in fig. 6, the method of the present embodiment includes:

s601: and determining a plurality of calibration poses corresponding to the end of the Kth axis.

Before S601, the camera is turned on, and the robot arm is controlled to operate such that the calibration plate of the K-th axis of the robot arm is positioned substantially at the center of the field of view of the camera.

Optionally, a plurality of calibration poses corresponding to the K-th axis end may be determined according to the field of view of the camera and the current pose of the K-th axis end. Illustratively, a plurality of calibration poses can be obtained by adjusting the current pose of the end of the K-th axis, and the obtained calibration poses are ensured not to exceed the visual field range of the camera.

In a possible implementation manner, the current pose of the end of the nth axis is obtained, the current pose of the end of the K axis is determined according to the current pose of the end of the nth axis, the current pose of the end of the K axis is randomly adjusted within the visual field range of the camera, and a plurality of calibration poses corresponding to the end of the K axis are generated.

S602: and controlling the tail end of the K-th shaft to run to each calibration pose, and shooting the calibration plate through the camera to obtain a second calibration plate image when the tail end of the K-th shaft runs to each calibration pose.

Optionally, the following method may be adopted to control the K-th axis end to run to each calibration pose: and determining a target pose of the Nth axis end according to each calibration pose corresponding to the Kth axis end, and controlling the Nth axis end to run to the target pose so that the Kth axis end runs to the calibration pose.

S603: and obtaining a calibration result according to the plurality of calibration poses and a second calibration plate image acquired when the end of the K-th axis runs to each calibration pose.

And calculating each calibration pose and the second calibration plate image corresponding to the calibration pose to obtain a calibration result corresponding to the calibration pose. And then, obtaining a calibration result of the current calibration processing according to the calibration results corresponding to the plurality of calibration poses.

Optionally, after the calibration result is obtained, it may be determined whether the calibration result is within the standard error range. If yes, determining that the calibration result is available. If not, determining that the calibration result is unavailable, and needing to be calibrated again.

In one example, when the eye system is determined to need to be calibrated, the current pose of the nth axis of the mechanical arm is obtained, the axis angles of the N axes of the mechanical arm are calculated through an inverse kinematics equation, and the pose of the end of the kth axis is calculated according to a positive kinematics equation and the axis angles of the N axes. And randomly adjusting the pose of the end of the K-th axis in the visual field range according to the visual field range of the camera to obtain the calibration pose corresponding to the end of the K-th axis. And according to the calibration pose corresponding to the end of the K-th axis, calculating the axis angles of the N axes of the mechanical arm through an inverse kinematics equation, and calculating the target pose corresponding to the end of the N-th axis of the mechanical arm through a positive kinematics equation and the axis angles of the N axes. And controlling the tail end of the Nth shaft of the mechanical arm to run to the target pose, shooting the calibration plate through the camera to obtain a calibration plate image, and storing the calibration plate image and the current calibration pose of the tail end of the Kth shaft.

And circularly executing the process until the preset circulation times are reached. And after the circulation is finished, acquiring a plurality of stored calibration poses and a plurality of calibration plate images, and calculating to obtain a calibration result. And if the calibration result is within the standard error range and represents that the calibration result is available, storing the calibration result.

When a subsequent hand-eye system runs, the position of the target object in the coordinate system of the mechanical arm is determined according to the image of the target object shot by the camera and the calibration result, and then the running track of the mechanical arm is determined, so that the end effector at the tail end of the mechanical arm can grab the target object.

Through the embodiment, the hand-eye system can be automatically calibrated without manually adjusting the pose of the mechanical arm by a user, so that the calibration efficiency of the hand-eye system is improved.

Fig. 7 is a schematic structural diagram of a calibration device of a hand-eye system according to an embodiment of the present application, where the calibration device of the hand-eye system according to the present embodiment may be in the form of software and/or hardware, and the calibration device may be disposed in the controller in fig. 1. As shown in fig. 7, the calibration apparatus 10 for a hand-eye system provided in this embodiment may include: a verification module 11 and a calibration module 12.

The inspection module 11 is used for controlling the tail end of the Nth shaft of the mechanical arm to run to a preset inspection pose;

the inspection module 11 is further configured to capture the calibration board through the camera to obtain a first calibration board image, and obtain first calibration information according to the first calibration board image;

a calibration module 12, configured to perform a first calibration process on the hand-eye system if an error between the first calibration information and second calibration information stored in a preset storage space is greater than a preset threshold.

In a possible embodiment, the checking module 11 is specifically configured to:

In a possible implementation, the calibration module 12 is specifically configured to:

and controlling the end of the Nth axis to run to the target pose.

In a possible embodiment, the verification module 11 is further configured to:

and storing the second calibration information to the preset storage space.

The calibration device for a hand-eye system provided in this embodiment may be used to implement the technical solution in any of the above method embodiments, and the implementation principle and the technical effect are similar, which are not described herein again.

Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device may act as a controller. As shown in fig. 8, the electronic device 20 of the present embodiment includes: a processor 21 and a memory 22; a memory 22 for storing a computer program; the processor 21 is configured to execute the computer program stored in the memory to implement the calibration method of the hand-eye system in the above embodiments. Reference may be made in particular to the description relating to the method embodiments described above.

Alternatively, the memory 22 may be separate or integrated with the processor 21.

When the memory 22 is a device independent from the processor 21, the electronic device 20 may further include: a bus 23 for connecting the memory 22 and the processor 21.

Optionally, the electronic device 20 may further include a communication component 24 for communicating with a robotic arm, camera, or the like.

The electronic device provided in this embodiment may be configured to execute the technical solution in any of the method embodiments, and the implementation principle and the technical effect are similar, which are not described herein again.

An embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium includes a computer program, and the computer program is used to implement a technical solution in any one of the above method embodiments.

An embodiment of the present application further provides a chip, including: the system comprises a memory, a processor and a computer program, wherein the computer program is stored in the memory, and the processor runs the computer program to execute the technical scheme of any one of the method embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules is only one logical division, and other divisions may be realized in practice, for example, a plurality of modules may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.

The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In addition, functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may exist alone physically, or two or more modules are integrated into one unit. The unit formed by the modules can be realized in a hardware form, and can also be realized in a form of hardware and a software functional unit.

The integrated module implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods according to the embodiments of the present application.

It should be understood that the Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in the incorporated application may be directly implemented by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

The memory may comprise a high-speed RAM memory, and may further comprise a non-volatile storage NVM, such as at least one disk memory, and may also be a usb disk, a removable hard disk, a read-only memory, a magnetic or optical disk, etc.

The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the buses in the figures of the present application are not limited to only one bus or one type of bus.

The storage medium may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be integral to the processor. The processor and the storage medium may reside in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the storage medium may reside as discrete components in an electronic device or host device.

Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. A calibration method of a hand-eye system is characterized in that the hand-eye system comprises a mechanical arm and a camera, the mechanical arm comprises N shafts, a calibration plate is arranged at the tail end of the Kth shaft of the mechanical arm, K is a natural number smaller than N, and the method comprises the following steps:

2. The method of claim 1, wherein obtaining first calibration information from the first calibration plate image comprises:

3. The method according to claim 1 or 2, wherein the first calibration process for the hand-eye system comprises:

4. The method of claim 3, wherein determining a plurality of calibration poses corresponding to the K-th axis end comprises:

5. The method according to claim 4, wherein determining a plurality of calibration poses corresponding to the K-th axis end according to the visual field range of the camera and the current pose of the K-th axis end comprises:

6. The method of claim 3, wherein controlling the K-th axis extremity to travel to each of the calibration poses comprises:

and controlling the end of the Nth axis to run to the target pose.

7. The method according to any one of claims 1 to 6, wherein controlling the Nth axis tip of the robot arm before the operation to the preset inspection posture further comprises:

and storing the second calibration information to the preset storage space.

8. The method of any one of claims 1 to 7, wherein the robotic arm comprises six axes, and the calibration plate is disposed at a fourth axis end of the robotic arm.

9. The utility model provides a calibration device of hand eye system, its characterized in that, hand eye system includes arm and camera, the arm includes N axle, the K axle end of arm is provided with calibration plate, and K is the natural number that is less than N, the device includes:

10. An electronic device, comprising: a memory for storing a computer program and a processor for executing the computer program to perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a computer program which, when executed by a processor, implements the method of any one of claims 1 to 8.