CN115250745A - Full-automatic fruit picking robot and picking method based on vision technology - Google Patents
Full-automatic fruit picking robot and picking method based on vision technology Download PDFInfo
- Publication number
- CN115250745A CN115250745A CN202210936842.2A CN202210936842A CN115250745A CN 115250745 A CN115250745 A CN 115250745A CN 202210936842 A CN202210936842 A CN 202210936842A CN 115250745 A CN115250745 A CN 115250745A
- Authority
- CN
- China
- Prior art keywords
- fruit
- mechanical arm
- fruits
- clamping jaw
- bionic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 235000013399 edible fruits Nutrition 0.000 title claims abstract description 189
- 238000000034 method Methods 0.000 title claims abstract description 20
- 238000005516 engineering process Methods 0.000 title claims abstract description 12
- 239000011664 nicotinic acid Substances 0.000 claims abstract description 45
- 210000000078 claw Anatomy 0.000 claims description 57
- 239000002420 orchard Substances 0.000 claims description 21
- 230000000007 visual effect Effects 0.000 claims description 17
- 238000003860 storage Methods 0.000 claims description 15
- 230000036544 posture Effects 0.000 claims description 13
- 238000001514 detection method Methods 0.000 claims description 11
- 239000011159 matrix material Substances 0.000 claims description 9
- 238000013135 deep learning Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 7
- 241000238413 Octopus Species 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- 239000000463 material Substances 0.000 claims description 4
- 238000013519 translation Methods 0.000 claims description 3
- 238000011179 visual inspection Methods 0.000 claims description 3
- 241000566145 Otus Species 0.000 claims description 2
- 230000004888 barrier function Effects 0.000 claims description 2
- 238000003708 edge detection Methods 0.000 claims description 2
- 238000001914 filtration Methods 0.000 claims description 2
- 238000011065 in-situ storage Methods 0.000 claims description 2
- 238000002360 preparation method Methods 0.000 claims description 2
- 238000013461 design Methods 0.000 abstract description 2
- 230000009471 action Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000010276 construction Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000033001 locomotion Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N Silicium dioxide Chemical compound O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000003306 harvesting Methods 0.000 description 1
- 239000010720 hydraulic oil Substances 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000000741 silica gel Substances 0.000 description 1
- 229910002027 silica gel Inorganic materials 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D46/00—Picking of fruits, vegetables, hops, or the like; Devices for shaking trees or shrubs
- A01D46/30—Robotic devices for individually picking crops
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D91/00—Methods for harvesting agricultural products
- A01D91/04—Products growing above the soil
Landscapes
- Life Sciences & Earth Sciences (AREA)
- Environmental Sciences (AREA)
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Manipulator (AREA)
Abstract
The invention discloses a full-automatic fruit picking robot and a picking method based on a vision technology, and the robot comprises a mobile platform, a lifting platform, an industrial personal computer, a mechanical arm, a vision system, a bionic clamping jaw and a collecting device, wherein the mobile platform is connected with the lifting platform through a connecting rod; the mechanical arm, the industrial personal computer and the lifting platform are arranged on a supporting bottom plate of the mobile platform, and the collecting device is arranged on the lifting platform; the tail end of the mechanical arm is provided with a bionic clamping jaw and a vision system. The design of the octopus-imitating clamping jaw can increase the contact area between the clamping jaw and the fruit, so that the clamping jaw can more stably grab the target fruit, the adaptability to different target volumes can be realized, and the universality is increased.
Description
Technical Field
The invention belongs to the field of agricultural machinery, and particularly relates to a full-automatic fruit picking robot and a picking method based on a vision technology.
Background
The orchard planting area and fruit yield in China stably live at the first position of the world all the year round, and fruit picking is an important link for orchard planting. A conventional fruit picking machine, for example, CN110506501A discloses an automatic fruit picking machine, which fixes and shakes a fruit tree by an engaging vibration device on a vehicle body to drop the fruit, and then receives the dropped fruit by a receiving device, thereby harvesting the fruit; however, in the picking and shaking process of the device, the target fruits and branches and leaves can be mixed and fall off, the subsequent fruit and leaf separation step is increased, and the device can also cause certain damage to the surfaces of fruit trees and fruits. Patent CN109302885A discloses a fruit picking robot, picking fruits by a manipulator, and placing the picked fruits to a collecting device; however, the manipulator of the picking robot cannot adapt to fruits of different sizes, damage to the fruit with a large size to a certain extent can be caused during actual picking, manual operation is needed, and full automation cannot be achieved completely. Therefore, how to reduce the damage to fruits and fruit trees in the fruit picking process and improve the picking efficiency is a technical problem which needs to be solved urgently in the field.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provide a full-automatic fruit picking robot and a picking method based on a vision technology, wherein a bionic clamping jaw can adapt to the size of fruits and has small damage to the fruits and fruit trees, the picking robot can determine the picking position and angle through vision, and then the height of a lifting platform is automatically controlled to expand the picking range and improve the picking efficiency.
The purpose of the invention is realized by the following technical scheme:
a full-automatic fruit picking robot based on a vision technology comprises a moving platform, a lifting platform 2, an industrial personal computer 3, a mechanical arm 4, a vision system 5, a bionic clamping jaw 6 and a collecting device; the mechanical arm 4, the industrial personal computer 3 and the lifting platform 2 are arranged on a supporting bottom plate 8 of the mobile platform, and the collecting device is arranged on the lifting platform 2; the tail end of the mechanical arm 4 is provided with a bionic clamping jaw 6 and a vision system 5.
The moving platform comprises a moving trolley 1, a supporting bottom plate 8 and an ultrasonic ranging sensor 9; a supporting bottom plate 8 is fixed at the top of the mobile trolley 1; the front end of the moving trolley 1 is provided with an ultrasonic ranging sensor 9.
The vision system 5 comprises a first binocular camera 13, a second binocular camera 14, a camera support 15 and an inertial navigation module 16, wherein the two binocular cameras are installed on the camera support 15, the camera support 15 is fixedly installed on a mechanical arm flange plate 17, and the inertial navigation module 16 is fixed behind the camera support 15. The first binocular camera 13 and the inertial navigation module 16 form a visual slam system to complete the construction work of the orchard map; the second binocular camera 14 captures the current frame image of the camera in real time while moving, and recognizes and locates fruits present in the current frame image through deep learning.
The 4 bionic clamping jaws 6 are uniformly distributed on a connecting piece 18, the connecting piece 18 is fixed on a flange plate 17 of the mechanical arm, and a motor spindle of a stepping motor 19 is sleeved with the connecting piece 18. The bionic clamping jaw 6 is designed by adopting a bionic octopus and comprises an outer jaw 25, an inner jaw 22 and a sucking disc 24; the front end of the inner claw 22 is hinged with the front end of the outer claw 25, and two springs 21 are fixedly arranged between the inner claw 22 and the outer claw 25; the front end of the second connecting rod 27 is hinged with the tail end of the outer claw 25, the protruding part of the second connecting rod 27 is provided with an arc-shaped notch, and the tail end of the inner claw 22 can slide in the arc-shaped notch; the outer claw 25 is sleeved with a torsion spring 28 at the hinged position of the second connecting rod 27, the torsion spring torsion adjusting device 26 is arranged at the connecting position of the outer claw 25 and the second connecting rod 27, and the length of the torsion spring 28 is controlled by adjusting the depth of a nut of the torsion spring torsion adjusting device, so that the torsion force of the torsion spring is controlled.
The inner claw 22 is of an arc-shaped structure, more than one sucking disc 24 is arranged on the surface of the inner claw, the sucking discs 24 are made of flexible materials and are arranged on the ball stud, and the other end of the ball stud is fixed on the inner claw 22.
One end of the first connecting rod 20 is connected with a motor base 23 of the stepping motor 19 to form a rotating pair; the other end of the first link 20 is connected with the protruding part of the second link 27 in a living hinge manner, and the end of the second link 27 is connected with the connecting member 18 in a living hinge manner and limits the rotation angle of the outer jaw to 0-30 °.
The working principle of the bionic clamping jaw 6 is as follows: (1) The bionic clamping jaw 6 is designed by adopting a bionic octopus, and as the surfaces of most fruits are arc-shaped, the inner jaw 22 is designed into an arc shape so as to fit the surfaces of the fruits for picking; a row of sucking discs 24 are designed on the surface of the inner claw, the sucking discs can freely rotate on the ball stud according to the stress condition, when fruit is grabbed, the sucking discs are stressed to enable the surfaces of the sucking discs to be tightly attached to the surfaces of the fruit, the contact area of the bionic clamping jaws and the fruit is increased, on one hand, the surfaces of the fruit are protected from being damaged, and on the other hand, the grabbing stability of the bionic clamping jaws is also increased; (2) When the inner jaw 22 is pressed, the inner jaw 22 starts to rotate towards the outer jaw 25, and the spring 21 generates a reaction force to increase the grabbing force of the inner jaw 22 to the target; the fruit gripping device has the advantages that the torsion spring torsion adjusting device is not changed, when the spring between the inner claw and the outer claw is compressed to the maximum, the bionic gripping claw can grip the maximum volume of fruit, if the volume of the fruit to be gripped exceeds the value, the torsion spring torsion can be reduced through changing the scale of the nut on the torsion spring torsion adjusting device, so that the outer claw can rotate by a certain angle, and the effective gripping volume of the gripping claw is increased. Due to the dual functions of the inner claw and the outer claw, the invention can realize the self-adaptability to different target volumes without algorithm control, and enhances the universality of picking fruits with different sizes.
A full-automatic fruit picking method based on a vision technology adopts the full-automatic fruit picking robot and comprises the following steps:
(1) Early preparation: calibrating a single binocular camera and a binocular camera to obtain an internal parameter matrix and an external parameter matrix and a reprojection matrix of each camera; performing hand-eye calibration on the second binocular camera 14 to obtain a camera coordinate system and a rotation translation matrix of a mechanical arm base coordinate system, and converting points in the camera coordinate system into points in the mechanical arm base coordinate system;
(2) Constructing a map: a visual slam system consisting of a first binocular camera 13 and an inertial navigation module 16 is used for constructing a map in the orchard, and a three-dimensional point cloud map of the whole orchard, including position information of a starting point and an end point, is obtained;
(3) Visual inspection: loading a three-dimensional point cloud map of the orchard, and automatically walking the mobile trolley along the map track; the second binocular camera 14 acquires and detects each frame of image in real time by using a Yolov5 network, when a target fruit appears, the mobile cart stops, the second binocular camera stores the current frame of image, and if not, the second binocular camera continues to walk;
(4) And (3) autonomous obstacle avoidance: when the mobile trolley walks, the ultrasonic ranging sensor 9 detects the obstacles in real time and feeds back the distance between the obstacle and the industrial personal computer; when the distance is less than 1m, the mobile trolley stops, and if the obstacle disappears within 10s, the mobile trolley continues to travel; otherwise, the mobile trolley turns left or right in situ, and continues to walk after bypassing the barrier;
(5) Automatic lifting: when the mobile trolley stops due to fruit detection, the outline of each fruit in the stored image in the step (3) is extracted, the mass center of all the outlines of the fruits is calculated to obtain a mass center intermediate value, the three-dimensional space position of the mass center intermediate value is used as the initial picking posture of the mechanical arm, namely the position information of the mass center intermediate value is transmitted to the industrial personal computer, and the industrial personal computer controls the height of the lifting platform to enable the tail end of the mechanical arm to be over against the position of the mass center intermediate value.
(6) Positioning a target: calculating the center points of the fruits and the fruit stalks in the image, judging the inclination angles of the fruits and the horizontal plane, and controlling the tail end postures of the mechanical arms; then, fruit centroids in the left image and the right image of the binocular camera are matched one by utilizing an SAD (sum of absolute differences) matching algorithm, the spatial position of each fruit centroid is calculated, and then the spatial position of the fruit centroids is converted into the base coordinates of the mechanical arm;
(7) Picking fruits: the mechanical arm sequentially goes to the center of mass point of each fruit; when the target fruit completely enters the bionic clamping jaw, the bionic clamping jaw starts to be closed, the sucking discs on the inner jaws rotate through the reaction force of the surface of the fruit until the sucking discs contacting the fruit are tightly attached to the surface of the fruit, and meanwhile, the springs between the inner jaws and the outer jaws give acting force, so that the sucking discs can more firmly grasp the fruit;
(8) Placing fruits: after the bionic clamping jaw grabs the fruits, the mechanical arm moves to the upper part of the storage box, and the fruits are placed in the storage box until the fruits in the current visual field are picked; the infrared sensor on the storage box can detect whether the fruit in the box is full; the mobile trolley can continue to pick along the three-dimensional point cloud map of the orchard until the whole orchard picking is completed.
In the step (2), the first binocular camera and the inertial navigation module are used for constructing the map, and the inertial navigation module can accurately convey rotation and displacement generated in the motion process in real time, so that the map constructing accuracy is improved.
In the step (3), a large number of fruit samples and fruit stem pictures thereof are prepared in advance to construct a fruit image library, and then the fruit image library is trained by using a YOLOv5 deep learning network, so that the YOLOv5 deep learning network can identify fruits and fruit stems in the pictures.
In the step (5), after the stored image is extracted, carrying out gray processing on a detection frame containing fruits, then carrying out binarization operation on the detection frame by using an OTUS method, and then carrying out operations such as filtering processing and the like on the image to eliminate noise; finally, extracting outline information by using a canny operator and an edge detection algorithm function, and calculating the centroid (x) of the outline i ,y i ) (ii) a Repeating the operation until obtaining the centroids of all the fruit outlines in the image, and adding the horizontal and vertical coordinates of each centroid to obtain the sum w i ＝x i +y i (ii) a It is added with w i Arranging according to the sequence from big to small, and taking the middle value; when the total number (n) of fruits in the visual field is even, the center of mass is (x) n/2 ,y n/2 ) (ii) a When the total number of fruits in the visual field isWhen the number is odd, the center of mass is (x) (n+1)/2 ,y (n+1)/2 ) Or is (x) (n-1)/2 ,y (n-1)/2 ) (ii) a Then, calculating the actual spatial position of the center of mass median by using a triangulation principle, and controlling the height of the lifting platform by using an industrial personal computer to enable the tail end of the mechanical arm to be directly opposite to the fruit where the center of mass median is located; the attitude of the robotic arm is now set to the initial attitude so that all fruit is in a pickable state from the perspective of the robotic arm.
In the step (6), detecting frames of the fruits and the fruit stalks are identified by deep learning, and the central points (Fx, fy) of the fruit detecting frames and the central points (Fx, fy) of the corresponding fruit stalk detecting frames are obtained; calculating the slope k = Fy-Fy/Fx-Fx of the two points, and then calculating alpha by using an inverse trigonometric function arctan alpha = k, namely the inclination angle of the fruit relative to the horizontal plane; if alpha is more than 15 degrees, the bionic clamping jaw is rotated by a corresponding angle; if alpha < =15 degrees, the bionic clamping jaw keeps the initial posture unchanged.
The working principle of the invention is as follows: carrying out map construction on the orchard through a visual slam system, and reading a map by a mobile trolley and walking along the map; when the trolley travels, the ultrasonic ranging sensor carried by the trolley can feed back the distance between the trolley and the obstacle in the traveling direction, so that the trolley is driven to bypass the obstacle to realize autonomous obstacle avoidance; meanwhile, in the walking process, the other binocular camera performs real-time detection. The working space of the mechanical arm is relatively small, and for the initial pose of the mechanical arm, a phenomenon that picking fails due to the fact that a lot of fruits are far away exists, so that after the target fruits are detected, the industrial personal computer sends signals to enable the trolley to stop moving and controls the height of the lifting platform, all the fruits are in a picking state under the visual angle of the mechanical arm, and the picking rate is improved; before picking, the posture of the tail end is changed by judging the inclination angle of the fruit and the horizontal plane; when picking, the reaction force of the fruit enables the sucker arranged on the ball head bolt to rotate, the sucker is enabled to be tightly attached to the surface of the fruit, the contact area of the clamping jaw and the fruit is increased, and the bionic clamping jaw can adaptively grab the fruit under the double action of the inner jaw and the outer jaw. After picking, putting the fruits into the storage box, and judging whether the storage box is filled with the fruits or not through the infrared sensor.
Compared with the prior art, the invention has the following advantages and effects:
(1) According to the method, the orchard map is constructed by adopting a mode of combining the binocular camera and the inertial navigation module, an accurate pose can be provided when the mobile trolley moves violently, the condition of losing the characteristic points cannot occur, and the accuracy of the acquired map point cloud is greatly improved.
(2) The invention determines the height of the lifting platform and the initial pose of the mechanical arm in each picking process through an algorithm, can increase the picking success rate and the picking range, and avoids the situation of picking failure caused by exceeding the working range of the mechanical arm.
(3) The invention adopts the ultrasonic ranging sensor, and can realize the autonomous obstacle avoidance function of the moving trolley during walking.
(4) The invention has good bionic effect, and the design of the octopus-imitating clamping jaw can increase the contact area between the clamping jaw and the fruit in the picking process, so that the clamping jaw can more stably grab the target fruit. For fruits with moderate volume, the spring between the inner claw and the outer claw can increase the grabbing force of the tail end and cannot damage the surface of the fruits; for the fruit with larger volume, if the spring between the inner claw and the outer claw is compressed to the maximum value, the outer claw can continue to rotate outwards under the action of the fruit to increase the effective capacity of the clamping claw, the double action of the inner claw and the outer claw ensures that the tail end can realize the self-adaptability to different target volumes without an algorithm, and the universality is increased.
(5) The fruit inclination degree is calculated through an algorithm, and then the rotation angle of the clamping jaw is controlled. When the fruit with relatively inclined growth posture faces, the clamping jaws can well grab the fruit, and the situation that the clamping jaw parts push the fruit open is avoided.
(6) The inner claw of the bionic octopus clamping jaw and the sucker arranged on the inner claw are made of soft silica gel materials, so that the weight of the tail end can be reduced, and the surface of a fruit can be protected from being damaged.
(7) The invention can be suitable for full-automatic picking of various fruits and has strong universality.
Drawings
Fig. 1 is a schematic diagram of the overall structure of a full-automatic fruit picking robot.
Fig. 2 is a schematic structural view of a bionic clamping jaw and a vision system.
Fig. 3 is a schematic structural diagram of the bionic clamping jaw.
Fig. 4 is a picking work flow chart of the full-automatic fruit picking robot.
Wherein: 1. moving the trolley; 2. a lifting platform; 3. an industrial personal computer; 4. a mechanical arm; 5. a vision system; 6. bionic clamping jaws; 7. a storage tank; 8. a support base plate; 9. an ultrasonic ranging sensor; 10. a power plant; 11 a control cabinet and a power supply cabinet of the mechanical arm; 12. an infrared sensor; 13. a first binocular camera; 14. a second binocular camera; 15. a camera support; 16. an inertial navigation module; 17 mechanical arm flange plates; 18. a connecting member; 19. a stepping motor; 20. a first link; 21. a spring; 22. an inner jaw; 23. a motor base; 24. a suction cup; 25. an outer jaw; 26. a torsion spring torsion force adjusting member; 27. a second link 2; 28. a torsion spring.
Detailed Description
In order that the invention may be readily understood, reference will now be made in detail to the present invention as illustrated in the accompanying examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that various changes and modifications can be made by those skilled in the art without departing from the spirit of the invention, and these changes and modifications are all within the scope of the invention.
Example 1
A full-automatic fruit picking robot based on a vision technology is shown in figure 1 and comprises a moving platform, a lifting platform 2, an industrial personal computer 3, a mechanical arm 4, a vision system 5, a bionic clamping jaw 6 and a collecting device; the mechanical arm 4, the industrial personal computer 3 and the lifting platform 2 are arranged on a supporting bottom plate 8 of the mobile platform, and the collecting device is arranged on the lifting platform 2; the tail end of the mechanical arm 4 is provided with a bionic clamping jaw 6 and a vision system 5. The moving platform comprises a moving trolley 1, a supporting bottom plate 8 and an ultrasonic ranging sensor 9; a supporting bottom plate 8 is fixed at the top of the mobile trolley 1; the front end of the moving trolley 1 is provided with an ultrasonic ranging sensor 9. The crawler-type travelling car can be selected as the travelling car 1, can adapt to various terrains, can drive over small obstacles, and simultaneously the travelling car 1 also provides 220V power for other components. The lifting platform 2 is a hydraulic lifting platform, the hydraulic lifting platform and a power device 10 thereof are fixedly arranged on the supporting bottom plate 8, and the lifting function is realized through the pressure transmission of hydraulic oil. The industrial personal computer 3 is equipment for carrying out algorithm processing on the whole system, integral communication is realized by the industrial personal computer, and the industrial personal computer is fixed on the supporting bottom plate 8. The mechanical arm 4 is a mechanical arm with six degrees of freedom, each mechanical arm is driven by a special motor, and the mechanical arm has great flexibility in practical application, so that the mechanical arm can finish picking actions under different growth postures of fruits. The control cabinet and the power supply cabinet 11 are fixed on the supporting bottom plate 8.
As shown in fig. 2, the vision system 5 includes a first binocular camera 13, a second binocular camera 14, a camera support 15 and an inertial navigation module 16, the two binocular cameras are mounted on the camera support 15, the camera support 15 is fixedly mounted on a mechanical arm flange 17, and the inertial navigation module 16 is fixed behind the camera support 15. The first binocular camera 13 and the inertial navigation module 16 form a visual slam system to complete the construction work of the orchard map; the second binocular camera 14 captures the current frame image of the camera in real time while moving, and identifies and locates fruits existing in the current frame image through deep learning.
As shown in fig. 2, 4 bionic clamping jaws 6 are uniformly distributed on a connecting piece 18, the connecting piece 18 is fixed on a flange plate 17 of the mechanical arm, and a motor spindle of a stepping motor 19 is sleeved with the connecting piece 18. As shown in fig. 3, the bionic clamping jaw 6 is designed by a bionic octopus and comprises an outer jaw 25, an inner jaw 22 and a suction cup 24; the front end of the inner claw 22 is hinged with the front end of the outer claw 25, and two springs 21 are fixedly arranged between the inner claw 22 and the outer claw 25; the front end of the second connecting rod 27 is hinged with the tail end of the outer claw 25, the protruding part of the second connecting rod 27 is provided with an arc-shaped notch, and the tail end of the inner claw 22 can slide in the arc-shaped notch; the outer claw 25 is sleeved with a torsion spring 28 at the hinged position of the second connecting rod 27, the torsion spring torsion adjusting device 26 is arranged at the connecting position of the outer claw 25 and the second connecting rod 27, and the length of the torsion spring 28 is controlled by adjusting the depth of a nut of the torsion spring torsion adjusting device, so that the torsion force of the torsion spring is controlled. The inner claw 22 is of an arc-shaped structure, more than one sucking disc 24 is arranged on the surface of the inner claw, the sucking discs 24 are made of flexible materials and are installed on the ball stud, and the other end of the ball stud is fixed on the inner claw 22. One end of the first connecting rod 20 is connected with a motor base 23 of the stepping motor 19 to form a revolute pair; the other end of the first link 20 is connected with the protruding portion of the second link 27 in a living hinge manner, and the end of the second link 27 is connected with the connecting member 18 in a living hinge manner, and restricts the rotation angle of the outer jaw to 0 to 30 °. The collecting device comprises a storage tank 7 and an infrared sensor 12. The storage box 7 is placed on the lifting platform 2, and the freedom degrees of the storage box in five directions are limited through the clamping groove and the lifting platform, so that the stable state during movement is ensured. An infrared sensor 12 is mounted at the top edge of the bin for determining whether the interior of the bin is filled with fruit.
As shown in fig. 4, the fully automatic fruit picking method based on the visual technology comprises the following steps:
(1) Constructing a map: a visual slam system consisting of a first binocular camera 13 and an inertial navigation module 16 is utilized to construct a map in the orchard, and a three-dimensional point cloud map of the whole orchard, including position information of a starting point and an end point, is obtained;
(2) Visual inspection: loading a three-dimensional point cloud map of the orchard, and automatically walking the mobile trolley along the map track; the second binocular camera 14 acquires and detects each frame of image in real time by using a Yolov5 network, when a target fruit appears, the mobile cart stops, the second binocular camera stores the current frame of image, and if not, the second binocular camera continues to walk;
(3) And (3) autonomous obstacle avoidance: when the moving trolley walks, the ultrasonic ranging sensor 9 detects the obstacles in real time and feeds back the distance between the obstacle and the moving trolley to the industrial personal computer; when the distance is less than 1m, the mobile trolley stops, and if the obstacle disappears within 10s, the mobile trolley continues to travel; otherwise, the mobile trolley turns left or right in place (if the left side also has an obstacle, the mobile trolley turns right), and continues to walk after bypassing the obstacle;
(4) Automatic lifting: when the mobile trolley stops due to fruit detection, the outline of each fruit in the stored image in the step (3) is extracted, the mass center of all the outlines of the fruits is calculated to obtain a mass center intermediate value, the three-dimensional space position of the mass center intermediate value is used as the initial picking posture of the mechanical arm, namely the position information of the mass center intermediate value is transmitted to the industrial personal computer, and the industrial personal computer controls the height of the lifting platform to enable the tail end of the mechanical arm to be over against the position of the mass center intermediate value.
(5) Positioning a target: calculating the center points of the fruits and the fruit stalks in the image, judging the inclination angles of the fruits and the horizontal plane, and controlling the tail end postures of the mechanical arms; then, fruit centroids in the left image and the right image of the binocular camera are matched one by utilizing an SAD (sum of absolute differences) matching algorithm, the spatial position of each fruit centroid is calculated, and then the spatial position of the fruit centroids is converted into the base coordinates of the mechanical arm;
(6) Picking fruits: the mechanical arm sequentially goes to the center of mass point of each fruit; when the target fruit completely enters the bionic clamping jaw, the bionic clamping jaw starts to be closed, the sucking discs on the inner jaws rotate through the reaction force of the surface of the fruit until the sucking discs contacting the fruit are tightly attached to the surface of the fruit, and meanwhile, the springs between the inner jaws and the outer jaws give acting force, so that the sucking discs can more firmly grasp the fruit;
(8) Placing fruits: after the bionic clamping jaw grabs the fruits, the mechanical arm moves to the upper part of the storage box and places the fruits in the storage box until the fruits in the current visual field are picked; the infrared sensor on the storage box can detect whether the fruit in the box is full; the mobile trolley can continue to pick along the three-dimensional point cloud map of the orchard until the whole orchard picking is completed.
Example 2
Under the condition of not changing a torsion spring torsion adjusting device, when the spring between the inner claw and the outer claw is compressed to the maximum, the bionic clamping jaw can grab the maximum volume of the fruit. When the size of the grabbed target is larger than the value, the torsion of the torsion spring can be reduced by changing the scale of the nut on the torsion spring torsion adjusting device, so that the outer claw of the clamping jaw rotates outwards under the action of force, and the effective grabbing size of the clamping jaw can be increased. The double action of the inner claw and the outer claw ensures that the tail end can realize the self-adaptability to different target volumes without an algorithm, thereby enhancing the universality of picking fruits with different sizes.
Example 3
When the growth posture of the grabbed fruit is not vertical, namely the fruit has a certain inclination angle with the horizontal plane, the positions of the fruit and the fruit stems thereof are predicted through deep learning, and then an inclination angle of the fruit relative to the horizontal plane is calculated through the central point positions of the fruit and fruit stem detection frames; when the angle is smaller than 15 degrees, the clamping jaws move forwards to grab the fruits at the initial posture, and when the angle is larger than 15 degrees, the clamping jaws rotate by the corresponding angle according to the calculated numerical value, so that the phenomenon that the clamping jaws push away fruit stalks or fruits when the clamping jaws move forwards to a picking point to cause picking failure is avoided.
The above description is only an example of the present invention, but the present invention is not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention are all equivalent substitutions and are intended to be included within the scope of the present invention.
Claims (10)
1. The utility model provides a full-automatic fruit picking robot based on vision technique which characterized in that: the bionic clamping jaw comprises a moving platform, a lifting platform, an industrial personal computer, a mechanical arm, a vision system, a bionic clamping jaw and a collecting device; the mechanical arm, the industrial personal computer and the lifting platform are arranged on a supporting bottom plate of the mobile platform, and the collecting device is arranged on the lifting platform; the tail end of the mechanical arm is provided with a bionic clamping jaw and a vision system.
2. The full-automatic fruit picking robot according to claim 1, characterized in that: the moving platform comprises a moving trolley, a supporting bottom plate and an ultrasonic ranging sensor; a supporting bottom plate is fixed at the top of the movable trolley; the front end of the movable trolley is provided with an ultrasonic ranging sensor.
3. The full automatic fruit picking robot of claim 1, wherein: the vision system comprises a first binocular camera, a second binocular camera, a camera support and an inertial navigation module, wherein the two binocular cameras are installed on the camera support, the camera support is fixedly installed on a mechanical arm flange plate, and the inertial navigation module is fixed behind the camera support.
4. The full automatic fruit picking robot of claim 1, wherein: 4 bionic clamping jaws are uniformly distributed on the connecting piece, the connecting piece is fixed on a flange plate of the mechanical arm, and a motor spindle of the stepping motor is sleeved with the connecting piece.
5. The full automatic fruit picking robot of claim 1 or 4, wherein: the bionic clamping jaw is designed by adopting a bionic octopus and comprises an outer jaw, an inner jaw and a sucker; the front end of the inner claw is hinged with the front end of the outer claw, and two springs are fixedly arranged between the inner claw and the outer claw; the front end of the second connecting rod is hinged with the tail end of the outer claw, the protruding part of the second connecting rod is provided with an arc-shaped notch, and the tail end of the inner claw can slide in the arc-shaped notch; the outer claw is sleeved with a torsion spring at the hinged position of the second connecting rod, the torsion spring torsion adjusting device is installed at the connecting position of the outer claw and the second connecting rod, and the length of the torsion spring is controlled by adjusting the depth of a nut of the torsion spring torsion adjusting device, so that the torsion force of the torsion spring is controlled.
6. The full automatic fruit picking robot of claim 5, wherein: the inner claw is of an arc-shaped structure, more than one sucker is arranged on the surface of the inner claw, the sucker is made of flexible materials and is installed on the ball stud, and the other end of the ball stud is fixed on the inner claw.
7. The full automatic fruit picking robot of claim 1 or 4, wherein: one end of the first connecting rod is connected with a motor base of the stepping motor to form a revolute pair; the other end of the first connecting rod is connected with the protruding part of the second connecting rod in a movable hinge mode, the tail end of the second connecting rod is connected with the connecting piece in a movable hinge mode, and the rotating angle of the outer claw is limited to be 0-30 degrees.
8. A full-automatic fruit picking method based on a vision technology is characterized in that: the full-automatic fruit picking robot is adopted, and comprises the following steps:
(1) Early preparation: calibrating a single binocular camera and a binocular camera to obtain an internal parameter matrix and an external parameter matrix and a reprojection matrix of each camera; performing hand-eye calibration on the second binocular camera to obtain a camera coordinate system and a rotation translation matrix of a mechanical arm base coordinate system, wherein the rotation translation matrix is used for converting points in the camera coordinate system into points in the mechanical arm base coordinate system;
(2) Constructing a map: constructing a map in the orchard by using a visual slam system consisting of a first binocular camera and an inertial navigation module, and obtaining a three-dimensional point cloud map of the whole orchard, wherein the three-dimensional point cloud map comprises position information of a starting point and an end point;
(3) Visual inspection: loading a three-dimensional point cloud map of the orchard, and automatically walking the mobile trolley along the map track; the second binocular camera acquires and utilizes a YOLOv5 network to detect each frame of image in real time, when a target fruit appears, the mobile trolley stops, the second binocular camera stores the current frame of image, and if not, the second binocular camera continues to walk;
(4) And (3) autonomous obstacle avoidance: when the mobile trolley walks, the ultrasonic ranging sensor detects the obstacles in real time and feeds back the distance between the ultrasonic ranging sensor and the obstacles to the industrial personal computer; when the distance is less than 1m, the mobile trolley stops, and if the obstacle disappears within 10s, the mobile trolley continues to travel; otherwise, the mobile trolley turns left or right in situ and continues to walk after bypassing the barrier;
(5) Automatic lifting: when the mobile trolley stops due to fruit detection, the outline of each fruit in the stored image in the step (3) is extracted, the mass center of all the outlines of the fruits is calculated to obtain a mass center intermediate value, the three-dimensional space position of the mass center intermediate value is used as the initial picking posture of the mechanical arm, namely the position information of the mass center intermediate value is transmitted to the industrial personal computer, and the industrial personal computer controls the height of the lifting platform to enable the tail end of the mechanical arm to be over against the position of the mass center intermediate value.
(6) Positioning a target: calculating the center points of the fruits and the fruit stalks in the image, judging the inclination angles of the fruits and the horizontal plane, and controlling the tail end postures of the mechanical arms; then, carrying out one-to-one matching on fruit centroids in left and right images of the binocular camera by utilizing an SAD matching algorithm, calculating the spatial position of each fruit centroid, and converting the spatial position of the fruit centroids into a mechanical arm base coordinate;
(7) Picking fruits: the mechanical arm sequentially goes to the center of mass point of each fruit; when the target fruit completely enters the bionic clamping jaw, the bionic clamping jaw starts to be closed, the sucking discs on the inner jaws rotate through the reaction force of the surface of the fruit until the sucking discs contacting the fruit are tightly attached to the surface of the fruit, and meanwhile, the springs between the inner jaws and the outer jaws give acting force, so that the sucking discs can more firmly grasp the fruit;
(8) Placing fruits: after the bionic clamping jaw grabs the fruits, the mechanical arm moves to the upper part of the storage box, and the fruits are placed in the storage box until the fruits in the current visual field are picked; the infrared sensor on the storage box can detect whether the fruit in the box is full; the mobile trolley can continue to pick along the three-dimensional point cloud map of the orchard until the whole orchard picking is completed.
9. The fully automatic fruit picking method based on visual technology according to claim 8, characterized in that: after the stored image is extracted, carrying out gray level processing on a detection frame containing fruits, then carrying out binarization operation on the detection frame by using an OTUS method, and then carrying out operations such as filtering processing on the image to eliminate noise; finally, contour information is extracted by using a canny operator and an edge detection algorithm function, and the centroid (x) of the contour is calculated i ,y i ) (ii) a Repeating the operation until obtaining the centroids of all the fruit outlines in the image, and adding the horizontal and vertical coordinates of each centroid to obtain the sum w i ＝x i +y i (ii) a It is added with w i Arranging according to the sequence from big to small, and taking the middle value; when the total number (n) of fruits in the visual field is even, the center of mass is (x) n/2 ,y n/2 ) (ii) a When the total number of the fruits in the visual field is odd, the center value of the mass center is (x) (n+1)/2 ,y (n+1)/2 ) Or is (x) (n-1)/2 ,y (n-1)/2 ) (ii) a And then calculating the actual spatial position of the center of mass by using a triangulation principle, and controlling the height of the lifting platform by using an industrial personal computer to enable the tail end of the mechanical arm to be directly opposite to the fruit where the center of mass is located.
10. The full-automatic fruit picking method based on the visual technology as claimed in claim 8, characterized in that: in the step (6), the detection frames of the fruits and the fruit stalks are identified by utilizing deep learning, and the central points (Fx, fy) of the fruit detection frames and the central points (Fx, fy) of the corresponding fruit stalk detection frames are obtained; calculating the slope k = Fy-Fy/Fx-Fx of the two points, and then calculating alpha by using an inverse trigonometric function arctan alpha = k, namely the inclination angle of the fruit relative to the horizontal plane; if alpha is more than 15 degrees, the bionic clamping jaw is rotated by a corresponding angle; if alpha < =15 degrees, the bionic clamping jaw keeps the initial posture unchanged.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210936842.2A CN115250745B (en) | 2022-08-05 | 2022-08-05 | Full-automatic fruit picking robot and picking method based on vision technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210936842.2A CN115250745B (en) | 2022-08-05 | 2022-08-05 | Full-automatic fruit picking robot and picking method based on vision technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115250745A true CN115250745A (en) | 2022-11-01 |
| CN115250745B CN115250745B (en) | 2024-03-12 |
Family
ID=83749611
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210936842.2A Active CN115250745B (en) | 2022-08-05 | 2022-08-05 | Full-automatic fruit picking robot and picking method based on vision technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115250745B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210337734A1 (en) * | 2018-10-08 | 2021-11-04 | Advanced Farm Technologies, Inc. | Autonomous crop harvester |
| CN115643902A (en) * | 2022-11-03 | 2023-01-31 | 华南农业大学 | Litchi paired-rake accurate vibration harvesting equipment and harvesting method thereof |
| CN115781766A (en) * | 2023-02-13 | 2023-03-14 | 安徽文达信息工程学院 | Mechanical arm with visual positioning system |
| CN115937314A (en) * | 2022-12-23 | 2023-04-07 | 南京林业大学 | Camellia oleifera fruit growth posture detection method |
| CN116079712A (en) * | 2022-12-01 | 2023-05-09 | 杭州电子科技大学 | Three-dimensional space clamping height control method of mechanical arm |
| CN116439018A (en) * | 2023-05-05 | 2023-07-18 | 仲恺农业工程学院 | Seven-degree-of-freedom fruit picking robot and picking method thereof |
| CN116985270A (en) * | 2023-07-19 | 2023-11-03 | 玉溪师范学院 | Bionic fossil repairing appliance |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101273688A (en) * | 2008-05-05 | 2008-10-01 | 江苏大学 | Apparatus and method for flexible pick of orange picking robot |
| CN203752160U (en) * | 2014-03-25 | 2014-08-06 | 甘肃农业大学 | Quick under-actuated absorbing picking manipulator |
| CN106508282A (en) * | 2016-09-22 | 2017-03-22 | 华南农业大学 | Clamping and shearing type manipulator end effector |
| CN107471212A (en) * | 2017-08-24 | 2017-12-15 | 佛山伊贝尔科技有限公司 | A kind of flexible pneumatic manipulator |
| CN107711078A (en) * | 2017-10-12 | 2018-02-23 | 谷新运 | Tomato absorption clamps synchronous picking end effector and corresponding picking mechanism and method |
| US20180146618A1 (en) * | 2016-11-29 | 2018-05-31 | Invia Robotics, Inc. | Harvesting Robots for Hydroponics |
| CN207836206U (en) * | 2017-11-21 | 2018-09-11 | 祝锐 | A kind of portable white heart shaddock stripper unit |
| CN109479522A (en) * | 2018-12-27 | 2019-03-19 | 甘肃农业大学 | A kind of fruit picking robot and its picking method |
| CN111656959A (en) * | 2020-07-27 | 2020-09-15 | 荆门掇刀双井刘伟西瓜专业合作社 | A pick device for watermelon |
| CN112743566A (en) * | 2021-03-05 | 2021-05-04 | 广东海洋大学 | Pulling-in type mechanical claw capable of changing form |
| CN112976029A (en) * | 2021-03-11 | 2021-06-18 | 南京农业大学 | Soft electrostatic adhesion coating type bionic octopus manipulator |
| CN113330915A (en) * | 2021-05-26 | 2021-09-03 | 华南农业大学 | Self-adaptive cotton harvesting method based on binocular vision recognition and intelligent mechanical harvesting device |
| CN114402806A (en) * | 2022-02-18 | 2022-04-29 | 湖北汽车工业学院 | Spherical fruit picking robot and picking method |
-
2022
- 2022-08-05 CN CN202210936842.2A patent/CN115250745B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101273688A (en) * | 2008-05-05 | 2008-10-01 | 江苏大学 | Apparatus and method for flexible pick of orange picking robot |
| CN203752160U (en) * | 2014-03-25 | 2014-08-06 | 甘肃农业大学 | Quick under-actuated absorbing picking manipulator |
| CN106508282A (en) * | 2016-09-22 | 2017-03-22 | 华南农业大学 | Clamping and shearing type manipulator end effector |
| US20180146618A1 (en) * | 2016-11-29 | 2018-05-31 | Invia Robotics, Inc. | Harvesting Robots for Hydroponics |
| CN107471212A (en) * | 2017-08-24 | 2017-12-15 | 佛山伊贝尔科技有限公司 | A kind of flexible pneumatic manipulator |
| CN107711078A (en) * | 2017-10-12 | 2018-02-23 | 谷新运 | Tomato absorption clamps synchronous picking end effector and corresponding picking mechanism and method |
| CN207836206U (en) * | 2017-11-21 | 2018-09-11 | 祝锐 | A kind of portable white heart shaddock stripper unit |
| CN109479522A (en) * | 2018-12-27 | 2019-03-19 | 甘肃农业大学 | A kind of fruit picking robot and its picking method |
| CN111656959A (en) * | 2020-07-27 | 2020-09-15 | 荆门掇刀双井刘伟西瓜专业合作社 | A pick device for watermelon |
| CN112743566A (en) * | 2021-03-05 | 2021-05-04 | 广东海洋大学 | Pulling-in type mechanical claw capable of changing form |
| CN112976029A (en) * | 2021-03-11 | 2021-06-18 | 南京农业大学 | Soft electrostatic adhesion coating type bionic octopus manipulator |
| CN113330915A (en) * | 2021-05-26 | 2021-09-03 | 华南农业大学 | Self-adaptive cotton harvesting method based on binocular vision recognition and intelligent mechanical harvesting device |
| CN114402806A (en) * | 2022-02-18 | 2022-04-29 | 湖北汽车工业学院 | Spherical fruit picking robot and picking method |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210337734A1 (en) * | 2018-10-08 | 2021-11-04 | Advanced Farm Technologies, Inc. | Autonomous crop harvester |
| US12004451B2 (en) * | 2018-10-08 | 2024-06-11 | Advanced Farm Technologies, Inc. | Autonomous crop harvester |
| CN115643902A (en) * | 2022-11-03 | 2023-01-31 | 华南农业大学 | Litchi paired-rake accurate vibration harvesting equipment and harvesting method thereof |
| CN115643902B (en) * | 2022-11-03 | 2024-05-10 | 华南农业大学 | Litchi targeting accurate vibration harvesting equipment and harvesting method thereof |
| CN116079712A (en) * | 2022-12-01 | 2023-05-09 | 杭州电子科技大学 | Three-dimensional space clamping height control method of mechanical arm |
| CN115937314A (en) * | 2022-12-23 | 2023-04-07 | 南京林业大学 | Camellia oleifera fruit growth posture detection method |
| CN115937314B (en) * | 2022-12-23 | 2023-09-08 | 南京林业大学 | Method for detecting growth posture of oil tea fruits |
| CN115781766A (en) * | 2023-02-13 | 2023-03-14 | 安徽文达信息工程学院 | Mechanical arm with visual positioning system |
| CN116439018A (en) * | 2023-05-05 | 2023-07-18 | 仲恺农业工程学院 | Seven-degree-of-freedom fruit picking robot and picking method thereof |
| CN116439018B (en) * | 2023-05-05 | 2024-01-02 | 仲恺农业工程学院 | Seven-degree-of-freedom fruit picking robot and picking method thereof |
| CN116985270A (en) * | 2023-07-19 | 2023-11-03 | 玉溪师范学院 | Bionic fossil repairing appliance |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115250745B (en) | 2024-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115250745B (en) | Full-automatic fruit picking robot and picking method based on vision technology | |
| Sarig | Robotics of fruit harvesting: A state-of-the-art review | |
| US20200376656A1 (en) | Mobile Robot Morphology | |
| CN108401685A (en) | A kind of spheral fruit picking robot | |
| KR100784830B1 (en) | Harvesting robot system for bench cultivation type strawberry | |
| CN113330915B (en) | Self-adaptive cotton harvesting method based on binocular vision recognition and intelligent mechanical harvesting device | |
| CN108811766B (en) | Man-machine interactive greenhouse fruit and vegetable harvesting robot system and harvesting method thereof | |
| CN110249785A (en) | A kind of multi-robot string Tomato-harvesting robot and its picking method | |
| CN111587665B (en) | Four-degree-of-freedom multi-eye visual rotary-flying picking robot and picking method thereof | |
| CN112154778A (en) | Intelligent fruit picking operation vehicle | |
| CN109910011A (en) | A kind of mechanical arm barrier-avoiding method and mechanical arm based on multisensor | |
| CN114679963B (en) | Cross-ridge double-arm cooperation strawberry harvesting robot, control system and picking method | |
| CN205552540U (en) | Both arms work robot | |
| CN109699300A (en) | A kind of control system and its control method of intelligence citrus picking machine | |
| CN216058333U (en) | Intelligent movement fruit picking robot | |
| CN113812262B (en) | Tea-oil camellia fruit picking robot based on machine vision | |
| CN111201895A (en) | Picking robot | |
| CN114175922A (en) | Unmanned robot capable of picking fruits without damage | |
| CN216982644U (en) | Robot for picking fruits | |
| CN108718704B (en) | Man-machine interactive field fruit and vegetable harvesting robot system and harvesting method thereof | |
| CN208175422U (en) | A kind of spheral fruit picking robot | |
| Liu et al. | History and present situations of robotic harvesting technology: a review | |
| CN117158260A (en) | Automatic picking method and device for mushrooms | |
| CN116965236A (en) | Small watermelon picking robot and picking method based on greenhouse three-dimensional cultivation mode | |
| CN212520048U (en) | Four-degree-of-freedom multi-vision rotary flying type picking robot |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |