Open AccessArticle

Manipulability-Aware Task-Oriented Grasp Planning and Motion Control with Application in a Seven-DoF Redundant Dual-Arm Robot

Department of Electrical and Computer Engineering, Tamkang University, 151 Yingzhuan Road, Tamsui District, New Taipei City 25137, Taiwan

Author to whom correspondence should be addressed.

Electronics 2024, 13(24), 5025; https://doi.org/10.3390/electronics13245025

Submission received: 12 November 2024 / Revised: 10 December 2024 / Accepted: 17 December 2024 / Published: 20 December 2024

Download

Browse Figures

Figure 1
Architecture of the proposed M-aware grasp planning and motion control system. "> Figure 2
The JL-score plot of the joint limits evaluation function proposed in (a) previous studies [<a href="#B22-electronics-13-05025" class="html-bibr">22</a>,<a href="#B23-electronics-13-05025" class="html-bibr">23</a>] and (b) this work. The evaluation function in (a) features an unbounded value range, making it challenging to quantitatively assess the degree of joint limits. In contrast, the function in (b) has a bounded value range between 0 and 1, simplifying the quantitative assessment of the degree of joint limits. "> Figure 3
System configuration of the lab-made 7-DoF redundant dual-arm robot used in this study. Each arm has an additional prismatic joint to adjust its z-axis position to extend the workspace along the z0-axis. "> Figure 4
The S-score plot of the proposed singularity evaluation functions (15) and (16). These plots illustrate that the singularity evaluation of the robot arm is treated as a joint limit evaluation of its wrist singularity factors. Consequently, the S-score of the evaluation function remains within the bounded range of 0 to 1. "> Figure 5
System architecture of the proposed M-aware grasp planning method, which aims to determine maximum M-score grasp poses for the robot arm to perform object grasping tasks. "> Figure 6
The result of each step in the TOGD method: (a) the input RGB image, (b) the result of Mask R-CNN, (c) the affordances of all segmented objects (the darker regions), and (d) the feasible grasp rectangles. "> Figure 7
The state definition of the grasp rectangle used in this work. "> Figure 8
Definition of position and orientation errors used in grasp matching: (a) position error and (b) orientation error. "> Figure 9
Illustration of (a) the definition of the target pose vector used in this study and (b) the proposed grasp pose transformation to transform a target pose vector from the camera frame to the robot base frame. "> Figure 10
The hardware equipment and environment settings used in the experiments of this study. "> Figure 11
Three target objects used in the experiment and their default affordance masks: (a) a PET bottle with cap and body masks, (b) a cup with mouth and body masks, and (c) a scraper with handler and head masks. "> Figure 12
Comparison between the proposed TOGD method and the existing Grasp R-CNN method: (a) Input image showing two target objects (bottle and cup) and three non-target objects (vacuum gripper, ruler, and screwdriver), (b) task-agnostic grasp detection results of the Grasp R-CNN method, presenting multiple possible grasp rectangles for target and non-target objects, (c) object detection results obtained by the Mask R-CNN method, and (d) task-oriented grasp detection results of the proposed TOGD method, showing multiple grasp rectangles only for the two target objects. "> Figure 13
A virtual 7-DoF dual-arm robot model was used to test the performance of all compared methods in the Gazebo simulator. "> Figure 14
Comparison results of the manipulability evolution of each control method in (a) Test 1 and (b) Test 2. "> Figure 15
Comparison results of the angle evolution of all joints in Test 3 (a) without and (b) with the proposed JOSI method. "> Figure 16
Experimental setup for verifying the performance of the proposed system, in which the planar platform in front of the robot is divided into five regions (regions A to E) for object grasping tests to verify the performance of the proposed M-aware grasp planning and motion control system. "> Figure 17
Demonstration of the proposed system controlling the dual-arm robot to perform a task-oriented pouring action: (a) system initialization; (b,c) the robot uses its left arm to grasp the bottle object; (d,e) the robot uses its right arm to grasp the cup object; (f) the robot prepares to perform the pouring action; (g–i) the robot performs the pouring action. ">

Versions Notes

Abstract

Task-oriented grasp planning poses complex challenges in modern robotics, requiring the precise determination of the grasping pose of a robotic arm to grasp objects with a high level of manipulability while avoiding hardware constraints, such as joint limits, joint over-speeds, and singularities. This paper introduces a novel manipulability-aware (M-aware) grasp planning and motion control system for seven-degree-of-freedom (7-DoF) redundant dual-arm robots to achieve task-oriented grasping with optimal manipulability. The proposed system consists of two subsystems: (1) M-aware grasp planning; and (2) M-aware motion control. The former predicts task-oriented grasp candidates from an RGB-D image and selects the best grasping pose among the candidates. The latter enables the robot to select an appropriate arm to perform the grasping task while maintaining a high level of manipulability. To achieve this goal, we propose a new manipulability evaluation function to evaluate the manipulability score (M-score) of a given robot arm configuration with respect to a desired grasping pose to ensure safe grasping actions and avoid its joint limits and singularities. Experimental results demonstrate that our system can autonomously detect the graspable areas of a target object, select an appropriate grasping pose, grasp the target with a high level of manipulability, and achieve an average success rate of about 98.6%.

Keywords:

redundant dual-arm robots; manipulability awareness; task-oriented grasp planning; grasp motion control; joint limits and singularity avoidance

1. Introduction

In the development of intelligent robotic systems, robot arms play an important role due to their high levels of efficiency, flexibility, and precision in industrial applications, such as object grasping, object pick-and-place tasks, component assembly, etc. For a robot arm to handle various tasks with a high performance, it is necessary to develop an efficient object grasp planning algorithm [1,2,3,4,5], which aims to generate the best pose for the end-effector of the robot to stably grasp the object to perform subsequent tasks. According to [5], current object grasp planning methods for robotic arms can be divided into two categories: task-agnostic [4] and task-oriented [5]. For task-agnostic grasping, the robot arm only considers grasping objects in highly stable poses for application in object pick-and-place tasks. In contrast, for task-oriented grasping, the robot arm not only needs to stably grasp the object, but also needs to select the most appropriate affordance part of the object according to the subsequent manipulation task to be performed. Furthermore, the types, sizes, shapes, and materials of the target objects in grasp planning tasks are usually different and diverse. Therefore, determining the most suitable grasping posture of the robot arm according to the grasping affordances of different objects is also one of the core issues of task-oriented grasping. Note that in robotics, the term “affordances of an object” refers to the potential actions that a robot can perform in a given situation with an object based on its properties [6]. Understanding affordances enables robots to effectively recognize and utilize objects in a variety of tasks, such as grasping, manipulating, or using objects as tools. We encourage interested readers to refer to [6] for more details on the concept of object affordances.

In this paper, we focus on task-oriented and manipulability-optimized grasp planning and control for multi-arm robotic systems. The research goal is for a multi-arm robot to determine the arm with the highest level of manipulability and grasp the correct affordance of an object of interest (OOI) with the most dexterous grasping pose. To achieve this goal, we integrated deep learning, task-oriented grasp planning, and manipulability maximization techniques to improve the dexterous manipulation ability of the robot arm. The main contributions of this paper are as follows:

To quantitatively evaluate the manipulability of the robot arm, a new manipulability evaluation function is proposed to estimate the M-score of a given robot arm configuration with respect to a desired grasping pose, ensuring that the object grasping pose of the robot continuously and simultaneously moves away from both its joint limits and singularity points;
Based on the proposed manipulability evaluation function, we propose a novel M-aware grasp planning method, which consists of two key components: deep-learning-based task-oriented grasp detection (TOGD) and M-aware grasp pose estimation. Together, these components enable task-oriented grasping while optimizing the manipulability of the robot arm;
For the motion control of redundant robot arms, we introduce a novel M-aware motion control method, which consists of two algorithms: manipulability incremental redundant angle selection (MIRAS) and joint over-speed inhibition (JOSI). These algorithms aim to improve the M-score and suppress the joint over-speed of the redundant robot arm, respectively, thereby improving its manipulability. The proposed JOSI method is also applicable to non-redundant robots;
By integrating the proposed M-aware grasp planning and M-aware motion control methods, we implement a novel M-aware grasp planning and motion control system, which allows the redundant multi-arm robot to perform task-oriented grasping and grasping control with a high level of manipulability and safety.

The proposed system has been validated on a lab-made redundant dual-arm robot. Experimental results show that our system can autonomously detect the graspable affordance parts of the target object, select an appropriate grasping pose, and grasp the target with a high level of manipulability and an average success rate of about 98.6%.

The rest of this paper is organized as follows. Section 2 reviews the related work in the field of task-oriented grasp planning and manipulability optimization. Section 3 introduces the architecture of the proposed system. Section 4 presents the proposed manipulability evaluation function and the definition of the M-score used in this study. Section 5 presents the proposed M-aware grasp planning method, which consists of TOGD and M-aware grasp pose estimation stages. Section 6 presents the proposed M-aware motion control method, which consists of MIRAS and JOSI algorithms for the motion control of the redundant robot arm. Section 7 reports several experimental results to validate the performance of the proposed system on a lab-made redundant dual-arm robot. Section 8 summarizes the contributions of this work.

2. Related Work

As mentioned in the previous section, current object grasp planning methods for manipulators can be divided into two categories: task-agnostic and task-oriented. In recent years, due to the rise of deep learning technology, various object grasp planning methods based on deep learning have been proposed in the literature. For task-agnostic grasping, Kumra and Kanan [7] used RGB and depth images as the input of a residual convolutional neural network (CNN) to predict grasp rectangles of an OOI. Guo et al. [8] proposed a grasp detection deep network to detect the grasp rectangle of an OOI based on visual and tactile sensing. Zeng et al. [9] proposed a multi-affordance grasping framework that uses fully convolutional networks (FCNs) to estimate the pixel-wise affordances of an OOI for the determination of the optimal picking action agnostic to the object identity. As for task-oriented grasping, Iizuka and Hashimoto [10] proposed an RGB-D-based semantic segmentation method to recognize the part affordance of an OOI and generate corresponding semantic grasping parameters. Chu et al. [11] proposed a deep learning framework to segment affordance maps of object parts for a task-oriented manipulation application.

In recent years, point-cloud-based grasp detection methods have been developed to detect six-dimensional (6-D) grasping poses directly from object point clouds, taking into account the shape of the object. For example, Shao and Hu [12] proposed a special CNN that combines RGB images and depth information to predict picking regions without recognition or pose estimation. They found that the RGB points input yielded the best performance. Liang et al. [13] proposed an end-to-end grasp evaluation model called PointNetGPD for localizing robot grasp configurations directly from object point clouds. Their experimental results show that even with sparse point clouds, the proposed PointNetGPD can capture the complex geometry of the contact region between the gripper and the object. Liu and Cao [14] presented a grasp pose detection method based on point cloud shape simplification for grasping unknown objects. The authors also proposed a grasp strategy based on these simplified shape features to suggest the main graspable parts of unknown objects. Yin et al. [15] proposed an RGB-D-based robotic grasping method that integrates a CNN-based instance segmentation network with clustering and plane extraction algorithms. This combination produces more stable and superior results compared to end-to-end grasping networks. In a recent review paper [16], it was highlighted that for situations in which a robot arm is required to grasp an object from a single angle, 2D image-based grasping is sufficient. In contrast, if the robot needs to grasp objects from multiple angles, a 6-D space grasping method is required. Given this suggestion, this study specifically focuses on the issue of grasping a target object from an optimal angle and attempts to develop a grasp planning method based on deep learning and RGB-D images to meet the requirement of task-oriented manipulation.

Although the above research studies successfully integrated deep learning and robot arm grasp planning, they did not consider the hardware constraints of the robot arm, such as the geometric constraints of the gripper. To address this problem, Mousavian et al. [17] proposed Grasp Sampler, Grasp Evaluator, and Grasp Refinement networks based on variational autoencoders and PointNet architecture. The Grasp Sampler aims to generate diverse sets of candidate grasps, and the Grasp Evaluator evaluates the quality of each candidate grasp based on the object point cloud and the gripper point cloud. The quality of the final grasp pose is improved iteratively using the Grasp Refinement network. Similarly, Choi et al. [18] considered the grasp detection problem as a robot arm approaching direction selection problem and employed an FCN to exploit a grasp quality network for evaluating the quality of an approaching direction based on a geometry-based prior derivative-free optimization method. However, both grasp planning methods proposed in [17,18] are task-agnostic and only consider the geometric constraints of the gripper. In general, the object grasp planning algorithm also needs to consider the motion constraints of the robotic arm to maximize its manipulability during object grasping. However, the above-mentioned deep-learning-based robot arm grasp planning methods cannot meet this requirement of manipulability maximization.

To measure the quality of a grasp pose for a given robot arm configuration, Yoshikawa [19] proposed a quality metric based on the volume of a high-dimensional manipulability ellipsoid. Several grasp planning algorithms with manipulability optimization have been published based on this quality metric. For instance, Huo and Baron [20] proposed a new kinetostatic performance index based on the manipulability metric to evaluate the kinematic quality of robot poses to avoid joint limits and singularities when performing arc-welding tasks. Rozo et al. [21] proposed a manipulability transfer framework that allows robots to learn and reproduce desired manipulability ellipsoids from expert demonstrations. However, these existing manipulability-aware grasp planning algorithms are also task-agnostic. This shortcoming motivated us to develop a manipulability-aware, task-oriented grasp planning method based on deep learning techniques.

3. Overview of the Proposed System

Figure 1 shows the proposed M-aware grasp planning and motion control system, which can be divided into M-aware grasp planning and M-aware motion control subsystems. The M-aware grasp planning subsystem includes TOGD and M-aware grasp pose estimation methods, which are responsible for determining task-oriented grasp poses for a given object. The TOGD method first uses two neural networks to detect multiple task-oriented grasp rectangle candidates from RGB images. Each grasp rectangle is defined by five parameters, including the object’s two-dimensional centroid position, width, height, and rotation angle. Then, the M-aware grasp pose estimation method takes each candidate as the input to generate several corresponding six-degree-of-freedom grasp pose candidates and determines the optimal task-oriented grasp pose with the most suitable robot arm to execute the grasp task. Note that in this study, the optimal grasp pose is defined as the pose that provides the maximum manipulability for the selected robot arm among all combinations between robot arms and grasp pose candidates. To quantitatively evaluate the manipulability of the robot arm with respect to a specific grasp pose, we formulate a novel M-score function that takes into account both the joint limits and the wrist singularity of redundant robot arms. Its purpose is to measure how far a given grasp pose is away from the joint limits and singularities of the robot arm. The technical details of the proposed M-score function and M-aware grasp planning subsystem are presented in Section 4 and Section 5, respectively.

On the other hand, the M-aware motion control subsystem consists of MIRAS and JOSI methods, which are responsible for maintaining the manipulability of the robot arm during motion control. The proposed MIRAS method is a null-space motion control method for the redundant robot arm to choose an appropriate redundant angle that helps the robot avoid both joint limits and wrist singularities. However, when the robot arm is operating in the area around one of its singular points, the motion controller may output higher-joint-velocity commands, resulting in the sudden acceleration of the robot. To address this issue, the proposed JOSI method aims to ensure that the joint velocity commands for each joint are within the hardware specification limits at all times. This method works for both non-redundant and redundant robots. The technical details of the proposed M-aware motion control subsystem are presented in Section 6.

Remark 1.

In general, task-oriented grasping tasks need to satisfy two types of constraints [5]. First, the robot gripper must stably hold the target object. Second, the grasping pose must satisfy a set of physical and semantic constraints for a specific task, including the joint limits and singularities of the robot arm and the affordances of the object. In the proposed system, the M-aware grasp planning subsystem handles object semantic constraints, while the M-aware motion control subsystem addresses the physical constraints of the robot arm. To illustrate this, consider a pouring task involving a bottle and a cup as target objects. In this case, the grasping pose must obey physical constraints to avoid the joint limits and singularities of the robot arm, while the semantic constraints are limited to the regions of the bottle body and the cup body, respectively. The proposed MIRAS method utilizes the M-score function to handle the physical constraints of the robot arm, ensuring manipulability while avoiding physical limitations. Meanwhile, the proposed TOGD method adopts deep-learning-based object segmentation to provide basic semantic constraints on target objects by generating grasp rectangle candidates that comply with these constraints.

4. The Proposed Manipulability Evaluation Function

In this section, we present a novel M-score function to evaluate the manipulability of a 7-DoF redundant robot arm with respect to its joint limits and wrist singularities. The following subsections sequentially describe the proposed method for evaluating the degree of joint limits, wrist singularity, and manipulability of a 7-DoF redundant robot arm.

4.1. Joint Limits Evaluation Function

Let

θ^{j} = [\begin{matrix} θ_{1}^{j} & θ_{2}^{j} & \dots & θ_{N_{D O F}}^{j} \end{matrix}]

denote the vector of joint angles of the j-th N-DoF redundant robot arm of the system. The hardware constraints of each joint angle of the j-th robot arm are recorded in a 2-by-N_DOF matrix such that the following is true:

C_{a r m}^{j} = [\begin{matrix} θ_{1, \min}^{j} & θ_{2, \min}^{j} & \dots & θ_{N_{D O F}, \min}^{j} \\ θ_{1, \max}^{j} & θ_{2, \max}^{j} & \dots & θ_{N_{D O F}, \max}^{j} \end{matrix}],

where

θ_{k, \min}^{j}

and

θ_{k, \max}^{j}

denote the minimum and maximum values of the k-th joint angle of the j-th robot arm, respectively. In previous studies [22,23], the joint limits score (JL-score) of the j-th robot arm was evaluated using the following evaluation function:

H (θ^{j}, C_{a r m}^{j}) = \sum_{k = 1}^{N_{D O F}} \frac{1}{4} \frac{{(θ_{k, \max}^{j} - θ_{k, \min}^{j})}^{2}}{(θ_{k, \max}^{j} - θ_{k}^{j}) (θ_{k}^{j} - θ_{k, \min}^{j})} .

(1)

The value range of the evaluation function (1) is unbounded. Figure 2a illustrates the plot of this function, showing that the JL-score approaches 1.0 as the robot’s joint angles move farther from their respective joint limits. In contrast, when the robot pose has at least one joint angle close to its limit, the JL score diverges rapidly, reflecting that it is close to the joint constraints. However, since the range of evaluation function (1) is not confined to a specific interval, it becomes challenging to quantitatively assess the degree of joint limits. To address this issue, we propose a novel joint limits evaluation function

f_{J L}^{j}

to estimate the JL-score of the k-th joint angle of the j-th robot arm as follows:

f_{J L} (θ^{j}, C_{a r m}^{j}, k) = [1 - {(\frac{|θ_{k}^{j} - θ_{k, m i d}^{j}|}{θ_{k, \max}^{j} - θ_{k, m i d}^{j}})}^{4}] [u (θ_{k}^{j} - θ_{k, \min}^{j}) - u (θ_{k}^{j} - θ_{k, \max}^{j})],

(2)

where

θ_{k, m i d}^{j}

is the middle value of the k-th joint angle of the j-th robot arm calculated by the following:

θ_{k, m i d}^{j} = θ_{k, \max}^{j} - (θ_{k, \max}^{j} - θ_{k, \min}^{j}) / 2,

(3)

and the function u(x) in (2) is the unit step function defined as follows:

u (x) = \{\begin{matrix} 1 & i f x \geq 0, \\ 0 & i f x < 0 . \end{matrix}

(4)

Figure 2b illustrates the plot of the proposed joint limits evaluation function (2). It is clear that the value range of this function is between 0 and 1. A value of 0 indicates that the joint angle is at or beyond its joint limit, while a higher value indicates that the joint angle is farther from its limit.

It should be noted that the proposed evaluation function (2) only estimates the JL-score for one joint angle of the robot arm. To estimate the JL-score of the whole robot arm, the following evaluation function is proposed based on (2) such that the following is true:

S_{J L} (θ^{j}, C_{a r m}^{j}) = \min \{f_{J L} (θ^{j}, C_{a r m}^{j}, k) |k = 1, 2, \dots, N_{D O F}\},

(5)

which means that for the j-th robot arm, we consider the joint angle with the minimum JL-score as the degree of its joint limits.

4.2. Singularity Evaluation Function

For serial manipulators, singularity issues are divided into three types: shoulder singularity, elbow singularity, and wrist singularity [24]. When the robot arm moves past a singularity in the task space trajectory, the angular velocity of at least one joint of the robot arm will instantaneously over-speed, causing the robot system to stop due to safety precautions. To detect the singularity, Yoshikawa [19] proposed a manipulability index based on the determinant of the Jacobian matrix to quantify the singularity level of the robot arm. However, this manipulability index does not serve as a measure of the distance to the robot singularity, since the determinant value of the matrix does not serve as an indicator of the distance to a point [25]. This challenge motivates us to develop a novel method to evaluate the distance between a given grasping pose and the singularity of the redundant robot arm.

Figure 3 shows the system configuration of the lab-made 7-DoF redundant dual-arm robot used in this study. Table 1 specifies the Denavit–Hartenberg (D-H) parameters (a_i, α_i, d_i, and q_i, where i = 0, 1, 2, …, 7) for the left arm of the robot. In this context, a_i denotes the link length of the i-th joint, α_i is the link twist, d_i is the link offset, and q_i is the joint angle relative to the base frame. It is worth noting that we set two prismatic joints on the z₀-axis of the robot’s base joint. This configuration allows the height position of each arm to be adjusted independently to expand the robot’s workspace. Since this design only changes the z-axis position of the end-effector of each arm, the kinematic analysis of each arm can focus on the configuration of joints 1 to 7. For each arm, the first three joints (J1–J3) and the last three joints (J5–J7) are configured as a spherical wrist. According to [26], this configuration helps to simplify the Jacobian matrix of the robot arm and allows for the decoupling of the singularity of the robot arm into position and orientation singularities. For the rear spherical wrist J57, we calculate the 3-by-3 orientation Jacobian matrix of the last three joints (J5–J7) as follows:

J_{J 57}^{o r i} = [\begin{matrix} 0 & s_{5} & - c_{5} s_{6} \\ 0 & - c_{5} & - s_{5} s_{6} \\ 1 & 0 & c_{6} \end{matrix}],

(6)

where

c_{i} \equiv \cos θ_{i}

s_{i} \equiv \sin θ_{i}

, and

θ_{i}

is the joint angle of the i-th joint. It is obvious that when the sixth joint satisfies

s_{6} = 0

, the wrist singularity occurs. Therefore, the wrist singularity factor of the robot arm is given by

θ_{6} = 0^{\circ}, \pm 180^{\circ}

On the other hand, we also calculate the 3-by-4 position Jacobian matrix for the first four joints (J1–J4) using the joint angles

θ_{1}

θ_{4}

as follows:

J_{J 14}^{p o s} = [\begin{matrix} J_{1} & J_{2} & J_{3} & J_{4} \end{matrix}],

(7)

where

J_{1} = [\begin{matrix} - (c_{1} s_{2} c_{3} + s_{1} s_{3}) (a_{4} c_{4} + a_{3}) - c_{1} c_{2} (a_{4} s_{4} + d_{3}) \\ 0 \\ (s_{1} s_{2} c_{3} - c_{1} s_{3}) (a_{4} c_{4} + a_{3}) + s_{1} c_{2} (a_{4} s_{4} + d_{3}) \end{matrix}],

(8)

J_{2} = [\begin{matrix} - s_{1} c_{2} c_{3} (a_{4} c_{4} + a_{3}) + s_{1} s_{2} (a_{4} s_{4} + d_{3}) \\ - s_{2} c_{3} (a_{4} c_{4} + a_{3}) - c_{2} (a_{4} s_{4} + d_{3}) \\ - c_{1} c_{2} c_{3} (a_{4} c_{4} + a_{3}) + c_{1} s_{2} (a_{4} s_{4} + d_{3}) \end{matrix}],

(9)

J_{3} = [\begin{matrix} (s_{1} s_{2} s_{3} + c_{1} c_{3}) (a_{4} c_{4} + a_{3}) \\ - c_{2} s_{3} (a_{4} c_{4} + a_{3}) \\ (c_{1} s_{2} s_{3} - s_{1} c_{3}) (a_{4} c_{4} + a_{3}) \end{matrix}],

(10)

J_{4} = [\begin{matrix} a_{4} [s_{1} (s_{2} c_{3} s_{4} - c_{2} c_{4}) - c_{1} s_{3} s_{4}] \\ - a_{4} (c_{2} c_{3} s_{4} + s_{2} c_{4}) \\ a_{4} [c_{1} (s_{2} c_{3} s_{4} - c_{2} c_{4}) + s_{1} s_{3} s_{4}] \end{matrix}] .

(11)

In Equations (8)–(11), the symbols a₃, a₄, and d₃ are the D-H parameters listed in Table 1. From the position Jacobian matrix, two forearm singular factors can be found as follows:

c_{2} = c_{3} = 0 or, equivalently, θ_{2} = θ_{3} = \pm 90^{\circ}, \pm 270^{\circ} .

(12)

c_{2} = s_{4} = 0 or, equivalently, θ_{2} = \pm 90^{\circ}, \pm 270^{\circ} and θ_{4} = \pm 0^{\circ}, \pm 180^{\circ} .

(13)

Observing the above two conditions, it is obvious that there is a common singularity factor

c_{2} = 0

, which is also the wrist singularity factor of the 3-by-3 orientation Jacobian matrix of the forearm spherical wrist J13 as follows:

J_{J 13}^{o r i} = [\begin{matrix} 0 & - c θ_{1} & - s_{1} c_{2} \\ 1 & 0 & - s_{2} \\ 0 & s_{1} & - c_{1} c_{2} \end{matrix}] .

(14)

Based on this observation, we can simplify the evaluation of the forearm singularity factor of the robot arm to

θ_{2} = \pm 90^{\circ}, \pm 270^{\circ}

. It should be noted that since the left and right robot arms have the same configuration, the above singularity analysis results also apply to the right robot arm.

Based on the above analysis results and applied to the singularity distance evaluation of the grasping pose of the robot arm, we extend the concept of the joint limits evaluation function (2) to the two wrist singular factors such that the following is true:

S_{J 13} (θ_{2}^{j}) = [1 - {(\frac{|θ_{2}^{j} - S_{J 13, m i d}^{j}|}{S_{J 13, \max}^{j} - S_{J 13, m i d}^{j}})}^{4}] [u (θ_{2}^{j} - S_{J 13, \min}^{j}) - u (θ_{2}^{j} - S_{J 13, \max}^{j})],

(15)

S_{J 57} (θ_{6}^{j}) = [1 - {(\frac{|θ_{6}^{j} - S_{J 57, m i d}^{j}|}{S_{J 57, \max}^{j} - S_{J 57, m i d}^{j}})}^{4}] [u (θ_{6}^{j} - S_{J 57, \min}^{j}) - u (θ_{6}^{j} - S_{J 57, \max}^{j})],

(16)

where the parameters

(S_{J 13, \min}^{j}, S_{J 13, \max}^{j})

and

(S_{J 57, \min}^{j}, S_{J 57, \max}^{j})

denote the minimum and maximum angles of the 2nd and 6th joints of the j-th robot arm, respectively.

S_{J 13, m i d}^{j}

and

S_{J 57, m i d}^{j}

, respectively, are the middle value of the parameters

(S_{J 13, \min}^{j}, S_{J 13, \max}^{j})

and

(S_{J 57, \min}^{j}, S_{J 57, \max}^{j})

given by the following:

S_{J 13, m i d}^{j} = S_{J 13, \max}^{j} - (S_{J 13, \max}^{j} - S_{J 13, \min}^{j}) / 2,

(17)

S_{J 57, m i d}^{j} = S_{J 57, \max}^{j} - (S_{J 57, \max}^{j} - S_{J 57, \min}^{j}) / 2 .

(18)

Here, the values of these four parameters depend on the wrist singularity factor of the 2nd and 6th joints of the j-th robot arm as follows:

(S_{J 13, \min}^{j}, S_{J 13, \max}^{j}) = \{\begin{array}{l} (- 90^{\circ} {, 90}^{\circ}), & i f (- 90^{\circ} \leq θ_{2}^{j} \leq 90^{\circ}), \\ (90^{\circ} {, 270}^{\circ}), & i f (90^{\circ} < θ_{2}^{j} \leq 18 0^{\circ}), \\ (- 270^{\circ}, - 90^{\circ}), & i f (- 18 0^{\circ} < θ_{2}^{j} < - 90^{\circ}), \end{array}

(19)

(S_{J 57, \min}^{j}, S_{J 57, \max}^{j}) = \{\begin{array}{l} (0^{\circ} {, 180}^{\circ}), & i f (0^{\circ} \leq θ_{6}^{j} \leq 180^{\circ}), \\ (- 180^{\circ} {, 0}^{\circ}), & i f (- 180^{\circ} < θ_{6}^{j} < 0^{\circ}) . \end{array}

(20)

Equations (15) and (16) indicate that the singularity evaluation of the j-th robot arm is addressed as a joint limit evaluation of its wrist singularity factors. This approach ensures that the singularity score (S-score) of the evaluation function remains within the bounded range of 0 to 1, as shown in Figure 4.

Remark 2.

The difference between the proposed singularity evaluation method and the existing methods is twofold. First, according to [27], the existing Jacobian-based manipulability index can be extended to include singularity and joint limit evaluations by incorporating a penalty function that considers robot joint limits. However, even with this extension, such an evaluation function is still unbounded and cannot provide a realistic measure of how close to the singularity it is. Secondly, current singularity analysis methods based on inverse kinematics usually ignore the evaluation of robot joint limits [28]. In contrast, the proposed method can simultaneously evaluate the singularity and joint limits of the robot. Furthermore, the proposed evaluation function is bounded and provides a realistic measure of how close it is to the singularity and joint limits, as shown in Figure 2 and Figure 4.

4.3. Manipulability Evaluation Function

In order to comprehensively evaluate the JL-score and the S-score of the entire robot arm in the motion state, we combine the above three evaluation functions (5), (15), and (16) to propose the following manipulability evaluation function for redundant robot arms:

Q (θ^{j}, C_{a r m}^{j}) ≜ \min [S_{JL} (θ^{j}, C_{a r m}^{j}), S_{J 13} (θ_{2}^{j}), S_{J 57} (θ_{6}^{j})],

(21)

which can be used to evaluate the M-score of the j-th robot arm of the multi-arm robotic system. Note that since the proposed joint limits and singularity evaluation functions have the same range of values and are independent of each other, the minimum of the three functions can be used as the overall M-score of the robot arm under the given joint angles and hardware constraints. The M-score function (21) provides us with a useful tool to evaluate the manipulability of the redundant robot arm in object grasping control applications.

5. The Proposed M-Aware Grasp Planning Method

Figure 5 shows the system architecture of the proposed M-aware grasp planning method, which can be divided into two stages, including (1) the TOGD stage and (2) the M-aware grasp pose estimation stage. In the first stage, the TOGD method is used to segment object affordance regions and detect feasible grasp rectangles. In the second stage, the M-aware grasp pose estimation method first generates corresponding 6-DOF grasp pose candidates based on the results of TOGD and the depth image of target objects. Next, the proposed M-score function is used to evaluate the manipulability of each grasping pose with respect to a given robot arm, and the grasping pose with the highest level of manipulability is selected for the robot arm to perform the object grasping task. Note that the proposed M-aware grasp planning method is also applicable to the case of multi-arm robotic systems.

5.1. TOGD Method

The proposed TOGD method is a deep-learning-based hybrid affordance segmentation method that integrates Mask R-CNN [29] with a grasp detection network [30] (called Grasp R-CNN in this paper) to segment object affordances and detect feasible grasp rectangles. It consists of two processes, including grasp rectangle detection and grasp matching.

5.1.1. Grasp Rectangle Detection

Figure 6 presents the result of each step in the TOGD method. Figure 6a shows an example of an input RGB image. In Figure 6b, the proposed TOGD method uses Mask R-CNN to detect the segmentation masks of each object. Next, a segmentation map that records the affordances of all segmented objects, represented by the darker regions, is generated from the results of Mask R-CNN (Figure 6c). Finally, the Grasp R-CNN is used to generate multiple feasible grasp rectangles from the segmented object affordances, as shown in Figure 6d.

In order to use Mask R-CNN to detect and segment the graspable affordances of target objects, we segment each target object into three instance regions during the training phase of the network model. For example, the bottle object in Figure 6b is divided into three instance areas: (1) representing the entire object, (2) representing the bottle cap, and (3) representing the bottle body. In this case, the bottle cap is specifically marked for the tool manipulation task, while the bottle body is designated for the robot to perform task-oriented grasping tasks. During the training process of the Grasp R-CNN model, we included 280 different object types into the training set to enhance the robustness of the model and ensure adaptability to various grasping tasks.

In this study, the state vector of the grasp rectangle is defined as

g = {\{x, y, w, h, φ\}}^{T}

, where x and y are the 2D centroid point positions of the grasp rectangle; w, h, and

φ

are the width, height, and the rotation angle of the grasp rectangle, respectively. Figure 7 illustrates the state definition of the grasp rectangle used in this work.

5.1.2. Grasp Matching

Based on the outputs of Mask R-CNN and Grasp R-CNN, we have to remove low-quality grasp rectangles predicted by Grasp R-CNN and find high-quality matches between the object affordance masks and the grasp rectangles, i.e., which grasp rectangle belongs to which object affordance mask. To achieve this purpose, we define two metrics, position error and orientation error, to evaluate the degree of matching between the affordance mask and the grasp rectangle. Figure 8a,b illustrate the definition of the position error and orientation error, respectively. In this work, the position error is defined as the distance between the center point of the affordance mask and the grasp rectangle. The orientation error is defined as the angle between the major vector of the grasp rectangle and the minor vector of the oriented bounding box of the affordance mask. Based on these two metrics, the proposed grasp matching process aims to find the best match of each affordance mask to a unique grasp rectangle with minimum position and orientation errors via an exhaustive search method. Otherwise, all unmatched grasp rectangles will be removed from the matching process.

5.2. M-Aware Grasp Pose Estimation Method

The proposed M-aware grasp pose estimation method takes each matched grasp rectangle as input to generate a corresponding grasp pose suitable for the robot arm to grasp the target object with maximum level of manipulability. As shown in Figure 5, the proposed M-aware grasp pose estimation method consists of three steps: (1) grasp pose generation, (2) grasp pose transformation, and (3) grasp quality evaluation. Each step is described in detail in the following subsections.

5.2.1. Grasp Pose Generation

Grasp pose generation aims to produce multiple 6-DoF grasp pose candidates based on the results of TOGD and the depth image containing the target object. This task consists of two processing steps. First, for each pair of matching affordance mask and grasp rectangle, we select their intersection region as the graspable region and apply a down-sampling method to generate multiple sub-regions in the graspable region. Each sub-region can be used to compute a corresponding normal vector in the graspable region based on the depth gradient obtained from the depth image. Second, as shown in Figure 9a, a target pose vector

[P_{t a r,} n_{t a r,} φ_{t a r}] \in ℝ^{7 \times 1}

is established for each sub-region, where P_tar is a 3D position vector of the center point of the grasp rectangle in the RGB-D camera frame; n_tar is the normal vector mentioned above; and φ_tar is the rotation angle of the grasp rectangle. Each target pose vector is used as a grasp pose candidate to represent a feasible 3D pose for the end-effector of the robot arm to grasp the object.

5.2.2. Grasp Pose Transformation

To control the robot arm for the desired target grasp pose, it is necessary to transform the target pose vector from the camera frame to the robot base frame. The transformation of the position vector P_tar can be simply achieved by applying a fixed transformation matrix from the camera frame to the robot base frame [31], denoted by

T_{c a m}^{b a s e} \in ℝ^{4 \times 4}

, such that the following is true:

{\hat{P}}_{t a r} = T_{c a m}^{b a s e} {\bar{P}}_{t a r},

(22)

where

{\bar{P}}_{t a r}

denotes the homogenous coordinate of the position vector P_tar. On the other hand, the orientation of the target pose must be transformed into a direction cosine matrix (DCM) relative to the robot base frame. To achieve this purpose, Rodrigues’ rotation formula serves as an important tool for calculating the rotation matrix required to rotate a specified angle around a specific axis. Figure 9b illustrates the concept of the proposed grasp pose transformation method, which consists of three steps. Let A_rot and φ_rot denote, respectively, the cross product vector and the angle between the normal vector n_tar and the Z-axis such that the following is true:

A_{r o t} = \frac{Z \times n_{t a r}}{‖Z \times n_{t a r}‖} and ϕ_{r o t} = {cos}^{- 1} (\frac{Z \cdot n_{t a r}}{‖Z‖ ‖n_{t a r}‖}),

(23)

where Z = [0, 0, 1]^T is a unit vector on the Z-axis; the symbols × and · denote the cross product and dot product of two vectors. Let φ_z be the angle between the normal vector n_tar and the cross vector A_rot. Then, the first step is to rotate the Z-axis by the angle φ_z to align the Y-axis with the A_rot vector. The second step is to rotate the A_rot vector by the angle φ_rot to align the Z-axis with the n_tar vector. The final step is to rotate n_tar vector by the angle φ_tar to establish the stable grasping pose of the two-finger gripper. Therefore, the proposed grasp pose transformation method to transform the orientation vector [n_tar, φ_tar] in the camera frame into a DCM representation in the robot base frame can be expressed as follows:

R_{D C M} = R_{r o d} (n_{t a r}, φ_{t a r}) R_{r o d} (A_{r o t}, φ_{r o t}) R_{z} (φ_{z}),

(24)

where the two parametric rotation matrices R_rod(n,θ) and R_z(θ) are given by the following:

R_{r o d} (n, θ) = I_{3} + (1 - c o s θ) [\begin{matrix} n_{x}^{2} & n_{x} n_{y} & n_{x} n_{z} \\ n_{x} n_{y} & n_{y}^{2} & n_{y} n_{z} \\ n_{x} n_{z} & n_{y} n_{z} & n_{z}^{2} \end{matrix}] + \sin θ [\begin{matrix} 0 & - n_{z} & n_{y} \\ n_{z} & 0 & - n_{x} \\ - n_{y} & n_{x} & 0 \end{matrix}],

(25)

R_{z} (θ) = [\begin{matrix} \cos θ & - \sin θ & 0 \\ \sin θ & \cos θ & 0 \\ 0 & 0 & 1 \end{matrix}] .

(26)

where n = [n_x, n_y, n_z]^T is a unit vector in

ℝ^{3}

, and I₃ is a 3-by-3 identity matrix. Therefore, the grasp pose vector of the robot arm can be obtained by taking the first three elements of the position vector

{\hat{P}}_{t a r}

and the three Euler angles of the rotation matrix R_DCM.

Remark 3.

Although the proposed TOGD method can generate multiple feasible 2D grasp rectangle candidates, these candidates do not consider the surface shape of the target object, which may reduce the success rate of object grasping. To deal with this problem, the proposed M-aware grasp pose estimation method simultaneously utilizes the depth image of the target object and its corresponding grasp rectangle candidates for grasping pose estimation. Specifically, the proposed method incorporates the 3D point cloud of the object surface from the graspable region of the target object during grasp pose estimation. The surface normal vector n_tar in each target pose vector is obtained based on this local 3D point cloud. This approach improves the accuracy of grasping pose estimation, thereby improving the success rate of object grasping. A similar approach can be found in [9], which aggregates affordance predictions from multi-view RGB-D images onto a combined 3D point cloud to estimate the surface normal vector for each 3D point. Instead, our method aggregates grasp rectangle candidates from a single-view RGB-D image onto the local 3D point cloud of the object surface to estimate the surface normal vector for each grasp rectangle.

5.2.3. Grasp Quality Evaluation

In order to select the best grasp pose from all grasp pose candidates for a target object, we have to quantitatively evaluate the grasp quality of each candidate. Thanks to the proposed M-score function (21), we can efficiently achieve this based on the M-score of each grasp pose candidate. Let

{\hat{θ}}_{i}^{j} = [\begin{matrix} {\hat{θ}}_{i, 1}^{j} & {\hat{θ}}_{i, 2}^{j} & \dots & {\hat{θ}}_{i, N_{D O F}}^{j} \end{matrix}]

denote the desired joint angle vector of the j-th robot arm with respect to the i-th grasp pose candidate, and let

{\hat{θ}}_{i, k}^{j}

denote the k-th element of the joint angle vector

{\hat{θ}}_{i}^{j}

. Given the hardware constraints of the j-th robot arm

C_{a r m}^{j}

, the M-score of the j-th robot arm for the i-th grasp pose candidate can be computed using (21) such that the following is true:

{M_{i}^{j}|}_{C_{a r m}^{j}} ≜ Q ({\hat{θ}}_{i}^{j}, C_{a r m}^{j}),

(27)

where the desired joint angle vector

{\hat{θ}}_{i}^{j}

can be obtained from the inverse kinematics model of the j-th robot arm given the i-th grasp pose candidate. Based on the robot–pose matching M-score function (27), a 2D exhaustive search method is proposed to find the best matching indices (i^*, j^*) between the best grasping pose and the robot arm with the maximum M-score such that the following is true:

(i^{*}, j^{*}) = \arg \max_{j \in {1, 2, \dots, J}} \{\max_{i \in {1, 2, \dots, I}} \{{M_{i}^{j}|}_{C_{a r m}^{j}}\}\},

(28)

where I and J are the number of grasp pose candidates and robot arms, respectively. Therefore, the proposed M-aware grasp planning module can effectively help the multi-arm robotic system to determine the most suitable robot arm to perform grasping tasks with maximum level of manipulability.

6. The Proposed M-Aware Motion Control Method

In this section, a new approach for the 7-DoF robot arm to maximize its manipulability is introduced. Based on the aforementioned M-score function (21), a novel MIRAS control method for 7-DoF robot arms is proposed to avoid both joint limits and wrist singularity problems. Next, a novel JOSI method is designed to suppress the joint over-speed of the robot arm.

6.1. MIRAS Method

When the robot arm moves along the trajectory, the joint limits and wrist singularity are two crucial problems that need to be solved. For the 7-DoF robot arm to simultaneously avoid its joint limits and wrist singularity while performing task space trajectory planning, the proposed MIRAS method aims to select a redundant angle that increases the M-score of the robot arm during motion based on the proposed M-score function (21). The pseudocode of the proposed MIRAS method is shown in Algorithm 1. Note that the “Inverse Kinematics” sub-function in Algorithm 1 is implemented using an existing analytical solution for the inverse kinematics of a redundant 7-DoF robot arm [32]. Since the inverse kinematics solution is not unique, we calculate all feasible solutions as described in [32] and then select the one that minimizes joint motion as the optimal solution. We encourage interested readers to refer to [32] for more technical details on the analytical solution of the inverse kinematics of the redundant 7-DoF robot arm.

The proposed MIRAS method can be applied to each robot arm in the multi-arm robotic system. For the j-th robot arm, it requires its current joint angles

θ_{curr}^{j}

and redundant angle

ϕ_{curr}^{j}

, followed by the position P_next and orientation O_next at the next time step. During initialization, it needs to set a maximum variation value of the redundant angle ϕ_max and the hardware constraints matrix of the j-th robot arm

C_{a r m}^{j}

Let m_curr denote the manipulability value relative to the current joint angles

θ_{curr}^{j}

of the j-th robot arm. To determine the variation value in the redundant angle, we evaluate the change in M-score compared to the m_curr value in two different directions

ϕ_{d i r} \in \{1, - 1\}

. If one of the directions increases the M-score above the m_curr value, we update the redundant angle in that direction by adding an adaptive variation value based on the m_curr value such that

Δ ϕ_{curr}^{j} = ϕ_{d i r}^{*} ϕ_{\max} (1 - m_{curr})

, where

ϕ_{d i r}^{*} \in \{1, 0, - 1\}

indicates the rotation direction to obtain a better redundant angle to increase the manipulability of the robot arm. Finally, the proposed MIRAS method outputs manipulability incremental redundant angle of the j-th robot arm at the next time step by updating the current redundant angle such that

ϕ_{next}^{j} = ϕ_{curr}^{j} + Δ ϕ_{next}^{j}

Algorithm 1. Pseudocode of the proposed MIRAS method

01: Input: Current joint angles of the j-th robot arm

θ_{curr}^{j}

,
02: Current redundant angle of the j-th robot arm

ϕ_{curr}^{j}

,
03: Position P_next and orientation O_next at the next time step (n + 1)
04: Output: Redundant angle of the j-th robot arm at the next time step

ϕ_{next}^{j}

05: Initialize: Maximum variation value of the redundant angle ϕ_max,
06: Hardware constraints matrix of the j-th robot arm

C_{a r m}^{j}

07: Begin: Evaluate the current manipulability:

m_{curr} \leftarrow Q (θ_{curr}^{j}, C_{a r m}^{j})

by Formula (21)
08:

θ_{tar}^{j} \leftarrow Inverse Kinematics (P_{next}, O_{next}, θ_{curr}^{j}, ϕ_{curr}^{j})

09:

m_{tar} \leftarrow Q (θ_{tar}^{j}, C_{a r m}^{j}), ϕ_{d i r}^{*} \leftarrow 0

10: for each

ϕ_{d i r} \in \{1, - 1\}

do
11:

θ_{tar}^{j} \leftarrow Inverse Kinematics (P_{next}, O_{next}, θ_{curr}^{j}, ϕ_{curr}^{j} + ϕ_{d i r} ϕ_{\max})

12: Evaluate the target manipulability:

m_{temp} \leftarrow Q^{j} (θ_{tar}^{j}, C_{a r m}^{j})

by Formula (21)
13: if

m_{temp} > m_{tar}

then
14:

m_{tar} \leftarrow m_{temp}, ϕ_{d i r}^{*} \leftarrow ϕ_{d i r}

15: end if
16: end for
17: Calculate the adaptive variation of the redundant angle

Δ ϕ_{curr}^{j} \leftarrow ϕ_{d i r}^{*} ϕ_{\max} (1 - m_{curr})

18: Return:

ϕ_{next}^{j} \leftarrow ϕ_{curr}^{j} + Δ ϕ_{curr}^{j}

6.2. JOSI Method

When the robot arm is moving near the singularity region, the motion controller may output higher-joint-velocity commands, which may cause a safety issue for the robot arm. To address this issue, we propose the JOSI method to inhibit the joint over-speed. The pseudocode of the proposed JOSI method is shown in Algorithm 2. The details of the JOSI method can be divided into the following three main steps.

Algorithm 2. Pseudocode of the proposed JOSI method

01: Input: Current joint angles

θ_{curr}

, target joint angles

θ_{tar}

02: Output: Joint angles at next time step

θ_{next}

03: Initialize: Set a maximum value of the joint angle variation Δθ_max,
04: Begin:

Δ θ_{tar} \leftarrow \max (|θ_{curr} - θ_{tar}|)

05: if (

Δ θ_{tar}

Δ θ_{\max}

) then
06:

θ_{next} \leftarrow θ_{curr} + (Δ θ_{\max} / Δ θ_{tar}) (θ_{tar} - θ_{curr})

07: else
08:

θ_{next} \leftarrow θ_{tar}

09: end if
10: Return:

θ_{next}

6.2.1. Detect Robot Over-Speed

The first step is to calculate the maximum variation value between the current and the target joint angles, denoted as Δθ_tar. If the value of Δθ_tar is larger than a preset maximum variation threshold Δθ_max, then the robot may suffer from joint over-speed.

6.2.2. Suppress Robot Joint Speed

According to the two maximum variation values Δθ_tar and Δθ_max, the JOSI formula (29) is proposed to calculate the joint angles at next time step, denoted by

θ_{next}

, such that the following is true:

θ_{next} \leftarrow θ_{curr} + (Δ θ_{\max} / Δ θ_{tar}) (θ_{tar} - θ_{curr})

(29)

where θ_curr is the vector of current joint angles, and θ_tar is the vector of target joint angles calculated by the inverse kinematics.

7. Experimental Results

In the experiments of this study, we used a lab-made seven-degree-of-freedom redundant dual-arm robot to evaluate the performance of the proposed M-aware grasp planning and motion control system. Figure 10 shows the hardware equipment and environment settings used in the experiments, in which a planar platform is set in front of the robot as a workspace, and a Realsense D435i RGB-D camera is mounted on the robot to detect objects placed on the platform. Each robot arm is equipped with a ROBOTIQ 2F-85 gripper for task-oriented object grasping tasks. The software environment comprised Ubuntu 18.04 with the Melodic version of Robot Operating System (ROS), along with MoveIt for trajectory planning, Gazebo for motion control simulation, and programming tools such as Python 2.7, Python-OpenCV 4.1.1.26, and TensorFlow 1.9.0.

In addition, we selected three objects as the grasping targets in the experiments. Figure 11 presents these three target objects and their default affordance masks. Each object contains two different affordance masks for the purpose of task-oriented grasping applications.

7.1. Experiments of the Proposed TOGD Method

In this section, we compare the experimental results of the proposed TOGD method with the existing Grasp R-CNN method to highlight the difference between task-oriented and task-agnostic grasp detection. Figure 12a shows an input image captured from the planar platform, which contains two target objects (a bottle and a cup) and three non-target objects (a vacuum gripper, ruler, and screwdriver). Figure 12b shows the grasp detection results produced by the Grasp R-CNN method, which is task-agnostic and thus produces multiple possible grasp rectangles on both target and non-target objects. In contrast, due to the integration of the Mask R-CNN, the proposed method can provide object detection and affordance segmentation results for target objects, as shown in Figure 12c. Therefore, by leveraging the Mask R-CNN, the proposed TOGD method only detects multiple grasp rectangles for the two target objects, as shown in Figure 12d. This property enables the proposed method to define the semantic constraints required in grasping tasks for task-oriented grasping applications.

7.2. Experiments for the Proposed M-Aware Motion Control Method

In the experiments for the proposed M-aware motion control method, we designed three different tests so that the seven-degree-of-freedom redundant robot arm encounters hardware-limited situations during motion, including joint limits, wrist singularity (Test 1 and Test 2), and joint over-speed (Test 3). In these tests, we compare the proposed M-aware motion control method with a traditional inverse kinematics method (termed InvK) and an existing Jacobian-based WLN method (termed Jacobian WLN) [23] to evaluate the control performance of the proposed method. Note that in the experiments, all compared methods were implemented and tested on a virtual seven-degree-of-freedom dual-arm robot model in the Gazebo simulator, as shown in Figure 13, to prevent these methods from damaging the actual dual-arm robot. Only the proposed method was implemented and tested on the actual dual-arm robot since it protects the robot from damage.

7.2.1. Joint Limit and Singularity Avoidance Tests

Table 2 records the parameter settings used in Test 1 and Test 2, including the initial posture and target posture of the end-effector, and the maximum variation in the redundant angle. In Test 1, the end-effector of the robot arm needs to rotate minus 90 degrees along the x-axis while moving right and backward to the target position. During this process, the robot arm will encounter its joint limits. Figure 14a shows the experimental results of Test 1, which shows that the traditional InvK control method stops working when the joint limit is reached, and the manipulability of the robot arm drops to zero. In contrast, both the existing Jacobian WLN method and the proposed method can reach the target position without encountering the joint limits, but the proposed method can maintain a high degree of manipulability during the motion control of the robot arm.

In Test 2, the robot arm moves in a straight line to the right while rotating 90 degrees along the x-axis. According to this setting, the robot arm will easily encounter its joint limits and wrist singularity during the motion control process. Figure 14b presents the experimental results of Test 2. As shown in the figure, when using the traditional InvK control method, the M-score of the robot arm drops to zero rapidly within 1 s after it starts moving. This implies that the robot arm has reached its joint limits and cannot move further. Similarly, the Jacobian WLN method also causes the robot arm to stop moving within 2 s due to the same problem. In contrast, the proposed method successfully enables the robot arm to avoid its joint limits. In addition, during the singularity avoidance process, the proposed method not only selects the best redundant angle for avoidance, but also outputs smooth angular velocity commands via the proposed JOSI method. This feature can also be observed in Test 3.

7.2.2. Joint Over-Speed Suppression Test

Table 3 records the parameter settings used in Test 3 to verify the performance of the proposed JOSI method. In Test 3, the motion of the robot arm is controlled around its wrist singularity. Therefore, during motion control, the angular velocity of at least one joint of the robot arm will be instantly accelerated to a large value. We use this phenomenon to verify the ability of the proposed JOSI method to suppress the angular velocity of over-speeding joints. Figure 15a,b show the angle evolution of all joints without and with the proposed JOSI method, respectively. It can be seen from Figure 15a that if the proposed JOSI method is not used, the angular velocity of two joints of the robot arm will greatly and instantaneously over-speed, resulting in a rapid change in the joint angle. In contrast, in the case of using the proposed JOSI method, the angular changes in each joint of the robot arm can be controlled within the preset range, so that each joint angle can change smoothly, as shown in Figure 15b.

7.3. Experiments of the Proposed System

To verify the performance of the proposed M-aware grasp planning and motion control system, we first divided the planar platform in front of the robot into five regions, as shown in yellow regions of A to E in Figure 16. This division ensures the balanced coverage of the robot’s workspace to facilitate a more systematic analysis. During the experiment, each target object to be grasped was sequentially placed in the yellow region, and five grasping tests were performed. Therefore, a total of 25 grasping tests were performed for each target object. In addition, according to the dual-arm robot used in the experiment, we divided the experiment into single-arm (the left arm of the robot) and dual-arm object grasping tests to verify the performance of the proposed M-aware grasp planning and motion control system in multi-arm robotic systems. Note that we did not calculate the accuracy for each specific region. Instead, we performed an overall accuracy assessment of the system’s performance across the entire workspace, ensuring a balanced and unbiased evaluation of the proposed method’s effectiveness.

Table 4 records the performance evaluation results of the proposed system on a total of 75 object grasping tests for all three target objects. In all grasping tests of each target object, we counted the number of successful grasps to calculate the success rate and recorded the M-score of each grasping pose of the robot arm. Observing Table 4, it can be found that in terms of success rate results, when only the left arm of the robot is used for grasping, an average success rate of 94.6% can be achieved. The main reason for the grasping failure is that when using the left arm to grasp the object placed on the right side of the platform (such as the B or E regions), the required grasping posture may be very close to its hardware limit. This leads to a reduced M-score, as observed in the grasping test for the cup target object.

On the other hand, when the robot uses the dual-arm mode to grasp the target object, it can choose an arm with a higher M-score to perform the object grasping operation due to the proposed M-aware grasp planning method. Therefore, one can observe from Table 4 that the minimum M-score value can be significantly improved when using the dual-arm mode. For example, the minimum M-score value increased by 0.73 when grasping the cup target object. Compared to the single-arm mode, the average value of the minimum M-score using the dual-arm mode is significantly improved by 0.55. Therefore, the experimental results verify that the proposed system can effectively maintain the high level of manipulability of the multi-arm robotic system in performing object-grasping tasks.

Figure 17 illustrates a demonstration of the proposed system applied to a task-oriented object grasping application. Initially, the cup and bottle objects are placed in region C of the planar platform, ready for the dual-arm robot to perform a pouring task (Figure 17a). Depending on the placement of different target objects, the proposed system effectively assists the robot in selecting the arm with the best manipulability for the grasping action. Consequently, in this application, the robot autonomously decides to grasp the body part of the bottle with its left arm (Figure 17b,c) and the body part of the cup with its right arm (Figure 17d,e). Finally, the robot uses its arms to align the cap part of the bottle over the mouth part of the cup, completing the pouring action (Figure 17f–i). A video clip of the experimental results is available online at https://youtu.be/n6brUjdHu_Q (accessed on 16 December 2024).

Note that the proposed method enables each robot arm to independently determine its best grasping object and corresponding grasping pose based on the M-score function. If one arm has multiple choices with the same M-score value, the M-score of the other arm determines the final choice. For example, in the above experiment, if the left arm’s optimal M-scores for grasping the cup and bottle objects were the same, then the right arm’s M-score for grasping these objects was decisive.

If the right arm exhibits a better M-score for grasping the cup than the bottle, the robot will choose to grasp the cup with the right arm and the bottle with the left arm, or vice versa. Therefore, the proposed method strategically selects the best grasping object and grasping pose for each arm by maximizing the cumulative sum of the M-scores for all the arms.

8. Conclusions and Future Work

This paper proposes a novel M-aware grasp planning and motion control system for multi-arm robotic systems to handle task-oriented grasp planning tasks. The design of the proposed system is based on a novel M-score function that can quantitatively evaluate the manipulability of a robot arm with respect to a given grasp pose, taking into account both the joint limits and wrist singularity of the robot arm. The proposed system consists of M-aware grasp planning and M-aware motion control subsystems. The former contains a deep-learning-based TOGD method and an M-aware grasp pose estimation method for the robot to predict task-oriented grasp candidates from the RGB-D image and select the best grasping pose among the candidates. The latter, consisting of the MIRAS and JOSI algorithms, enables the multi-arm robot to choose an appropriate arm to perform the grasping task while maintaining a high degree of manipulability and safety. In the experiment, we evaluated the proposed system using a lab-made 7-DoF redundant dual-arm robot and three target objects randomly placed on a planar platform. The experimental results demonstrate that our system can achieve task-oriented grasp detection, the affordance detection of target objects, the selection of appropriate grasping poses, and the successful grasping of the target with a high M-score of about 0.88 and an average success rate of about 98.6%.

The proposed method still has room for improvement. For example, the proposed M-aware motion control method does not address the obstacle avoidance problem during motion. Integrating obstacle avoidance algorithms into this method represents a crucial research direction. Additionally, regarding task-oriented object grasping, this study does not address the object re-grasp planning problem caused by the partial occlusion of the target object. This problem often occurs in complex work environments, making the integration of occlusion detection and object re-grasping algorithms into the proposed system another important research topic for future exploration. Furthermore, adopting directional performance indexes for singularity evaluation could improve this work, as these indexes account for the direction of motion, providing a more detailed and nuanced assessment. Integrating such indexes into the proposed method could significantly improve its effectiveness and represents an important direction for future research.

Furthermore, exploring how the Jacobian-based inverse kinematics method can be integrated with the proposed M-score function would be interesting. For example, the Jacobian WLN method uses the partial derivative of the evaluation function (1) to determine the weight matrix of the Jacobian-based controller, ensuring that joint motions remain within the physical range. Therefore, it is possible to use the proposed M-score function instead of the evaluation function (1) in the Jacobian WLN method to maintain the manipulability of the robot arm during the motion control process. This integration also represents an important future research direction.

Author Contributions

Methodology, C.-Y.T. and Y.-C.L.; Software, Y.-C.L.; Validation, Y.-C.L. and S.-W.W.; Formal analysis, C.-Y.T. and Y.-C.L.; Investigation, Y.-C.L. and S.-W.W.; Resources, C.-C.W.; Data curation, Y.-C.L. and S.-W.W.; Writing—original draft, C.-Y.T., Y.-C.L. and S.-W.W.; Writing—review & editing, C.-C.W. and C.-Y.T.; Visualization, S.-W.W.; Supervision, C.-C.W.; Project administration, C.-C.W.; Funding acquisition, C.-C.W. and C.-Y.T. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by the National Science and Technology Council of Taiwan, R.O.C., under grant NSTC 112-2221-E-032-035-MY2 and grant NSTC 112-2221-E-032-036-MY2.

Data Availability Statement

Data are contained within the article.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this paper:

6-D	Six-Dimensional
CNN	Convolutional Neural Network
DCM	Direction Cosine Matrix
DoF	Degree-of-Freedom
FCN	Fully Convolutional Network
JL-score	Joint Limits score
JOSI	Joint Over-Speed Inhibition
M-aware	Manipulability-Aware
M-score	Manipulability score
MIRAS	Manipulability Incremental Redundant Angle Selection
OOI	Object of Interest
R-CNN	Region-based Convolutional Neural Network
S-score	Singularity score
TOGD	Task-Oriented Grasp Detection

References

Billard, A.; Kragic, D. Trends and challenges in robot manipulation. Science 2019, 364, eaat8414. [Google Scholar] [CrossRef] [PubMed]
Bohg, J.; Morales, A.; Asfour, T.; Kragic, D. Data-driven grasp synthesis—A survey. IEEE Trans. Robot. 2013, 30, 289–309. [Google Scholar] [CrossRef]
Mahler, J.; Matl, M.; Liu, X.; Li, A.; Gealy, D.; Goldberg, K. Dex-net 3.0: Computing robust vacuum suction grasp targets in point clouds using a new analytic model and deep learning. In Proceedings of the IEEE International Conference on Robotics and Automation, Brisbane, Australia, 21–25 May 2018; pp. 5620–5627. [Google Scholar]
Mahler, J.; Matl, M.; Satish, V.; Danielczuk, M.; DeRose, B.; McKinley, S.; Goldberg, K. Learning ambidextrous robot grasping policies. Sci. Robot. 2019, 4, eaau4984. [Google Scholar] [CrossRef] [PubMed]
Fang, K.; Zhu, Y.; Garg, A.; Kurenkov, A.; Mehta, V.; Li, F.-F.; Savarese, S. Learning task-oriented grasping for tool manipulation from simulated self-supervision. Int. J. Robot. Res. 2020, 39, 202–216. [Google Scholar] [CrossRef]
Lüddecke, T.; Kulvicius, T.; Wörgötter, F. Context-based affordance segmentation from 2D images for robot actions. Robot. Auton. Syst. 2019, 119, 92–107. [Google Scholar] [CrossRef]
Kumra, S.; Kanan, C. Robotic grasp detection using deep convolutional neural networks. In Proceedings of the IEEE International Conference on Intelligent Robots and Systems, Vancouver, BC, Canada, 24–28 September 2017; pp. 769–776. [Google Scholar]
Guo, D.; Sun, F.; Fang, B.; Yang, C.; Xi, N. Robotic grasping using visual and tactile sensing. Inf. Sci. 2017, 417, 274–286. [Google Scholar] [CrossRef]
Zeng, A.; Song, S.; Yu, K.; Donlon, E.; Hogan, F.R.; Bauza, M.; Ma, D.; Taylor, O.; Liu, M.; Romo, E.; et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. Int. J. Robot. Res. 2022, 41, 690–705. [Google Scholar] [CrossRef]
Iizuka, M.; Hashimoto, M. Detection of semantic grasping-parameter using part-affordance recognition. In Proceedings of the 19th International Conference on Research and Education in Mechatronics, Hague, The Netherlands, 7 June 2018; pp. 136–140. [Google Scholar]
Chu, F.; Xu, R.; Vela, P.A. Learning affordance segmentation for real-world robotic manipulation via synthetic images. IEEE Robot. Autom. Lett. 2019, 4, 1140–1147. [Google Scholar] [CrossRef]
Shao, Q.; Hu, J. Combining RGB and points to predict grasping region for robotic bin-picking. arXiv 2019, arXiv:1904.07394v2. [Google Scholar]
Liang, H.; Ma, X.; Li, S.; Görner, M.; Tang, S.; Fang, B.; Sun, F.; Zhang, J. PointNetGPD: Detecting grasp configurations from point sets. In Proceedings of the International Conference on Robotics and Automation, Montreal, QC, Canada, 20–24 May 2019; pp. 3629–3635. [Google Scholar]
Liu, H.W.; Cao, C.Q. Grasp pose detection based on point cloud shape simplification. IOP Conf. Ser. Mater. Sci. Eng. 2020, 717, 012007. [Google Scholar] [CrossRef]
Yin, R.; Wu, H.; Li, M.; Cheng, Y.; Song, Y.; Handroos, H. RGB-D-based robotic grasping in fusion application environments. Appl. Sci. 2022, 12, 7573. [Google Scholar] [CrossRef]
Yin, Z.; Li, Y. Overview of robotic grasp detection from 2D to 3D. Cogn. Robot. 2022, 2, 73–82. [Google Scholar] [CrossRef]
Mousavian, A.; Eppner, C.; Fox, D. 6-DOF GraspNet: Variational grasp generation for object manipulation. In Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp. 2901–2910. [Google Scholar]
Choi, Y.; Kee, H.; Lee, K.; Choy, J.; Min, J.; Lee, S.; Oh, S. Hierarchical 6-DoF grasping with approaching direction selection. In Proceedings of the IEEE International Conference on Robotics and Automation, Paris, France, 31 May–31 August 2020; pp. 1553–1559. [Google Scholar]
Yoshikawa, T. Manipulability of robotic mechanisms. Int. J. Robot. Res. 1985, 4, 3–9. [Google Scholar] [CrossRef]
Huo, L.; Baron, L. The joint-limits and singularity avoidance in robotic welding. Ind. Robot. 2008, 35, 456–464. [Google Scholar] [CrossRef]
Rozo, L.; Jaquier, N.; Calinon, S.; Caldwell, D.G. Learning manipulability ellipsoids for task compatibility in robot manipulation. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Vancouver, BC, Canada, 24–28 September 2017. [Google Scholar]
Zghal, H.; Dubey, R.V.; Euler, J.A. Efficient gradient projection optimization for manipulators with multiple degrees of redundancy. In Proceedings of the IEEE International Conference on Robotics and Automation, Cincinnati, OH, USA, 13–18 May 1990; pp. 1006–1011. [Google Scholar]
Chan, T.F.; Dubey, R.V. A weighted least-norm solution based scheme for avoiding joint limits for redundant joint manipulators. IEEE Trans. Robot. Autom. 1995, 11, 286–292. [Google Scholar] [CrossRef]
Hayes, M.; Husty, M.; Zsombor-Murray, P. Singular configurations of wrist-partitioned 6r serial robots: A geometric perspective for users. Trans. Can. Soc. Mech. Eng. 2002, 26, 41–55. [Google Scholar] [CrossRef]
Park, T.-W.; Yang, H.-S. A study on singularity avoidance and robust control of redundant robot. In Proceedings of the 4th World Congress on Intelligent Control and Automation, Shanghai, China, 10–14 June 2002; pp. 1687–1691. [Google Scholar]
Zaplana, I.; Hadfield, H.; Lasenby, J. Singularities of serial robots: Identification and distance computation using geometric algebra. Mathematics 2022, 10, 2068. [Google Scholar] [CrossRef]
Vahrenkamp, N.; Asfour, T.; Metta, G.; Sandini, G.; Dillmann, R. Manipulability analysis. In Proceedings of the 12th IEEE-RAS International Conference on Humanoid Robots, Osaka, Japan, 29 November–1 December 2012; pp. p 568–573. [Google Scholar]
Shi, X.; Guo, Y.; Chen, X.; Chen, Z.; Yang, Z. Kinematics and singularity analysis of a 7-DOF redundant manipulator. Sensors 2021, 21, 7257. [Google Scholar] [CrossRef] [PubMed]
He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2961–2969. [Google Scholar]
Chu, F.-J.; Xu, R.; Vela, P.A. Real-world multiobject, multigrasp detection. IEEE Robot. Autom. Lett. 2018, 3, 3355–3362. [Google Scholar] [CrossRef]
Xiang, T.; Jiang, X.; Qiao, G.; Gao, C.; Zuo, H. Kinematics parameter calibration of serial industrial robots based on partial pose measurement. Mathematics 2023, 11, 4802. [Google Scholar] [CrossRef]
Singh, G.K.; Claassens, J. An analytical solution for the inverse kinematics of a redundant 7DoF manipulator with link offsets. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Taipei, Taiwan, 18–22 October 2010; pp. 2976–2982. [Google Scholar]

Figure 1. Architecture of the proposed M-aware grasp planning and motion control system.

Figure 2. The JL-score plot of the joint limits evaluation function proposed in (a) previous studies [22,23] and (b) this work. The evaluation function in (a) features an unbounded value range, making it challenging to quantitatively assess the degree of joint limits. In contrast, the function in (b) has a bounded value range between 0 and 1, simplifying the quantitative assessment of the degree of joint limits.

Figure 3. System configuration of the lab-made 7-DoF redundant dual-arm robot used in this study. Each arm has an additional prismatic joint to adjust its z-axis position to extend the workspace along the z₀-axis.

Figure 4. The S-score plot of the proposed singularity evaluation functions (15) and (16). These plots illustrate that the singularity evaluation of the robot arm is treated as a joint limit evaluation of its wrist singularity factors. Consequently, the S-score of the evaluation function remains within the bounded range of 0 to 1.

Figure 5. System architecture of the proposed M-aware grasp planning method, which aims to determine maximum M-score grasp poses for the robot arm to perform object grasping tasks.

Figure 6. The result of each step in the TOGD method: (a) the input RGB image, (b) the result of Mask R-CNN, (c) the affordances of all segmented objects (the darker regions), and (d) the feasible grasp rectangles.

Figure 7. The state definition of the grasp rectangle used in this work.

Figure 8. Definition of position and orientation errors used in grasp matching: (a) position error and (b) orientation error.

Figure 9. Illustration of (a) the definition of the target pose vector used in this study and (b) the proposed grasp pose transformation to transform a target pose vector from the camera frame to the robot base frame.

Figure 10. The hardware equipment and environment settings used in the experiments of this study.

Figure 11. Three target objects used in the experiment and their default affordance masks: (a) a PET bottle with cap and body masks, (b) a cup with mouth and body masks, and (c) a scraper with handler and head masks.

Figure 12. Comparison between the proposed TOGD method and the existing Grasp R-CNN method: (a) Input image showing two target objects (bottle and cup) and three non-target objects (vacuum gripper, ruler, and screwdriver), (b) task-agnostic grasp detection results of the Grasp R-CNN method, presenting multiple possible grasp rectangles for target and non-target objects, (c) object detection results obtained by the Mask R-CNN method, and (d) task-oriented grasp detection results of the proposed TOGD method, showing multiple grasp rectangles only for the two target objects.

Figure 13. A virtual 7-DoF dual-arm robot model was used to test the performance of all compared methods in the Gazebo simulator.

Figure 14. Comparison results of the manipulability evolution of each control method in (a) Test 1 and (b) Test 2.

Figure 15. Comparison results of the angle evolution of all joints in Test 3 (a) without and (b) with the proposed JOSI method.

Figure 16. Experimental setup for verifying the performance of the proposed system, in which the planar platform in front of the robot is divided into five regions (regions A to E) for object grasping tests to verify the performance of the proposed M-aware grasp planning and motion control system.

Figure 17. Demonstration of the proposed system controlling the dual-arm robot to perform a task-oriented pouring action: (a) system initialization; (b,c) the robot uses its left arm to grasp the bottle object; (d,e) the robot uses its right arm to grasp the cup object; (f) the robot prepares to perform the pouring action; (g–i) the robot performs the pouring action.

Table 1. D-H parameters of the left arm of the dual-arm robot.

Left DH-Link	a_i (m)	α_i (Degree)	d_i (m)	q_i (Degree)
Joint 0 (Base)	0	−90	d₀	0
Joint 1	0	−90	d₁	θ₁ + 90
Joint 2	0	−90	0	θ₂ − 90
Joint 3	a₃	90	d₃	θ₃
Joint 4	a₄	−90	0	θ₄
Joint 5	0	90	d₅	θ₅
Joint 6	0	−90	0	θ₆
Joint 7	0	0	d₇	θ₇ + 90

Table 2. Parameter settings for the joint limit and singularity avoidance tests.

Parameter	Symbol	Value
Parameter	Symbol	Test 1	Test 2
Initial posture of the end-effector	P_curr	[0.5, 0.3, −0.5]	[0.3, 0.2, −0.35]
	O_curr	[180°, 0°, 0°]	[90°, 0°, 0°]
	ϕ_curr	0°	0°
Target posture of the end-effector	P_tar	[0.1, −0.1, −0.5]	[0.3, 0.05, −0.35]
Target posture of the end-effector	O_tar	[90°, 0°, 0°]	[180°, 0°, 0°]
Maximum variation in the redundant angle	ϕ_max	2°	2°

Table 3. Parameter settings for the joint over-speed suppression test.

Parameter	Symbol	Value
Initial posture of the end-effector	P_curr	[0.2, 0.55, −0.6]
	O_curr	[180°, 0°, 0°]
	ϕ_curr	0°
Target posture of the end-effector	P_tar	[−0.2, 0.55, −0.6]
Target posture of the end-effector	O_tar	[180°, 0°, 0°]
Maximum variation in joint angle	Δθ_max	1.7°

Table 4. Performance evaluation results of the proposed system for M-aware object grasping tasks on the dual-arm robot.

Target Object		PET Bottle	Cup	Scraper	Average
Total number of grasping tests		25	25	25	25
Success rate	Single-arm	96%	92%	96%	94.6%
Success rate	Dual-arm	100%	100%	96%	98.6%
Minimum M-score	Single-arm	0.38	0.16	0.47	0.33
Minimum M-score	Dual-arm	0.91	0.89	0.84	0.88
Improvement on minimum M-score		+0.53	+0.73	+0.37	+0.55

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Wong, C.-C.; Tsai, C.-Y.; Lai, Y.-C.; Wong, S.-W. Manipulability-Aware Task-Oriented Grasp Planning and Motion Control with Application in a Seven-DoF Redundant Dual-Arm Robot. Electronics 2024, 13, 5025. https://doi.org/10.3390/electronics13245025

AMA Style

Wong C-C, Tsai C-Y, Lai Y-C, Wong S-W. Manipulability-Aware Task-Oriented Grasp Planning and Motion Control with Application in a Seven-DoF Redundant Dual-Arm Robot. Electronics. 2024; 13(24):5025. https://doi.org/10.3390/electronics13245025

Chicago/Turabian Style

Wong, Ching-Chang, Chi-Yi Tsai, Yu-Cheng Lai, and Shang-Wen Wong. 2024. "Manipulability-Aware Task-Oriented Grasp Planning and Motion Control with Application in a Seven-DoF Redundant Dual-Arm Robot" Electronics 13, no. 24: 5025. https://doi.org/10.3390/electronics13245025

APA Style

Wong, C. -C., Tsai, C. -Y., Lai, Y. -C., & Wong, S. -W. (2024). Manipulability-Aware Task-Oriented Grasp Planning and Motion Control with Application in a Seven-DoF Redundant Dual-Arm Robot. Electronics, 13(24), 5025. https://doi.org/10.3390/electronics13245025

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Menu

Manipulability-Aware Task-Oriented Grasp Planning and Motion Control with Application in a Seven-DoF Redundant Dual-Arm Robot

Abstract

1. Introduction

2. Related Work

3. Overview of the Proposed System

4. The Proposed Manipulability Evaluation Function

4.1. Joint Limits Evaluation Function

4.2. Singularity Evaluation Function

4.3. Manipulability Evaluation Function

5. The Proposed M-Aware Grasp Planning Method

5.1. TOGD Method

5.1.1. Grasp Rectangle Detection

5.1.2. Grasp Matching

5.2. M-Aware Grasp Pose Estimation Method

5.2.1. Grasp Pose Generation

5.2.2. Grasp Pose Transformation

5.2.3. Grasp Quality Evaluation

6. The Proposed M-Aware Motion Control Method

6.1. MIRAS Method

6.2. JOSI Method

6.2.1. Detect Robot Over-Speed

6.2.2. Suppress Robot Joint Speed

7. Experimental Results

7.1. Experiments of the Proposed TOGD Method

7.2. Experiments for the Proposed M-Aware Motion Control Method

7.2.1. Joint Limit and Singularity Avoidance Tests

7.2.2. Joint Over-Speed Suppression Test

7.3. Experiments of the Proposed System

8. Conclusions and Future Work

Author Contributions

Funding

Data Availability Statement

Conflicts of Interest

Abbreviations

References

Share and Cite

Article Metrics

Article Access Statistics

Further Information

Guidelines

MDPI Initiatives

Follow MDPI