Grasping and retrieving unknown hazardous objects with a mobile manipulator

1.1 Motivation

Manipulators in industrial production as well as mobile transport platforms in logistics are robust and mature. Hence, robot arms and mobile robots account for an important and big part of the value creation in many countries by moving, transporting and manipulation goods. The ability to move and handle objects reliably has been the most important ability of robots for many years. Recently, a change in value creation is taking place, away from mechatronics and towards more intelligence and autonomy based on advanced algorithms. The robot skills are improving and new applications are made possible outside industry and logistics.

In these new application fields like health care, public and private assistance and inspection the ability to quickly adapt to dynamic, changing environments and new tasks is essential. Therefore, reusable, modular and adaptive architectures are needed to comply with these requirements. Within this work, the focus is to help and support humans in hazardous environment or with hazardous objects and reduce environmental and health risks by applying intelligent mobile manipulators. Most other works are dedicated to a subset of the challenges that have to be addressed. Many works focus on grasping and handling of known or unknown objects. Others are more dedicated to advanced image or sensor data processing and interpretation. And there are works enabling robots to map unknown environments and enable them to safely navigate. Within this work it is necessary to address all challenges. The resulting issues require a shift in thinking – shared perception and control to reach the desired robustness und reliability.

This work presents a modular, reusable hardware and software approach that is based on standard off-the-shelf components together with Open Source libraries, machine learning models and frameworks like Google Cartographer, PyTorch and ROS. The use of powerful established implementations in combination with new software components is the basis of this work. These new components like the 3D mapping framework OpenVBD_Mapping are made available to the Community as Open Source repositories. The combination of easy to train instance segmentation networks with powerful 3D mapping and path planning functions makes this system highly flexible. On the other hand, the consequent integration of human expertise throughout the whole pipeline put a strong focus on the development to intuitive visualizations, operation and control concepts makes it easy to interact with the robot on all levels. Consequently, this contributes to a reusable and transferable shared grasping and retrieving approach for unknown, hazardous objects with a mobile robot platform.

1.2 Related work

The variety of autonomous mobile safety robots is large. Some of the robots enhance human safety by searching for landmines [1] or performing bridge inspections to detect cracks [2]. In the field of hazardous or nuclear objects the robot systems are almost exclusively tele-operated [3, 4]. Mapping the environment is one of the fundamental challenges in the use of autonomous mobile robots. Sensor-based sensing for mobile robots is typically performed using laser scanners, RGB (Red Green Blue) and RGB-D (Red Green Blue -Depth) sensors [5–8]. While simultaneous localization and mapping (SLAM) methods in two dimensional structured environments, such as office buildings, have been established for some time [9], three dimensional unstructured environments, e.g., outdoors, pose a particular challenge and are the subject of current research [10–12]. For the navigation of the robot, a distinction is made between global navigation with information about the environment, which can e.g. consist of a map, and local navigation without information about the environment [5]. One possibility for planning paths in unknown environments are so-called bug algorithms [13]. These move the robot in the direction of the target and react to too close objects by short path changes. Other work deals with the identification of dangerous spots based on images, e.g. to avoid rocks or soft terrain on a Mars expedition [14]. To support faster path finding, multiple robots can be used to share the acquired information with each other [15]. Grasping and handling objects in unknown environments based on visual and partly also haptic perception is an essential part of different domains, especially in the handling of hazardous materials. This is also an important topic in current research [16–20]. For the detection of obstacles and manipulable objects, segmentation and identification of related objects [21] plays an important role. Recent work uses e.g. well-known object models [22] or Deep Learning to narrow down the grasp selection [23].

1.3 Challenges within the context of ROBDEKON

Within the National Competence Center on Robotic Systems for Decontamination in Hazardous Environments – ROBDEKON – the research is evaluated in different application-oriented scenarios. Each of the scenarios was intensively discussed with industrial stakeholders from industry dealing with nuclear decommissioning and emergency on-call service experts for nuclear facilities. The handling and retrieval of unknown hazardous objects is a key use case in one of the scenarios. It comes with a number of challenges that have to be addressed:

1. Develop a mobile robot with robot arm and required sensor setup.

2. Create an online 3D map of the environment with no GPS localisation information.

3. Localise the mobile robot within this 3d Map with no GPS localisation information.

4. Identify and localise hazardous objects with only little information about the objects.

5. Navigate autonomously in the 3D Environment to desired goals.

6. Grasp and retrieve the objects in a shared operation and control approach.

2.1 Hardware- and software-architecture

The work on grasping and recovering hazardous goods was executed and evaluated on the mobile robot Husky, a UGV (Unmanned Ground Vehicle) from Clearpath Robotics (see Figure 1). The robot was initially equipped with a 6 DOF (Degrees Of Freedom) robot arm (Universal Robots UR5) with a payload capacity of 5 kg as well as a Velodyne HDL-32E LIDAR (LIght Detection And Ranging) and a DGPS (Differential Global Positioning System) from NovAtel (SMART6-L GPS). The LIDAR is used together with an additional MicroStrain 3DM-GX3-25 IMU (Inertial Measurement Unit) for the 3D live mapping of the environment and at the same time localization within this map (SLAM).

Figure 1:

Mobile robot husky with 6 DOF arm (left) and mobile ground-station with six monitors (right).

Up to now, the DGPS system has not been used, as the availability of GPS signals cannot be guaranteed in the foreseen scenarios. The official supported ROS interface from Clearpath Robotics to all components makes it possible to easily access and add new components.

First, a force-torque sensor (Schunk FTN-Delta Sl-660-60) and a highly flexible three finger-gripper (Robotiq 3-Finger Adaptive Robot Gripper) were mounted on the robot arm. Moreover, a 3D camera (Intel RealSense d435i) was connected to the end of the arm giving an optimal view to potential objects close to the gripper. A second camera was mounted on the top of the platform. This 360° camera (Ricoh Theta V) provides a high-resolution panorama of the surroundings of the robot. It helps the operator to get a better overview of the current environment but is also used by the instance segmentation network to automatically detect potentially hazardous goods. Some hardware modifications improve the overall performance of the platform. The new LiPo (Lithium-Polymer) batteries improved the runtime of the robot, now allowing autonomous operation of up to 6 h. The off-road tires were changed to tires with less profile that create a smoother motion on hard surfaced like concrete or asphalt. If needed the tires can be switched within a few minutes, making it possible to adapt to specific mission requirements. Storage bins have been added to the robot platform to safely retrieve items during a mission. For this purpose, a general storage container in the form of an open-top box and for special (simulated) hazardous goods and an extra metal container have been mounted on top and front of the robot. In order to detect objects under insufficient external illumination, two high power LEDs were mounted on the front of the robot.

In addition, a mobile ground station was built to facilitate field tests. A stable, transportable box was constructed and equipped with two powerful PCs (one with GPU), six full-HD screens and a LiPo battery charger. Due to the large number of screens, different views, sensor data, 3D maps and status reports can be displayed in parallel. In the future, it is planned to expand the system to include internal batteries that can be used to operate the control centre without an external power supply, at least for a short time. The wireless network setup was improved regarding bandwidth by adding a high-performance IEEE 802.11ac access point to the ground station and antenna to the robot.

The built hardware does not claim to be used in a real scenario with radiating sources or a nuclear environment in general. The hardware was built to develop and test various software components and assistance functions. For the use in e.g. nuclear environments there are special robot systems e.g. from Telerob GmbH, which are not really useful for scientific development and evaluation.

The software architecture regarding the interaction with the human operator, the interaction between the components as well as the shared control and operation modes is visualized in Figure 2.

Figure 2:

Software architecture with shared operation and control approach.

All skills described later in this work, the shared manipulation strategy, instance-based object detected, 3D mapping and path planning, were fully integrated and evaluated on this mobile husky platform.

2.2 Shared force-based grasping of unknown objects

The shared autonomy approach has the goal to increase the speed and success rate of complex robotic manipulation tasks with hazardous objects or in hazardous environments. Even in dark environments, humans have no problems grasping objects for which the positions are only roughly known. By touching the objects, these inaccurate positions are instantaneously corrected and it is possible to find a stable grasp. The grasping strategy follows this concept by combining information from a human operator and a force-torque sensor on the robot. The operator takes over the object detection and the selection of a suitable grasping strategy. This strategy consists of a gripper mode, a grasp mode and the position as well as the orientation of the gripper (gripper pose) with respect to the object. A good gripper pose for the object is essential. In literature, there are many solutions to find suitable grasps for all kinds of objects. In the industrial environment, this challenge is typically called bin picking and has been solved in many applications. Nevertheless, the calculation and selection of suitable positions and orientations of the gripper for a successful grasp of objects is still a challenge if these objects are unknown, deformable or with sparse sensor data. Therefore, this task is highly dependent on visibility and lighting conditions. A deterioration of the sensor data quality directly leads to a more difficult object recognition and thus to worse grasping results. On the other hand, this task is trivial for a human. Even little information about the object is sufficient for a human to select and perform a successful grasp.

Two different grasp modes were implemented for grasping objects. The first grasp mode was developed for unknown objects of all kinds. Within this grasp mode, the gripper is first moved close to the target pose. Then the gripper is moved to the desired target pose using a force-sensitive compliance controller. If a physical contact is detected, a set of different reactions is automatically triggered. Depending on the triggered situation, the gripper pose is automatically corrected or the gripper is closed. The position correction compensates for inaccurate operator inputs and ensures that the gripper is positioned in a way that the object can be grasped and picked up successfully. If the fingers are touched by the object, the gripper target pose is shifted in a positive or negative direction along the X-axis of the gripper. This process is repeated multiple times, if further contacts are detected. The first grasp mode is visualized in a flow chart (see Figure 3).

Figure 3:

Flow chart for the grasp mode for unknown objects. The user inputs are shown in the top blue box. The user initially selects the object and grasping strategy. The robot in the bottom orange box then automatically executes the force-adaptive grasping process.

A second grasp mode was developed for grasping objects with handles. For this purpose, additional steps were introduced into the algorithm and existing ones were modified to enable the touching of handles. More grasp modes were introduced for the easy handling of valves and picking up objects, but not further discussed in this work.

A gravity compensation of the gripper was implemented and subtracted from the measured values. The resulting forces and torques are only the contact forces acting on the wrist. The gripping force of the fingers cannot be adapted with the gripper used in this setup. The applied adaptive compliance control is based on previous work from Scherzinger et al. [24].

The shared manipulation tasks are operated via a graphical interface. The graphical interface displays the current environment close to the robot by means of a 3D point cloud from a RGB-D camera. The current position and kinematic configuration are also displayed using a 3D model of the robot. A virtual representation of the gripper is used to simply define the target gripper pose. The grasp mode is selected via a context menu. The grasp is then executed autonomously by the robot. Besides this shared control approach, the operator has a variety of assistance functions, in order to perform a wide range of tasks (see Figure 4). For the search-and-recovery tasks, for example, the following assistance functions are made available to the operator: 3D mapping of the environment, visual recognition of predefined objects, collision-free path planning on the 3D map, detailed images in colour and precise 3D point clouds, shared grasping of objects and supported picking and placing of objects.

Figure 4:

Intuitive and fast parametrization directly in rviz where also the RGBD sensor data are visualized.

2.3 Instance segmentation for object identification and localisation

The challenge regarding the recovery of unknown, hazardous objects is about recognizing hazardous objects in an unknown, cluttered environment and localize these objects with a mobile robot. This work is based on an initial visual detection using the camera data with deep neural networks. The robot is equipped with different cameras: one RGB-D camera on the robot arm and one 360° camera on the sensor frame on the back of the modified Husky robot. The camera on the robot arm provides colored depth information and has great flexibility in terms of its orientation. It is mainly intended for the perception of objects close to the robot, which can be inspected from different perspectives and to parametrize the force-based grasping strategy. The camera on the sensor frame, is a statically mounted 360° camera to provide a fast overview during the exploration phase. It can provide initial indications of where potentially hazardous objects are located (see Figure 5).

Figure 5:

Masked 360° image with the hazardous goods container detected by the mask R-CNN. Without the masking of the robot, false detections could occur on the robot itself.

Many state-of-the-art works in object detection rely on artificial deep neural networks. But usually large data sets are required to detect arbitrary objects using such approaches. However, the requirements for the scenario of retrieval of hazardous materials are special. In most cases, no large data sets are available for the target objects to be detected, e.g. specific barrel types or other rare hazardous material containers. In addition, many false detections can occur if the corresponding operational environment is not known to the model.

A Mask R-CNN architecture is used for this purpose. Pre-trained on the comprehensive MS-COCO dataset, a fine-tuning of the detection components in the higher-level layers with a few annotated images of the specific categories is sufficient to enable an accurate detection or segmentation of the objects. The Mask R-CNN architecture for neural networks, first presented in Mask R-CNN [25], is an extension of Faster R-CNN [26]. Parallel to the bounding box prediction of the objects, an exact mask representing all pixels that belong to the respective object is predicted. This mask provides more information about the pose of the object, but in return the computational effort for processing the images increases.

Once a small data set has been created for the hazardous material to be detected, it can be augmented as desired and is then suitable for training the designed neural detection architectures (see Figure 6). In addition, the background of the training images can be replaced by background images that are expected in the mission scenario. However, augmentation strategies also have their limitations when it comes to augmenting smaller data sets. There is a danger of overfitting. This refers to the process when the model parameters are optimized too much for the existing training data, so that the system’s detection capabilities degrade. In order to prevent this, a transfer learning method is applied on the model parameters of the network architecture, which was pre-trained on large open available data sets (e.g. MS-COCO). These large data sets enable basic capabilities in the model with respect to image recognition and visual information processing. Subsequently, the parameters are adapted to the concrete, very specific hazardous goods in the second training phase, the so-called fine-tuning. The task-oriented adaptation takes place in the higher, more abstract layers of the network architecture and requires significantly less data.

Figure 6:

Object identification and localization pipeline based on instance segmentation architectures and 3D maps.

Two neural network architectures are currently applicable in this work: Mask-R-CNN and CenterMask2-Lite [27]. Both perform very well in the COCO benchmark and show robust capabilities, also compared to many approaches in the literature. For example, the CenterMask2-Lite, a one-stage model, has been designed to be as lightweight as possible to be suitable for mobile computing devices. The Mask-R-CNN, a two-stage model, is designed for detection quality and requires more computational effort. More common architectures usually only provide bounding boxes for each detection. However, masking provides additional information about the orientation and the exact position, which is of great importance in robotic manipulation tasks. The two currently used architectures (Mask-R-CNN and CenterMask2-Lite) differ particularly in their complexity and the computational requirements on the hardware. As default, the images are processed on the ground station, where sufficiently powerful hardware is available. For missions, where no continuous communication with the ground station can be guaranteed, the lightweight CenterMask2-Lite architecture could serve as an onboard fallback. Then the image processing can take place locally on the mobile robot Husky. But in general, the Mask R-CNN has proven to be significantly more robust regarding the generalization capabilities with respect to unknown lighting conditions and new unseen instances of known categories.

A further challenge is to extract the 3D position of the target objects. Other classical vision or image processing approaches are able to recognize the 6D pose of an object in addition to the object category [28–30]. The Mask R-CNN approach as well as the CenterMask2-Lite do not provide any spatial information of the detected objects except the pixel coordinates. There are new NN architectures such as Mesh R-CNN [31] that address this lack. This approach is not used, because it requires a very large amount of training data and still does not provide the accuracy required for mobile manipulation tasks.

In this work, the robot platform autonomously navigates to the area where objects are initially detected in the 360° camera image. The target coordinates are determined by raycasting on the 3D map captured by the 3D LiDAR. This requires a transformation of the pixel coordinates to a ray in the global reference coordinates. The intersection point with the voxels in the 3D map are found by raycasting in the temporary position of the object, which is sufficiently accurate for rough navigation. The RGB-D camera on the robot arm is then used in the target area to re-localize the hazardous object. With this camera, the object can be localized with a higher precision and provide the desired depth information. A DBSCAN clustering algorithm (Density-Based Spatial Clustering of Applications with Noise) is used to preprocess the detections, which filters inaccuracies and prevents incorrect information from being entered into the map. The filter approach checks if the detections are occasional false detections or continuous detections with high confidence at the same or a nearby position. In this case, the detected and localized objects are additionally added to the 3D map as interactive markers to enable the operator to easily manage the found potentially hazardous objects.

2.4 Online 3D mapping for mobile robots

The 3D mapping approach is based on Google’s Cartographer, which was already successfully used in various works [32]. This is an open source software that is able to perform a real-time SLAM on the basis of laser scans or point clouds. In previous works the authors have only used Cartographer for 2D indoor navigation. In this work, a 3D mapping approach was developed enhancing the previous functionality. For this purpose, different sensor setups were tested and evaluated. In contrast to the 2D case, the 3D SLAM requires the values of an inertial measurement unit (IMU) in addition to the 3D point cloud data. The IMU provides information about the orientation, as well as the linear and angular accelerations. These are used by Cartographer to transform the input point cloud into the ground plane in order to reduce the problem of scan matching by 2 dimensions (roll and pitch angles), resulting in a higher processing speed of the individual point clouds. In addition to pure scan matching, Cartographer offers the possibility of feeding various other localization sources into the algorithm, such as pre-calculated odometry or the coordinates of a global navigation satellite system (e.g. GPS). In this work, the wheel-based odometry of the Husky was used as additional input. Although the odometry has a high slip, it nevertheless brings an enormous advantage in certain scenarios compared to a setup without this information.

After establishing a stable SLAM system, the goal was to create a 3D map from all the input data. In general, the Velodyne LiDaR point clouds are rather sparse, at least vertically. Despite this sparse initial scan data, the resulting total point clouds are very dense due to the continuous movement of the robot. On average, the Velodyne Puck accumulates about 14 million points per minute. A naive approach raises several problems. On the one hand, the map occupies a huge part of the system’s memory, which must also be kept available during the entire navigation. On the other hand, it is very time-consuming and inefficient to plan on the entire point cloud. Therefore, it is better to introduce an abstraction layer of the map representation instead of using the full point clouds.

For this purpose, the environment is first divided into a uniform 3D grid in which each cell covers a defined area depending on the spatial resolution and stores whether the corresponding cell is occupied or free. However, this level of abstraction is not yet sufficient because the grid grows too quickly due to its three dimensions. The main reason for this is that memory consumption does not scale directly with the number of relevant grid cells. Instead, the data structure grows proportionally to the space that embeds the data. A prominent example of this is the mapping of free areas on the map. These take up a large part of all cells, but provide relatively little information about the surrounding area.

To counteract this problem, instead of a regular complete 3D grid, more efficient tree structures are increasingly used as data management in such cases. A frequently used implementation for this is the ROS implementation OctoMap [33], which is based on octrees. This is a 3D data representation that discretises the space similar to occupancy grids and also offers efficient memory management. To insert new data into the octree, a ray casting beam is projected from the sensor to the data point through the entire data structure for each new point in a sensor scan. All traversed grid cells are classified as free or occupied with the help of a probability distribution. The result is a probabilistic occupancy map, similar to the occupancy grid maps used in the 2D case. However, one problem with the OctoMap is that the insertion of new data points and ray casting is too slow due to the data structure in relation to the sensor data rate. Since the mapping cannot insert new data fast enough, new measurements from the sensor are often ignored, resulting in a sometimes inaccurate and incomplete map.

Therefore, a new 3D mapping framework was developed that focuses on the fast and efficient integration of new data points. In this approach OpenVDB is used as an efficient data storage layer for the volumetric data [34]. OpenVDB has its origins in computer animation and is mainly used for volumetric modelling in animated movies. In this field of application, it is essential to render data structures efficiently together with optimised ray casting algorithms. However, this property can also be applied to the ray casting operations during the mapping process, which makes the OpenVDB much faster than OctoMap, see Figure 7. OpenVDB has also been optimised in terms of memory consumption. It also encodes the free space over low-resolution voxels in a memory-efficient way, similar to the described octree approach. In addition, it provides built-in compression algorithms to keep the data structure as small as possible.

Figure 7:

Comparison of the calculation time for datapoint insertion between OctoMap and our OpenVDB implementation with respect to different resolutions and raycasting distances [10].

In addition to replacing the entire mapping data storage layer, the way in which new sensor data is integrated has been fundamentally revised. The ray casting of all points was divided into two steps. First, all rays are projected through the data structure as usual. However, instead of directly updating the probabilities, a temporary grid is first created in which it is noted whether a cell needs to be updated or not. As soon as this has been done for all points, the second step begins. The temporary grid is compared with the actual map in order to update all necessary grid occupancy probabilities. This new 3D mapping approach showed an up to ten times faster mapping speed, depending on the map resolution and maximum sensor range. Figure 7 shows in detail how the two data structures behave with respect to different resolutions and ray casting distances.

Due to these good results, this new mapping approach was made available to a larger community by publishing the open source software package vdb_mapping [35]. In addition, all algorithms like the path planning for mobile robots, which were previously based on OctoMap, were ported to the new map format. An example of a dense VDB map of the FZI lab building that was created with a flying drone equipped with a LIDAR is shown in Figure 8.

Figure 8:

OpenVDB map of the backyard between the FZI lab buildings [10].

In many situations multiple robots are applied for a difficult task. Moreover, that map should also be available at the base station. Therefore, the maps have to be synchronised between the individual robot systems as well as the base station over the network for an effective use. This generates large, impracticable amounts of data. To counteract this effect, we have extended the approach with a remote mapping function. The idea here is to transmit only new parts of the environment that are currently detected by the lidar sensors over the wireless network instead of the whole map. For this purpose, we exploit the already existing data integration of VDB-Mapping. Once a new point cloud is acquired, ray casting through the map structure is performed for each data point to determine which cells are occupied and which are free. This information is stored in an efficient bit structure within the active mask of a VDB tree. These so-called update grids only cover the area of the map currently perceived by the LIDAR. In the approach the update grids are used to determine which occupancy probabilities in the map need to be updated to represent the current LIDAR data. As a result, it is only necessary to check within the update grid whether an update must be carried out on the corresponding cells of the map. Since the update grids do not contain any data except for the bit flag, they are relatively small and can therefore be sent over the network with a high update rate. To further reduce the size, the sparsely filled VDB tree is compressed, serialised and made available to other remote mapping nodes as a ROS topic. These nodes then unpack the data and perform the same update step as the robot in order to obtain the same map of the environment without having to send it over the network.