AttentionPose: Attention-driven end-to-end model for precise 6D pose estimation

1 Introduction

6D pose estimation (six degrees of freedom, three translational and three rotational) determines an object’s 3D position and orientation in a scene. It involves estimating the object’s translation (i.e., its position in space) and rotation (i.e., its orientation) in six degrees of freedom, which are three for position (x, y, z) and three for orientation (roll, pitch, yaw) [1].

The importance of 6D pose estimation lies in its ability to enable robots or autonomous systems to interact with objects in the real world [2]. By accurately determining the position and orientation of an object, a robot can perform various tasks, such as picking and placing objects, assembly, inspection, and manipulation. This technology has numerous applications in manufacturing, logistics, healthcare, and other fields where robots automate processes and perform repetitive tasks. Precise 6D pose estimation is crucial for enhancing object recognition in cluttered environments and for robotics and augmented reality (AR) applications. In the context of autonomous driving, it contributes to improved safety and decision-making. The integration of attention mechanisms specifically addresses the challenges posed by occlusions and clutter, thereby elevating the model’s real-world utility [3]. For successful robotic grasping and manipulation, accurate 6D pose estimation is required. It improves the efficiency and efficacy of robotic systems, reduces the need for human involvement, and eventually results in cost savings and enhanced output [4].

Moreover, AR [5] and autonomous driving [6] rely heavily on 6D position estimation. AR uses 6D pose estimation to precisely put virtual objects in the real environment. By detecting the location and orientation of the camera and the real-world objects, the AR [7] system may display virtual items that appear as part of the image, increasing the user experience and allowing new gaming, education, and design applications.

Autonomous driving uses 6D position estimation to recognize and track environmental objects, such as other cars, pedestrians, and barriers. By calculating the 6D position of these objects concerning the car, the autonomous driving system can forecast their future movements and prevent crashes or driving around them. In addition to being required for perceptual tasks such as lane identification, road sign recognition [8], and mapping, accurate 6D posture estimate is also essential for lane detection, sign recognition, and mapping. In AR and autonomous driving, the accuracy and efficiency of 6D posture estimation can substantially affect system performance and safety. Therefore, continued research and development in this field are essential for advancing these technologies and creating new applications [1,6,9].

Traditional methods for 6D pose estimation often rely on geometric or feature-based approaches. These methods use various algorithms to extract features from the image or point cloud data and then match them with a model of the object to determine its pose. To estimate an object’s pose, geometric techniques often employ geometric restrictions, such as correspondences between 3D points or lines. Perspective-n-Point (PnP) [10] and Random Sample Consensus (RANSAC) [11] are geometric approach examples. On the other hand, feature-based approaches estimate an object’s posture using an image or point cloud properties such as corners or edges. Scale-Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF) are feature-based algorithm examples [12]. However, these conventional approaches have several drawbacks, including susceptibility to noise, occlusion, and changes in lighting conditions. In addition, they may struggle to manipulate items that are complicated, non-rigid, or devoid of distinguishing characteristics [13].

Recent advances in deep learning (DL) techniques, such as Convolutional Neural Networks (CNNs) and Point Cloud Networks (PCNs) [9,14], have shown promising results in 6D posture estimation, especially for complicated or obstructed objects. These techniques usually involve training a deep neural network with a huge set of images or point clouds to learn the relationship between the input data and the object’s posture. A 2019 study [13] introduces a DL solution for 6D pose estimation using DL. A Pixel-wise Voting Network (PVNet) is designed to handle items with distinct textures or appearances, which can deal with partially or fully obscured objects. The challenges in 6D pose estimation primarily stem from occlusions, clutter, and variability in object appearance, especially in complex scenes. These intricacies can confound traditional algorithms, making pose estimation an intricate task. Computational efficiency is also a key concern for real-time applications. Our incorporation of attention mechanisms aims to mitigate these challenges by focusing computational resources on salient regions, thus enhancing accuracy and reliability in cluttered and occluded conditions.

The PVNet system comprises two key components: a pixel-by-pixel segmentation network and a posture estimation network. The segmentation network is initially trained to anticipate a binary mask for the object of interest, indicating which pixels belong to the object and which belong to the background. The network is then trained to predict the 6D posture characteristics of the item from the segmented image. Segmenting the object from the image using a segmentation network is the first step in inference [15,16]. The pose estimate network then uses the segmented image to predict the posture attributes of the item. Finally, the posture estimation network uses a pixel-wise voting mechanism to aggregate the projected postures from several image regions, minimizing the impact of occlusion and other noise sources. As a result, PVNet does not need 2D keypoint correspondences, which might be challenging to gather exactly for some objects. Instead, it uses the object’s unique texture or appearance for pixel-by-pixel segmentation and posture estimate. This approach enables PVNet to handle a variety of object shapes and appearances, and it has performed well in multiple 6D pose estimation benchmarks.

As with any other framework, PVNet has limitations, which include the following:

1. Texture-dependent: PVNet works best with objects with a unique texture or look, making applying to homogenous surfaces or reflection challenging.

2. Although PVNet has demonstrated impressive performance in various benchmarks, it may not be suited for all applications and use cases. Consequently, researchers continue investigating and creating novel ways to enhance 6D pose estimation.

Various approaches to 6D pose estimation have been introduced in the computer vision literature, some of which are as follows:

Iterative Closest Point (ICP), proposed by Zhang et al. [17], is a traditional method that refines an initial pose estimate by matching the 3D model points with the corresponding image points. Using an enhanced version of the ICP algorithm, the research provides a novel approach for rapidly and reliably registering two-point sets. The traditional point-to-point ICP is viewed as a majorization-minimization (MM) algorithm, and an Anderson acceleration strategy is presented to accelerate its convergence. They develop a robust error measure based on Welsch’s function, which is efficiently reduced using the MM method with Anderson acceleration. The suggested technique delivers accuracy comparable to or superior to SparseICP by at least an order of magnitude quicker. Vock et al. [18] proposed a method for template matching of 3D forms in point cloud data that generates transformation predictions using a targeted sampling strategy and an efficient mechanism for hypothesis confirmation. Experiments indicated an improved performance and accuracy over an unoptimized solution, with possible object detection and assembly verification applications. However, the proposed method assumes prior knowledge of the object template and may not be suitable for scenarios where the object is unknown or novel.

Significant progress has been made in 6D pose estimation using DL-based methods. Among these are CNNs, fully convolutional networks, and differentiable renderer networks. PoseCNN [19] is a CNN that estimates 6D item postures in crowded surroundings. The network predicts 3D translation by localizing the object’s centroid in the image, estimating its distance from the camera, and 3D rotation by regressing to quaternions. However, the study only works for known objects, and the input image quality can affect network performance. PVnet [13] is a neural network developed to estimate the posture of 6D objects. The network runs on 2D images and uses pixel-wise voting to determine the 6D posture of an item. It then links each keypoint with a 3D point on the object model. The network then computes the posture by aggregating the votes of all 3D points with which it is related. The method is capable of handling symmetric objects and has state-of-the-art performance on a variety of benchmarks. However, the approach requires a 3D model of the item and primarily relies on the accuracy of keypoint identification, which might be influenced by occlusion and background clutter. DenseFusion [20] is provided as a novel 6D object pose estimation method that combines information from RGB and depth images via an iterative dense fusion process, resulting in resilient and accurate object localization in congested environments. In addition, the technique uses a novel hybrid loss function that considers both 2D projection error and 3D geometric consistency. Experiments demonstrate that DenseFusion outperforms state-of-the-art approaches on many difficult datasets. However, the approach has certain restrictions. First, successful pose estimation requires depth information, which restricts its use when only RGB data are available. Second, the approach needs substantial processing resources, which may make its deployment in real-time applications hard. Finally, the efficiency of the approach may degrade when interacting with symmetric objects or objects with repeating textures.

DGECN [21] is proposed for monocular 6D pose estimation in computer vision (DGECN). The differentiable RANSAC algorithm uses the uncertainty of the predicted depth map to increase the accuracy and robustness of the output 6D pose. In contrast, the PnP algorithm via edge convolution explores topological links between 2D and 3D correspondences. However, noise and occlusion in the scene can impair the edge detection algorithm’s accuracy, which affects the proposed method’s performance. Therefore, the proposed method is also limited to monocular 6D pose estimation. ConvPoseCNN2 [22] is an end-to-end neural network for predicting the 6D pose of objects in cluttered scenes. The proposed technique employs CNNs to predict dense 6D poses and a refinement network to increase the accuracy of initial pose estimations. However, this method has limited occlusion robustness: The proposed method relies on visible parts of the object and may not work effectively when severe occlusion occurs.

Despite the significant advancements in 6D pose estimation, existing geometric and feature-based methodologies manifest certain limitations. Focusing on geometric techniques, the paper by Lepetit et al. [23] introduced an efficient approach to the PnP problem, known as Efficient Perspective-n-Point (EPnP). While reported as faster and more accurate compared to its predecessors, it, like other PnP-based methodologies, relies heavily on establishing precise correspondences between 3D points and 2D projections. This requirement poses challenges when confronted with occlusions, noise, or sparse data.

Moving toward feature-based strategies, Lowe’s paper [24] presented the SIFT, a method proficient in identifying and describing local image features. While its robustness to changes in scale, orientation, and affine distortion is commendable, it does exhibit limitations. Specifically, its performance is compromised under varying illumination conditions and in the presence of homogeneous textures or repetitive patterns. These conditions hinder the reliable execution of pose estimation.

Similarly, the paper by Bay et al. [25] introduced SURF designed to expedite and enhance the efficiency of the feature detection and description process. Notwithstanding its efficiency, SURF shares SIFT’s limitations. Specifically, it performs poorly when dealing with objects with less textured or repeating patterns. Furthermore, its reliability is diminished under varying lighting conditions and noise.

These traditional methods, albeit instrumental in the evolution of 6D pose estimation, collectively encounter shared obstacles. These include susceptibility to noise, varying lighting conditions, and occlusions. Additionally, their computational intensity might be unsuitable for real-time applications. They also frequently struggle with objects lacking distinct textures or having complex or non-rigid structures. This necessitates exploring more robust and computationally efficient techniques, with DL-based approaches gaining increasing attention in pursuing improved 6D pose estimation.

The attention mechanism [26] is a powerful technique in DL that has been used to enhance the performance of several computer vision tasks, such as object identification, segmentation, and posture estimation [27]. Compared to the existing works, this study’s novelty lies in the innovative integration of the attention mechanism into PVNet. In traditional methods, such as geometric and feature-based approaches, limitations exist due to susceptibility to noise, occlusion, and changes in lighting conditions. However, PVNet successfully addresses these limitations and effectively estimates 6D poses for a wide range of object shapes and appearances, making it a critical advancement in the field. Nevertheless, even with these achievements, PVNet is primarily texture-dependent. Hence, it performs best with objects with unique textures, challenging its application on homogeneous surfaces or reflections.

It can be noted that traditional methods for 6D pose estimation often employ geometric or feature-based techniques, which are prone to errors due to noise, occlusion, and lighting variations. On the other hand, DL approaches, such as CNNs and PCNs, have mitigated some of these issues. The work extends these advancements by integrating attention mechanisms into PVNet, enhancing its robustness in complex environments. Pixel-wise unit vectors of object keypoints, important in traditional methods for establishing 3D–2D correspondences, are less critical in this new model. Instead, the model leverages the object’s unique texture for segmentation and pose estimation, offering an innovative solution to traditional 6D pose estimation challenges.

In response to this limitation, our work presents a novel approach to embedding a spatial attention module into the PVNet structure. The attention mechanism, a powerful tool in DL, improves performance by selectively highlighting the most informative features of the object of interest while suppressing less relevant or distracting ones. As such, it significantly reduces the impact of noise and occlusion, offering a more accurate pose estimation. This approach is an innovative leap from the traditional methods, effectively addressing their limitations while offering enhanced performance. Adding attention mechanisms to our 6D pose estimation model, PVNet aims to enhance performance in complex, cluttered, and occluded environments. Drawing from its success in image segmentation, the attention module operates on intermediate feature maps to selectively focus on crucial spatial regions and feature channels. This approach improves the model’s robustness and accuracy by highlighting important features and suppressing noise, making it more applicable in real-world scenarios.

The contributions of this work can be summarized as follows:

1. Development of an attention-integrated PVNet: The primary contribution of our work is the innovative integration of an attention mechanism into the existing PVNet structure for 6D pose estimation. This novel approach represents a critical advancement in the field by effectively addressing the limitations of traditional methods and enhancing pose estimation precision and robustness.

2. Spatial attention mechanism design: Our work successfully designed and implemented a spatial attention module that selectively weights feature maps according to their spatial positions. This design represents an important contribution to DL techniques applied in computer vision tasks.

3. Adaptive to diverse object characteristics: By incorporating an attention mechanism into PVNet, our approach has proven to adapt effectively to various object shapes, textures, and appearances. This flexibility in handling diverse objects is an important contribution to the practical application of 6D pose estimation in real-world scenarios.

4. Robust performance under challenging conditions: The attention-integrated PVNet exhibits enhanced robustness under challenging conditions, such as noise, occlusion, and lighting variations. This improved performance under difficult conditions makes the method a substantial contribution to scenarios where pose estimation has traditionally been problematic.

5. Empirical validation: Our method was validated on a widely-used 6D pose estimation benchmark dataset, achieving state-of-the-art performance. This empirical validation demonstrates our method’s efficacy and provides a basis for further research and advancement.

6. Potential for application in various sectors: The precision and robustness of our proposed method open avenues for its application in critical sectors, such as robotics, autonomous driving, and AR. The potential for real-world application represents a significant contribution with a broad-reaching impact.

This work is structured as follows: Section 1 introduces the dataset and the premise of the suggested methods. Section 2 then details the experiments. The results and comprehensive discussions are offered in Section 3. Finally, the conclusion is presented in Section 4.

2 Method

This research proposes an innovative method for estimating the 6-DoF pose of an object. Pose estimation aims to identify objects within an image and determine their 3D orientations and translations. The process incorporates two neural network models: image segmentation, unit vector predictions, the RANSAC voting procedure, and the PnP solver algorithm. A comprehensive and detailed account of the procedure is presented below:

1. Input RGB image: The process begins by feeding the model with an RGB image of the object. The aim is to estimate the 6-DoF pose of this object.

2. Image segmentation: The RGB image is then inputted into the first neural network model, which performs the task of image segmentation. This step aims to isolate the object pixels in the image by generating a mask that distinguishes the object’s pixels from the rest of the image.

3. Keypoint predictions: Concurrently, the second neural network model produces pixel-wise unit vector predictions for each identified keypoint in the training data. These predictions symbolize the direction of the keypoints within the object’s coordinate system.

4. RANSAC voting procedure: After identifying object pixels via the image segmentation, the unit vector predictions associated with each keypoint are used in a RANSAC voting procedure. This step aims to create an array of weighted hypotheses. The weight of each hypothesis is determined by its alignment with the other identified object pixels.

5. Weighted average computation: Following the RANSAC voting, the generated hypotheses are weighted and averaged to yield a value for each keypoint. This results in a set of 2D keypoints for the object within the image.

6. Application of PnP solver function: The final stage pairs the derived 2D keypoints with corresponding 3D keypoints. This pairing can be accomplished either manually or through a labeled keypoint dataset. The associated keypoints are then introduced to a PnP solver function. Upon receiving 2D and 3D keypoints, the PnP solver returns the camera’s rotation and translation vectors within the object’s coordinate system. These vectors describe the 6-DoF pose of the object within the image.

The steps outlined above describe the flow of our proposed methodology, as depicted in Figure 1, While Algorithm 1 explains the proposed method’s pseudocode. The methodology aims to be a robust and accurate tool for object pose estimation, allowing for the reproducibility of results, thereby fostering trust and facilitating further advancement in the field.

Figure 1

Overview of the proposed method diagram.

Algorithm 1: The pseudocode of the proposed method.

3 Results and discussion

The benchmark LINEMOD dataset tests how well the proposed 6D pose estimation model with attention works. In addition, the method is also tested in challenging situations, such as occlusion, clutter, or different lighting. The approach performed effectively under these conditions and maintained high accuracy. In Table 1, the first column represents the object, the second column represents ADD, the third column represents MSE, the fourth column represents MAE, and the fifth column represents the loss.

Our experiments with the LINEMOD dataset objects served distinct purposes as follows:

• Duck and phone: These objects typify everyday items. The model achieved ADDs of 0.9992 and 0.9979, respectively, underscoring its general applicability. Additional analysis shows that the model successfully handled the varied textures and lighting conditions these objects were presented in.

• Lamp and iron: These objects have intricate shapes. The model posted ADDs of 0.9971 and 0.9979, attesting to its proficiency in managing complex geometric structures. Further analysis revealed that the attention mechanism was particularly effective in identifying the complex features of these objects.

• Hole puncher and glue: These are commonly found in cluttered environments like offices. With ADDs of 0.9971 and 0.9990, the results validate the model’s robustness in cluttered settings. Additional analysis confirmed the model’s ability to focus on the object, disregarding the surrounding clutter.

• Egg box and driller: These objects are usually located in specialized settings. The model’s ADDs of 0.9977 and 0.9979 reflect its adaptability to such specialized environments. Additional analysis found that the model effectively managed these objects’ unusual shapes and materials.

• Can and camera: Ubiquitous objects are meant to validate the model’s universal applicability. The model achieved ADDs of 0.9874 and 0.9978, respectively. Further analysis showed that the model was less susceptible to common pitfalls like occlusion and background noise.

• Bench vise, ape, and cat: These were chosen to test the model’s performance on objects with varied textures and complexities. With ADDs ranging from 0.9974 to 0.9996, the model also excelled in this category. Additional analysis indicated that the attention mechanism was proficient in identifying complex textures and patterns.

In addition to quantitative metrics, a qualitative evaluation of the model was performed. The findings of a selection of tough test images from the LINEMOD dataset were visually examined. The model consistently generated accurate and aesthetically appealing posture estimates, accurately localizing objects despite obstructions and crowded backdrops. Furthermore, the attention methods directed the model to focus on informative areas efficiently, resulting in precise posture estimation and enhanced object localization.

The results demonstrate that including the attention mechanism in the 6D pose estimate framework is beneficial. Furthermore, significant decreases in ADD, MSE, and MAE indicate that the attention mechanism considerably enhanced the accuracy and precision of pose estimation. These findings validate the potential of attention processes for strengthening 6D pose estimation tasks and emphasize their benefits in producing more accurate and reliable pose estimations.

The integration of the attention mechanism into the 6D pose estimation model provides several important benefits:

1. Enhanced localization: The attention mechanism enables the model to concentrate on relevant object regions, enhancing object localization precision by collecting important details and lowering false positives.

2. Refinement of pose estimation: By focusing selectively on informative regions, the attention mechanism assists the model in refining its pose estimation, resulting in more exact and trustworthy 6D pose estimations.

3. Robustness in challenging conditions: The attention method improves the model’s capacity to manage occlusions, crowded backdrops, and lighting fluctuations, resulting in robust performance and accurate pose estimation in even the most challenging circumstances.

4. Improved performance metrics: The attention-based model showed big improvements in performance metrics like ADD, MSE, and MAE, which means it is more accurate and precise, making fewer mistakes than baseline methods.

3.1 Qualitative evaluation

In addition to quantitative measures, a qualitative evaluation of the attention mechanism-based 6D pose estimation model was done. A visual examination was performed on a subset of challenging test images from the LINEMOD dataset. The model consistently generated accurate and aesthetically pleasing pose estimates, accurately localizing objects despite obstructions and cluttered backgrounds. Furthermore, the attention methods directed the model to focus on informative regions efficiently, resulting in precise posture estimation and enhanced object localization. Figure 2 displays some 6D pose estimation results. The green boxes reflect the ground truth postures, whereas the blue boxes represent the model predictions.

Figure 2

Some 6D pose estimation results.

3.2 Testing the method on an external object

The proposed method’s generalization ability was meticulously assessed by testing it on an external object divergent from those within the LINEMOD dataset. Utilizing a script from a publicly available source “https://github.com/F2Wang/ObjectDatasetTools,” object masks, bounding box labels, and a 3D reconstructed object mesh were generated for an RGB-D camera-captured object sequence, forming the basis for this evaluation.

The method’s performance was quantitatively evaluated using several metrics, including ADD, with a score of 0.9736, MAE of 0.0318, MSE of 0.0091, and a loss value of 0.0046. These metrics collectively provide a nuanced insight into the method’s ability to accurately estimate the 6D pose of the external object, with the ADD metric being particularly pertinent to the pose estimation task.

Figure 3 elucidates the practical application of the proposed method, depicting an example of a tested object referred to as “timer.” In this visualization, the ground truth postures are delineated by green boxes, while the blue boxes symbolize the model predictions. The congruence between the model predictions and the ground truth postures is emblematic of the method’s proficiency in accurately estimating the 6D pose of the external object.

Figure 3

An example of predicting 6D poses for an object outside the tested dataset.

The results garnered from this evaluation substantiate the proposed method’s capability to generalize beyond the confines of the LINEMOD dataset, accurately predicting the 6D pose of an object not previously encountered during training. This demonstration of generalization ability is pivotal, as it underlines the method’s potential for broad-spectrum applicability in real-world scenarios, encompassing a diverse range of objects and conditions. Through this rigorous evaluation of an external object, the proposed method showcases its robustness and adaptability, quintessential attributes for practical deployments in 6D pose estimation tasks.

4 Conclusion

This research marks a significant advancement in the field of 6D pose estimation by successfully incorporating attention mechanisms into existing frameworks. Theoretically, our model demonstrates how attention processes can selectively focus on informative regions, thereby significantly enhancing the precision and accuracy of pose estimation. This contribution enriches the academic discourse surrounding the interplay between attention mechanisms and pose estimation, offering a new benchmark for future studies. On the practical side, the model’s increased accuracy and robustness have immediate ramifications for various industries. Robotics and manufacturing sectors can benefit from more efficient object manipulation and assembly, while healthcare could see advances in surgical robotics and patient monitoring. Furthermore, the model promises to improve object recognition in autonomous driving systems and enrich user experiences in AR.

Our contributions to the research landscape are manifold. We introduced attention mechanisms into 6D pose estimation, significantly improving key performance metrics like ADD, MSE, and MAE. We validated the model’s robustness through extensive quantitative and qualitative evaluations, demonstrating its applicability in complex scenarios such as occlusions and cluttered environments. The model’s practical advantages are particularly notable in applications requiring high precision and reliability, such as industrial automation and medical procedures. Despite these contributions, the model has limitations, especially in handling dynamic lighting and rapid object movements.

The development of the attention-driven model has limitations, such as facing computational complexity due to the added attention layers and hyperparameter tuning becoming more intricate. Additionally, the model’s interpretability may be compromised due to its complexity.

For future research, several avenues are worth exploring. First, attention mechanisms could be refined through multi-scale or attention refinement techniques for even greater accuracy. Second, the model could be extended to cope with more dynamic environments by integrating temporal information or employing recurrent neural networks. Finally, the generalizability of attention mechanisms could be tested in related computer vision tasks like object tracking and instance segmentation. This study is pivotal in the theoretical understanding and practical application of attention mechanisms in 6D pose estimation.