CN110674829B - A 3D Object Detection Method Based on Graph Convolutional Attention Network - Google Patents

Abstract

The invention provides a three-dimensional target detection method based on a graph convolution attention network. (1) voxelized partitioning and random downsampling are carried out on point clouds; (2) performing local feature extraction in each grid voxel; (3) extracting a high-order feature map by middle layer convolution; (4) The area suggests the frame, category, and direction of the network predicted target. In order to enhance the connection relation between each point and adjacent points, the invention provides a feature extraction module which is based on an edge convolution form and introduces an attention mechanism, and meanwhile, a attention mechanism module with the same principle is introduced after an intermediate convolution layer, so that features of each channel of the feature map are reselected, and a more reasonable high-order feature map is obtained. The invention improves the target detection accuracy of the point cloud, and has good performance especially under the condition of serious shielding.

Description

A 3D object detection method based on graph convolutional attention network

Technical Field

The present invention relates to a computer vision three-dimensional point cloud processing method, specifically a three-dimensional target detection method.

Background Art

Object detection is a traditional visualization task that can simultaneously identify and locate objects, which is a prerequisite for realizing intelligent scenes. Today, two-dimensional detection has reached an unprecedented prosperity, but in areas such as mapping, indoor robots, and augmented reality, three-dimensional detection is significantly better than two-dimensional. It can provide more position and posture information, and is also one of the basic tasks of autonomous driving environment perception. RGB images used to be the mainstream data form for object detection tasks, but with the development of 3D sensors, lidar has become an increasingly popular detection tool in recent years.

Now, some methods based on LiDAR and cameras fuse point cloud data and image data together to achieve higher accuracy. But the fusion method also faces the problem of excessive computational cost, so the single sensor method is still competitive. Many studies have shown that point cloud is a more appropriate data form to describe the shape of objects. Point cloud can better represent Euclidean distance and does not have multi-scale problems. However, point cloud is a sparse data, which makes it difficult to directly apply two-dimensional methods.

When extracting features, most methods process points one by one and use symmetric functions to extract global features. This approach ignores the connection and relationship between points. Compared with image data, point cloud is a natural graph structure that is easy to build links. Some studies have used the idea of graph networks, considering that the relationship between adjacent points and edges helps to enhance the expression of local features, and proposed edge convolution methods. In three-dimensional convolution, considering that within the defined voxel range, many voxels are empty due to the sparsity of points, the use of sparse convolution can improve the calculation speed and reduce the loss of video memory without affecting the convolution effect.

Summary of the invention

The purpose of the present invention is to provide a three-dimensional target detection method based on a graph convolutional attention network, which can improve the accuracy of point cloud target detection and still have good performance under severe occlusion.

The object of the present invention is achieved in that:

(1) Voxelize and randomly downsample the point cloud;

(2) extracting local features in each grid voxel;

(3) The middle layer convolution extracts high-order feature maps;

(4) The region proposal network predicts the target’s bounding box, category, and orientation.

The present invention may also include:

1. The voxelization and random downsampling of the point cloud specifically include: using the voxel grid structure to divide the original point cloud, discarding outliers outside the specified range, dividing the point cloud into grids, and performing random downsampling in each voxel grid, and then numbering each grid and storing it.

The storage is performed using a hash table.

2. The local feature extraction in each grid voxel specifically includes: within each voxel grid, using a graph attention network module to extract features of corresponding points.

The method of using the graph attention network module to extract features of corresponding points is as follows: first, each point is connected to the surrounding adjacent points to form a graph structure with Euclidean distance as the judgment standard, and at the same time, each point is connected to the point itself by an edge, and the coordinates of the two endpoints of each edge and other information are extracted as the initial features of the edge, and then a convolution operation is performed on the edge, and finally, after the selection of a symmetric function, voxel-level features are obtained.

Before the side convolution operation, an attention mechanism is used to select initial features.

3. The intermediate layer convolution extracts the high-order feature map specifically including: using the sparse convolution method to compress the feature map into a dense structure, and then mapping it back to the original sparse spatial representation after convolution; after convolution abstraction, using the attention mechanism to redistribute the weights of different channels to obtain an attention map corresponding to the feature map, and superimposing the attention map on the high-order feature map obtained by convolution to obtain the final three-dimensional feature map.

4. The region proposal network predicts the target's box, category, and direction specifically by extracting features from the high-order feature map that has undergone multiple layers of convolution, and then using three separate fully connected layers to calculate the predicted values of the bounding box, category, and direction corresponding to each anchor point.

The three-dimensional target detection method based on graph convolutional attention network of the present invention is characterized by strengthening the expression of local relationships of point clouds and optimizing the feature selection process. The present invention uses the edge convolution method that can express the relationship between adjacent points for feature extraction of target detection. In the feature selection stage of the initial point, the attention mechanism is used to select the initial physical features that are more important for feature expression, so as to obtain better extracted features. In the process of convolution of the middle layer, multi-channel feature data is also generated. The present invention uses the idea of attention mechanism to optimize the convolution results, strengthen the proportion of channels with major influence, and obtain a more expressive feature map.

The point cloud data of a typical scene contains more than 100k points, so we consider using a specific data structure to preprocess the point cloud, namely voxelization. First, the original points are divided into voxels and the point features are extracted first, and then the downsampled voxel signals enter the convolution and region proposal to obtain the 3D bounding box.

The present invention considers strengthening the relationship representation between the underlying original points in the feature extraction process, and utilizes the idea of graph network in feature extraction. At the same time, in order to better strengthen the feature expression, an attention mechanism that imitates human cognitive acuity is considered, so that the multi-channel selection of features is more intelligent. The present invention applies the attention mechanism before the initial feature selection of the graph network edge convolution and after the sparse convolution feature map processing, which improves the expression power of the neural network module and makes the feature expression of each stage more interpretable.

The present invention has the following advantages:

1. The present invention uses a graph convolution method with an attention mechanism in the feature representation process of each voxel, which can better describe the relationship between each point in the point cloud and extract more expressive features.

2. After the intermediate layer convolution, the present invention uses the attention mechanism to redistribute the weights of the obtained high-order feature map, thereby obtaining a more reasonable high-order feature map.

3. The above two improvements work together, and the present invention can improve the accuracy of three-dimensional target detection in vehicle detection.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1: Feature extraction module based on graph network attention structure, where e represents edge, x represents point, i and j represent point numbers;

Figure 2: Voxel feature extraction;

Figure 3: Sparse convolution in the middle layer with attention mechanism;

Figure 4: Overall process.

DETAILED DESCRIPTION

The present invention is described in more detail with reference to the following examples.

Step 1: Voxelization and clustering of point clouds

The original point cloud data with more than 100k points is structured and downsampled by voxelization. First, points outside a certain range are cut off, and only points within the range of D, H, and W under the x, y, and z axes are retained. Since the number of points in a point cloud is too large, within the extraction range, the entire point cloud is divided using small voxel grids of size v _d , v _h , and v _w .

In order to solve the problem of uneven distribution of points in each voxel, this embodiment uses a random downsampling method to ensure that the number of points in each voxel does not exceed T. Finally, the processed voxel structures are numbered and stored in a hash table to eliminate voxels with empty internal points.

Step 2: Point cloud feature extraction in voxels

After voxelizing the original point cloud, in order to obtain voxel-level features, this embodiment uses a graph attention network module to extract features for each voxel.

为一个图，其中包括了n个点组成的点集

以及点之间的边集

例如，本发明定义一个d维的临近图，对于每个点x_i，在

中包含了(i，j_i1)，...，(i，j_ik)形式的边集，其中i与j都为点的编号，于是定义边特征为

其中h_θ与下列公式中的H为对称函数。Point cloud is a natural graph structure. In the feature extraction of point cloud, each point is usually considered separately and the connection between points is ignored.

is a graph consisting of a set of n points

and the set of edges between the points

For example, the present invention defines a d-dimensional proximity graph, where for each point x _i ,

contains an edge set of the form (i, j _i1 ), ..., (i, j _ik ), where i and j are both point numbers, so the edge feature is defined as

Where h _θ and H in the following formula are symmetric functions.

的信息作为初始的特征选择。此时，边特征的每一个通道对于总特征表述的贡献是不同的，于是，添加了一种注意机制方法。在边卷积的多层感知操作之后，使用一种对称操作H对边级的特征进行提取，得到对应的点级的特征。随后，通过将点级特征X＝{x′₁，...，x′_n}进行另一个对称操作提取得到最终的体素级特征。Generally speaking, a point cloud has three dimensions to represent its coordinates in the real world. In this embodiment, when describing the edge between two points, the center point _xi and the points connected to it by the h operation are combined.

The information is used as the initial feature selection. At this time, each channel of the edge feature contributes differently to the overall feature representation, so an attention mechanism is added. After the multi-layer perception operation of the edge convolution, a symmetry operation H is used to extract the edge-level features to obtain the corresponding point-level features. Subsequently, the final voxel-level features are extracted by performing another symmetry operation on the point-level features X = {x′ ₁ , ..., x′ _n }.

Step 3: Sparse convolution in the middle layer

This implementation uses 3D sparse convolution operations as convolution intermediate layers. Assume that ConvMD(c _in , c _out , k, s, p) is a convolution operator, where c _in and c _out are the number of input and output channels, and k, s, p correspond to the kernel size, stride size, and padding size, respectively. Each convolution operation contains 3D convolution, BatchNormal layer, and Relu layer. Finally, after converting the sparse map to a dense map, a high-level feature map is obtained, and an attention module is added to it.

There are many different scale feature maps during the convolution operation. Obviously, the contribution of each dimension of feature to the overall feature has different importance. In order to improve the description of the feature map and make it more reasonable, the present invention adds the attention map to the original feature map.

其中H为特征图高度，W为特征图宽度，C为通道数。然后使用avg-pool ing操作来提取每个通道从而得到一个提取特征，因此获得统计得到通道权重

然后使用多层感知来获得每个维度的一些高级特征，于是最终的注意图为s_c＝F_e(z_c，W)，其中F_e为提取函数。This implementation uses the SE attention module to generate attention feature maps. First, let the dense feature map input be

Where H is the feature map height, W is the feature map width, and C is the number of channels. Then use the avg-pooling operation to extract each channel to obtain an extracted feature, so as to obtain the channel weight statistically

Then, multi-layer perception is used to obtain some high-level features of each dimension, so the final attention map is _sc = _Fe ( _zc , W), where _Fe is the extraction function.

After _scaling function F, the attention feature map is added to the original map to obtain the comprehensive feature map of the final output

This attention mechanism operation is added after the intermediate layer, which can aggregate high-level information into the final intermediate layer feature map, thereby providing more information for subsequent region proposals.

Step 4: Region Proposal Network

Region Proposal Network (RPN) has become a typical embedding module in many detection frameworks. In this implementation, an end-to-end form similar to SSD is used as the region proposal architecture. The input of the region proposal layer is the feature map extracted by the intermediate layer. A region proposal layer contains a convolutional layer, a BatchNormal layer, and a Relu layer. After each individual RPN layer, the feature map is upsampled to the same fixed size and the maps are concatenated together. Finally, three 1×1 convolutions are used to generate predictions for the bounding box, class, and direction.

Claims (1)

1. A three-dimensional target detection method based on a graph convolution attention network is characterized by comprising the following steps of:

step one: voxel division clustering of point cloud

Structuring and downsampling original point cloud data in a voxelized mode, discarding outliers outside a specified range, dividing the point cloud into grids, randomly downsampling in each voxel grid, numbering each grid, and storing; using a method of random downsampling, so that the number of points in each voxel is not more than T; finally numbering the processed voxel structure, and storing the voxel structure in a hash table mode, so that voxels with empty internal points are eliminated;

step two: point cloud feature extraction in voxels

After voxelization of the original point cloud, extracting features of each voxel by using a graph annotation network module in order to obtain voxel level features;

the point cloud is a natural graph structure, and each point is conventionally considered independently and negligibly in the feature extraction of the point cloudDefinition of Point-to-Point relationship

Is a graph comprising a set of n points

And edge set between points ++>

The point cloud has three dimensions to represent its real world coordinates, and after the multi-layer perceptive operation of edge convolution, a symmetry operation H is used to extract edge-level features, which are obtained by extracting the corresponding point-level features by using the point-level features x= { X' ₁ ，...，x′ _n Performing another symmetrical operation extraction to obtain final voxel level characteristics;

step three: middle layer sparse convolution

Using three-dimensional sparse convolution operation as a convolution intermediate layer, assume ConvMD (c _in ，c _out K, s, p) is a convolution operator, where c _in And c _out The number of inputs and the output channels, k, s, p correspond to kernel size, stride size, and fill size, respectively; each convolution operation includes a 3D convolution, a batch normal layer and a Relu layer; after the sparse mapping is converted into dense mapping, an advanced feature mapping is obtained, and an attention module is added;

using SE attention module for generating attention feature map, first, let dense feature map input as

Wherein H is the height of the feature map, W is the width of the feature map, and C is the number of channels; each channel is then extracted using an avg-mapping operation to obtain an extracted feature, thus obtaining statistically derived channel weights

Multilayer perceptions are then used to obtain some advanced features for each dimension, with the final attention being given to s _c ＝F _e (z _c W), where F _e Is an extraction function;

at the scaling function F _scale Thereafter, the attention feature map is added to the original map to obtain a final output integrated feature map

Step four: regional advice network

Using an end-to-end form similar to SSD as a region suggestion architecture, wherein the input of a region suggestion layer is a feature map extracted by an intermediate layer, and one region suggestion layer comprises a convolution layer, a Batchnormal layer and a Relu layer; after each individual RPN layer, upsampling the feature maps to the same fixed size and concatenating the maps together; finally, three 1×1 convolutions are used to generate the predicted values for bounding boxes, classes and directions.

Images