CN115311553A - Target detection method and device, electronic equipment and storage medium - Google Patents

Abstract

An embodiment of the present invention provides a target detection method. The method includes: acquiring an image to be detected, where the image to be detected includes a rotating target to be detected; and performing feature extraction on the image to be detected by using a trained target detection model to obtain the Classification feature and auxiliary feature of the rotating object to be detected; perform auxiliary processing on the classification feature based on the auxiliary feature to obtain the detection result of the rotating object to be detected. By extracting the classification features and auxiliary features of the rotating target to be detected, and using the auxiliary features to perform auxiliary processing on the classification features, the detection results of the rotating target to be detected are obtained, and the anchor does not need to be designed, so no non-maximum value is required. Suppression improves the detection accuracy of rotating object detection, thereby improving the detection performance of rotating object detection.

Description

Target detection method, device, electronic equipment and storage medium

technical field

The invention relates to the field of artificial intelligence, in particular to a target detection method, device, electronic equipment and storage medium.

Background technique

Rotating target detection refers to detecting a target with a rotating direction, that is, the center point, width, height, and angle of the target need to be detected. It is more common in target detection of top view, such as remote sensing image target detection, aerial image target detection, etc. The current rotating target detection is often based on the anchor point and non-maximum value suppression target detection algorithm, because the size and position of the rotating target in the image are not the same, the target detection algorithm based on the anchor has some inherent shortcomings, such as wanting to detect all The size and position of the rotating target, the design of the anchor will be very complicated. It is necessary to design different proportions and different sizes, and the proportion and size of the anchor are not designed properly, which will also affect the subsequent non-maximum value suppression, resulting in error accumulation. Thus affecting the detection accuracy of the target detection model. Therefore, the existing rotating target detection algorithm has the problem of low detection accuracy.

Contents of the invention

An embodiment of the present invention provides a target detection method, which aims to solve the problem of low detection accuracy of the rotating target detection algorithm in the existing target detection process. By extracting the classification features and auxiliary features of the rotating target to be detected, the auxiliary features are used to assist the classification features, so as to obtain the detection result of the rotating target to be detected, and there is no need to design the anchor, so there is no need for non-maximum value Suppression, improves the detection accuracy of rotating object detection, and then improves the detection performance of rotating object detection.

In the first aspect, an embodiment of the present invention provides a target detection method, the target detection method is used for detecting a rotating target, and the method includes:

Acquiring an image to be detected, the image to be detected includes a rotating target to be detected;

performing feature extraction on the image to be detected through the trained target detection model to obtain classification features and auxiliary features of the rotating target to be detected;

An auxiliary process is performed on the classification feature based on the auxiliary feature to obtain a detection result of the rotating target to be detected.

Optionally, the trained target detection model includes a feature extraction network, a feature fusion network, a classification feature output network, and an auxiliary feature output network, and the trained target detection model performs feature extraction on the image to be detected, The classification features and auxiliary features of the rotating target to be detected are obtained, including:

performing feature extraction on the image to be detected through the feature extraction network to obtain multi-scale features of the image to be detected;

performing feature fusion on the multi-scale features through the feature fusion network to obtain the fusion features of the image to be detected;

The fusion feature is predicted by the classification feature output network to obtain the classification feature of the rotating target to be detected, the classification feature includes a feature channel, and different types of rotating targets to be detected correspond to different feature channels;

The fusion feature is predicted by the auxiliary feature output network to obtain the auxiliary feature of the rotating target to be detected.

Optionally, the auxiliary feature includes a height-width feature and a rotation angle feature, and performing auxiliary processing on the classification feature based on the auxiliary feature to obtain a detection result of the rotating target to be detected includes:

Carrying out key point extraction on the classification features to obtain target key points of the rotating target to be detected;

Based on the target key point, indexing a corresponding target height and width attribute in the height and width feature, and indexing a corresponding target rotation angle attribute in the rotation angle feature;

A detection result of the rotating object to be detected is obtained based on the key point of the object, the attribute of the height and width of the object, and the attribute of the rotation angle of the object.

Optionally, before performing feature extraction on the image to be detected through the trained target detection model to obtain classification features and auxiliary features of the rotating target to be detected, the method further includes:

Obtain a training data set, the training data set includes a sample image and a label frame, the sample image includes a sample rotation target, and the label frame is a label frame of the sample rotation target;

Obtain the target detection model, and train the target detection model through the training data set to obtain the trained target detection model. The target detection model includes a feature extraction network, a feature fusion network, a classification feature output network, and a height and width feature. Output network and rotation angle feature output network.

Optionally, the acquisition of the target detection model, and training the target detection model through the training data set to obtain a trained target detection model, including:

Inputting the sample image into the target detection model to obtain a sample detection frame corresponding to the sample rotation target;

Encoding the sample detection frame and the label frame respectively by a preset encoding function, respectively obtaining a sample function distribution corresponding to the sample detection frame and a label function distribution corresponding to the label frame;

adjusting the network parameters of the target detection model according to the metric distance between the sample function distribution and the label function distribution;

The network parameter adjustment process of the target detection model is iterated until the target detection model converges or reaches a preset number of iterations to obtain a trained target detection model.

Optionally, the inputting the sample image into the target detection model to obtain the sample detection frame corresponding to the sample rotation target includes:

Process the sample image through the target detection model to obtain a sample feature map corresponding to the sample image, and construct a matrix grid for the sample feature map according to the height and width of the sample feature map, the sample The feature image includes a classification feature map, a height-width feature map, and a rotation angle feature map corresponding to the sample image;

Establishing an index of a height and width attribute for each grid point in the height-width feature map, and establishing an index of a rotation angle attribute for each grid point in the rotation angle feature map;

According to each grid point in the sample feature map and its corresponding index attribute, a sample detection frame corresponding to the sample rotation target is obtained.

Optionally, the marked key points are included in the marked frame, and the network parameter adjustment of the target detection model is performed according to the measurement distance between the sample function distribution and the marked function distribution, including:

Calculate the first loss between the sample key point and the label key point according to the sample key point of the classification feature map;

converting the metric distance into a second loss through a preset conversion function;

Based on the first loss and the second loss, network parameters are adjusted for the target detection model.

In a second aspect, an embodiment of the present invention provides a target detection device, the device comprising:

A first acquiring module, configured to acquire an image to be detected, where the image to be detected includes a rotating target to be detected;

The extraction module is used to perform feature extraction on the image to be detected through the trained target detection model, and obtain classification features and auxiliary features of the rotating target to be detected;

A processing module, configured to perform auxiliary processing on the classification feature based on the auxiliary feature to obtain a detection result of the rotating target to be detected.

In a third aspect, an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the computer program, The steps in the target detection method provided by the embodiment of the present invention are realized.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the target detection method provided by the embodiment of the present invention is implemented. step.

In the embodiment of the present invention, the image to be detected is obtained, and the image to be detected includes the rotating target to be detected; the feature extraction is performed on the image to be detected through the trained target detection model, and the classification features and the classification features of the rotating target to be detected are obtained. Auxiliary features: performing auxiliary processing on the classification features based on the auxiliary features to obtain a detection result of the rotating target to be detected. By extracting the classification features and auxiliary features of the rotating target to be detected, the auxiliary features are used to assist the classification features, so as to obtain the detection result of the rotating target to be detected, and there is no need to design the anchor, so there is no need for non-maximum value Suppression, improves the detection accuracy of rotating object detection, and then improves the detection performance of rotating object detection.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a flow chart of a target detection method provided by an embodiment of the present invention;

Fig. 2 is a schematic structural diagram of a target detection model provided by an embodiment of the present invention;

Fig. 3 is a schematic structural diagram of a target detection device provided by an embodiment of the present invention;

Fig. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

Please refer to FIG. 1. FIG. 1 is a flowchart of a target detection method provided by an embodiment of the present invention. As shown in FIG. 1, the target detection method is used for detecting a rotating target, and the target detection method includes the following steps:

101. Acquire an image to be detected.

In the embodiment of the present invention, the image to be detected includes a rotating target to be detected. The image to be detected above can be a side image, a top view image, an up view image, etc., the side image can be an image taken from the side of the target, the top view image can be an image taken from above the target, and the look up image can be taken from the bottom of the target image.

The above-mentioned rotating target to be detected may be a solid target such as a person, a vehicle, an airplane, a building, or an object.

102. Perform feature extraction on the image to be detected through the trained target detection model, and obtain classification features and auxiliary features of the rotating target to be detected.

In the embodiment of the present invention, the image to be detected can be input into the trained target detection model, and feature extraction is performed on the image to be detected through the target detection model to obtain classification features and auxiliary features of the rotating target to be detected.

In a possible embodiment, before the image to be detected is input into the trained target detection model, the image to be detected can be preprocessed, and the above preprocessing can include normalization of image pixels and scaling of width and height to H ₀ × W ₀ size, where the sizes of H ₀ and W ₀ are integer multiples of 32.

The classification features of the rotating objects to be detected include category information of the rotating objects to be detected, for example, the categories of the rotating objects to be detected are people, vehicles, airplanes, buildings, objects, and the like. The above-mentioned auxiliary features of the rotating object to be detected may include attribute information such as height, width and rotation angle of the rotating object to be detected.

Specifically, the trained target detection model includes a classification feature branch structure and an auxiliary feature branch structure. The trained target detection model can first extract common features from the image to be detected, and output the corresponding classification features through the classification feature branch structure. The feature branch structure outputs the corresponding auxiliary features. The classification feature branch structure has different structural parameters from the auxiliary feature branch structure.

Further, the above target detection model may be constructed based on a deep convolutional neural network, and after training the deep convolutional neural network, a trained target detection model is obtained. Specifically, sample images can be collected. The sample images include sample rotation targets. The sample rotation targets can be people, vehicles, aircraft, buildings, objects, etc., and the sample rotation targets in the sample images are marked to obtain corresponding label data. The labels include The category label corresponding to the classification feature, and the attribute label corresponding to the auxiliary feature, the attribute label can include height and width labels and rotation angle labels. The deep convolutional neural network is trained through sample images and corresponding label data, so that the deep convolutional neural network learns the classification features of the rotating target and the auxiliary features of the rotating target for output, and the trained target detection model is obtained after training.

103. Perform auxiliary processing on the classification feature based on the auxiliary feature to obtain a detection result of the rotating target to be detected.

In the embodiment of the present invention, the auxiliary features of the rotating object to be detected may include attribute information such as the height, width, and rotation angle of the rotating object to be detected, and the attribute information corresponding to the auxiliary feature index may be added to the classification feature, thereby obtaining the rotating object to be detected test results.

The above detection results may include the position, category, height, width, and rotation angle of the rotating object to be detected.

In the embodiment of the present invention, the image to be detected is obtained, and the image to be detected includes the rotating target to be detected; the feature extraction is performed on the image to be detected through the trained target detection model, and the classification features and the classification features of the rotating target to be detected are obtained. Auxiliary features: performing auxiliary processing on the classification features based on the auxiliary features to obtain a detection result of the rotating target to be detected. By extracting the classification features and auxiliary features of the rotating target to be detected, the auxiliary features are used to assist the classification features, so as to obtain the detection result of the rotating target to be detected, and there is no need to design the anchor, so there is no need for non-maximum value Suppression, improves the detection accuracy of rotating object detection, and then improves the detection performance of rotating object detection.

Optionally, the trained target detection model includes a feature extraction network, a feature fusion network, a classification feature output network, and an auxiliary feature output network. After the trained target detection model is used to perform feature extraction on the image to be detected, the image of the rotating target to be detected is obtained. In the steps of classifying features and auxiliary features, the feature extraction network can be used to extract the features of the image to be detected to obtain the multi-scale features of the image to be detected; the feature fusion network is used to perform feature fusion on the multi-scale features to obtain the fusion features of the image to be detected; Predict the fusion features through the classification feature output network to obtain the classification features of the rotating target to be detected. The classification features include feature channels, and different types of rotating targets to be detected correspond to different feature channels; the fusion features are predicted through the auxiliary feature output network. , to obtain the auxiliary features of the rotating target to be detected.

In the embodiment of the present invention, the above-mentioned feature extraction network may be composed of a backbone network, such as VGG19, ResNet, MobileNet, etc., and the embodiment of the present invention does not impose any restrictions on the feature extraction network. The above feature extraction network can improve the features of the image to be detected at different scales, and obtain the multi-scale features of the image to be detected. It should be noted that in the feature extraction network, due to the existence of the downsampling layer, as the calculation depth of the feature extraction network becomes deeper, the extracted feature scale becomes smaller.

The above-mentioned feature fusion network may include an upsampling layer and a fusion layer, through which the features of smaller size are upsampled through the upsampling layer, so that the features of smaller size are upsampled into features of larger size, and then the upsampling layer is further sampled through the fusion layer The final features are fused with features of the same size. Specifically, the feature fusion network extracts multi-scale features from the features of different stages of the feature extraction network, and performs 2 times upsampling on the small-scale features one by one and fuses them with the features of the same scale from the feature extraction network. Finally, the high-scale features The fused features are output to the prediction network.

The above-mentioned classification feature output network and auxiliary feature output network can also be referred to as prediction networks. The corresponding classification features are output through the classification feature output network. The output classification features can include multiple feature channels, and each feature channel corresponds to a rotating target to be detected. a category of .

Specifically, the above-mentioned classification feature output network can be a classification feature output network based on CenterNetR, and the fusion feature can be input to the classification feature output network based on CenterNetR, and the central point heat map of the rotating target to be detected can be predicted by CenterNetR as a classification feature. The center point heat map of is distributed in different feature channels, therefore, the category of the rotating target to be detected can be determined by the feature channel.

The above-mentioned auxiliary features can also be an attribute feature output network based on CenterNetR, and the fusion feature can be input to the attribute feature output network based on CenterNetR, and the attribute information corresponding to each center point can be predicted by CenterNetR as auxiliary information, wherein the scale resolution of the above-mentioned auxiliary features The rate is the same as the scale resolution of the categorical features. The aforementioned auxiliary features may include attribute information such as the height, width and rotation angle of the rotating object to be detected. In the auxiliary feature, each position point corresponds to a set of attribute information, and the attribute information of the corresponding position can be indexed through the center point position of the classification feature.

Optionally, the auxiliary features include height-width features and rotation angle features. In the step of obtaining the detection result of the rotating target to be detected by performing auxiliary processing on the classification features based on the auxiliary features, key points can be extracted from the classification features to obtain the rotation to be detected. The target key point of the target; based on the target key point, index the corresponding target height and width attributes in the height and width feature, and index the corresponding target rotation angle attribute in the rotation angle feature; based on the target key point, target height and width attributes, and target rotation Angle attribute, get the detection result of the rotating target to be detected.

In the embodiment of the present invention, the features include a height-width feature and a rotation angle feature, the height-width feature corresponds to the height-width attribute of the rotating object to be detected, and the rotation angle feature corresponds to the rotation angle attribute of the rotating object to be detected. Each position point in the classification feature corresponds to a position point in a height-width feature and a position point in a rotation angle feature. Therefore, according to the target key point in the classification feature, the corresponding position point in the height-width feature can be indexed to the height The wide attribute is used as the target, and the corresponding position point in the rotation angle feature is indexed to the rotation angle attribute.

Specifically, the height-width feature and the rotation angle feature have the same scale resolution as the classification feature. The height of the above-mentioned classification feature is H, and the width is W. Similarly, the height and width feature is H, the width is W, and the rotation angle The height of the feature is H and the width is W. The classification feature can be a center point heat map. The center point of the heat map is the point with the highest heat value, and the center point can also be used as the target key point. Specifically, after the classification features are obtained, the classification features can be sampled through an n*n maximum pooling kernel, and the key points with high confidence are obtained as the target key points according to the preset confidence threshold. Wherein, n is smaller than H, and n is smaller than W. The role of the maximum pooling kernel is to sample the maximum value from the n*n area of the heat map. Taking n=3 as an example, the value with the highest thermal value can be sampled in the 3*3 area of the thermal map as the sampling value of the maximum pooling kernel.

After obtaining the target key point (i, j), where (i, j) represents the position coordinates of the target key point in the classification feature. The corresponding target height and width attribute (w, h) can be indexed in the height and width feature according to the target key point (i, j), where w represents the width of the rotation target to be detected in the classification feature, and h represents the rotation target to be detected in Height in categorical features. At the same time, the corresponding target rotation angle attribute θ can be indexed in the height-width feature according to the target key point (i, j), where θ represents the rotation angle of the rotation target to be detected in the classification feature.

According to the target key point (i, j), the target height and width attribute (w, h) and the target rotation angle attribute θ, the detection result (i, j, w, h, θ) of the rotating target to be detected can be obtained.

In a possible embodiment, the above-mentioned auxiliary feature may further include an offset feature, and the above-mentioned offset feature is used to describe the offset of the target key point. Specifically, the above bias feature can be obtained by adding a bias feature output network to the target detection model, and predicting the fusion feature through the bias feature output network. The bias feature and classification feature also have the same scale resolution, the height of the bias feature is H, and the width is W. Each location point in the classification feature corresponds to a location point in the bias feature. The target key point (i, j) can be indexed to the corresponding target bias attribute (dx, dy) in the bias feature, then the final position of the rotating target to be detected is (x, y), where x=i+dx , y=j+dy. Combined with the target height and width attributes (w, h) and the target rotation angle attribute θ, the detection result (x, y, w, h, θ) of the rotating target to be detected can be obtained. By obtaining the corresponding target offset attribute of the target key point in the offset feature index, the position of the rotating target to be detected can be determined more accurately.

Optionally, before performing feature extraction on the image to be detected through the trained target detection model to obtain the classification features and auxiliary features of the rotating target to be detected, a training data set can also be obtained. The training data set includes sample images and label boxes, sample The image includes the sample rotation target, and the label frame is the label frame of the sample rotation target; obtain the target detection model, and train the target detection model through the training data set to obtain the trained target detection model, the target detection model includes the feature extraction network , feature fusion network, classification feature output network, height and width feature output network, and rotation angle feature output network.

In the embodiment of the present invention, the target detection model may be trained before feature extraction is performed on the image to be detected through the trained target detection model. A sample image containing a sample rotating object can be collected for labeling to obtain a training data set, and the sample rotating object has the same category as the rotating object to be detected. Experts can mark the rotating target in the sample image to obtain a labeling frame, which includes the category of the sample rotating target, the position of the labeling frame, the height and width of the labeling frame, and the rotation angle of the labeling frame. In a possible embodiment, If the auxiliary features include bias features, then the target of the annotation box is biased. At this time, the target detection model also includes a bias feature output network. It should be noted that the classification feature output network, the height-width feature output network, the rotation angle feature output network, and the bias feature output network are all independent and parallel branch networks.

Specifically, please refer to FIG. 2. FIG. 2 is a schematic structural diagram of a target detection model provided by an embodiment of the present invention. As shown in FIG. 2, in the target detection model, the output of the feature extraction network is connected to the input of the feature fusion network , the output of the feature fusion network is respectively connected to the input of the classification feature output network, the height-width feature output network, the rotation angle feature output network and the bias feature output network.

The target detection model is trained through the training data set. During the training process, iteratively adjust the feature extraction network, feature fusion network, classification feature output network, height and width feature output network, rotation angle feature output network and bias feature output network. Network parameters until the target detection model converges or reaches the preset number of iterations to obtain a trained target detection model.

Optionally, in the step of obtaining the target detection model and training the target detection model through the training data set to obtain the trained target detection model, the sample image can be input into the target detection model to obtain the corresponding The sample detection frame; the sample detection frame and the label frame are respectively encoded by the preset encoding function, and the sample function distribution corresponding to the sample detection frame and the label function distribution corresponding to the label frame are respectively obtained; according to the difference between the sample function distribution and the label function distribution Measure the distance between them, adjust the network parameters of the target detection model; iterate the network parameter adjustment process of the target detection model until the target detection model converges or reaches the preset number of iterations, and a trained target detection model is obtained.

In the embodiment of the present invention, during the training process, the sample image may be input into the target detection model to obtain the sample detection frame output by the target detection model. The sample detection frame can be obtained through the prediction results output by the classification feature output network, height and width feature output network, rotation angle feature output network and bias feature output network in the target detection model. After the sample detection frame is obtained, the loss between the sample detection frame and the marked detection frame can be calculated, and the loss between the sample detection frame and the marked detection frame can be backpropagated to adjust the network parameters of each network in the target detection model, iteratively The above process completes the training of the target detection model.

Furthermore, in order to improve the auxiliary effect of the auxiliary features and further improve the detection accuracy of the target detection model, the embodiment of the present invention encodes the sample detection frame and the label frame respectively through the encoding function, and uses the encoding function to combine the classification features and the The auxiliary features are encoded and coupled. Specifically, the position of the label box is encoded and coupled with the width and height features, and the rotation angle features through the encoding function, so that the target detection model can learn this coupling connection, making the output of the trained target detection model more accurate. auxiliary features.

Furthermore, the above encoding function may be a nonlinear distribution function, such as a two-dimensional Gaussian distribution function. Specifically, it may be shown in the following formula:

μ=(x,y) ^T

Among them, the above (x, y, w, h, θ) is the expression form of the detection frame, (x, y) is the coordinates of the center point of the detection frame, (w, h) is the width and height of the detection frame, and θ is the detection frame The rotation angle of the rotation target in the box. Through the above formula, the detection frame (x, y, w, h, θ) is encoded into a two-dimensional Gaussian distribution form (μ, Σ), specifically, μ represents the mean value of the converted two-dimensional Gaussian distribution, and Σ represents the converted two-dimensional Gaussian distribution Covariance of a Gaussian distribution. The encoding of the sample detection frame and the label frame can be performed by the above formula.

After encoding the sample detection frame, the sample function distribution (μ ₁ , Σ ₁ ) is obtained; after encoding the annotation frame, the label function distribution (μ ₂ , Σ ₂ ) is obtained. Computes the metric distance between the sample function distribution (μ ₁ , Σ ₁ ) and the labeled function distribution (μ ₂ , Σ ₂ ). In the case where the sample image is a positive sample, the smaller the measurement distance, the more similar the sample detection frame is to the label frame, and the more consistent the detection result is with the real result; the larger the measurement distance, the less similar the sample detection frame is to the label frame. The detection results are more inconsistent with the real results. In the case where the sample image is a negative sample, the smaller the measurement distance, the less similar the sample detection frame is to the label frame, and the more consistent the detection result is with the real result; the larger the measurement distance, the more similar the sample detection frame is to the label frame. The detection results are more inconsistent with the real results.

The calculation of the above-mentioned metric distance can be performed by calculation methods such as Wasserstein distance and KL divergence. In the embodiment of the present invention, the Wasserstein distance is preferred to calculate the relationship between the sample function distribution (μ ₁ , Σ ₁ ) and the label function distribution (μ ₂ , Σ ₂ ). The metric distance of can be shown in the following formula:

Among them, d is the metric distance between the sample function distribution (μ ₁ , Σ ₁ ) and the label function distribution (μ ₂ , Σ ₂ ), and the Tr() function represents the trace of the calculated matrix.

In the training process, by encoding the detection frame and the label frame, the classification feature and the auxiliary feature are coupled through the encoding function, which improves the learning ability of the object detection model for the auxiliary feature, so that the trained object detection model can extract more Accurate helper features.

In a possible embodiment, the target detection model further includes a bias feature output network, which extracts multi-scale features of the sample image through a feature extraction network, and fuses the multi-scale features of the sample image through a feature fusion network. The fused features are predicted and processed through the classification feature output network to obtain classification features; the fused features are predicted and processed through the height and width feature output network to obtain height and width features; the fused features are obtained through the rotation angle feature output network Prediction processing to obtain the rotation angle feature; predictive processing of the fused features through the bias feature output network to obtain the bias feature. Extract the key points through the above classification features to obtain the target key points. Based on the target key points, index the corresponding target bias attributes in the offset feature, and obtain the sample rotation based on the target key points, target height and width attributes, and target rotation angle attributes. The detection result of the target, the detection result of the sample rotation target corresponds to the sample detection frame.

By adding the output network corresponding to the auxiliary features in the target detection model, the classification features are coupled with the auxiliary features, so that the trained target detection model can output more accurate auxiliary features.

Optionally, in the step of inputting the sample image into the target detection model to obtain the sample detection frame corresponding to the sample rotation target, the sample image can be processed through the target detection model to obtain the sample feature map corresponding to the sample image, and according to the sample The height and width of the feature map, constructing a matrix grid for the sample feature map, the sample feature image includes the corresponding classification feature map, height and width feature map and rotation angle feature map of the sample image; each grid point in the height and width feature map establishes a height The index of the wide attribute, and each grid point in the rotation angle feature map establishes an index of the rotation angle attribute; according to each grid point in the sample feature map and its corresponding index attribute, the sample detection corresponding to the sample rotation target is obtained frame.

In the embodiment of the present invention, the sample image may be processed by a target detection model to obtain a sample feature map corresponding to the sample image. The above sample feature map includes a classification feature map, a height-width feature map, and a rotation angle feature map, and in a possible embodiment, may also include an offset feature map. Classification feature maps, height-width feature maps, rotation angle feature maps, and bias feature maps have the same height H and width W.

According to the height H and width W of the sample feature map, a matrix grid can be created for the sample feature map. Specifically, a matrix grid can be created for the sample feature map through meshgrid. For a grid point (i ₀ , j ₀ ), the index relationship between the detection frame width attribute w ₀ and the detection frame height h ₀ corresponding to the grid point (i ₀ , j ₀ ) can be established in the height-width feature map , and the rotation angle attribute θ ₀ index relationship corresponding to the grid point (i ₀ , j ₀ ) can also be established in the rotation angle feature map, and the index relationship corresponding to the grid point (i ₀ , j 0 ) can also be established in the offset feature map The index relationship between the x-direction offset attribute dx ₀ and the y-direction offset attribute dy ₀ of j ₀ ). For the center point (i ₁ , j ₁ ) of the sample detection frame, it can be indexed to the sample height and width attribute (w ₁ , h ₁ ), the sample rotation angle attribute θ ₁ , the sample offset attribute (dx ₁ , dy ₁ ), Thus, the sample detection frame (x ₁ , y ₁ , w ₁ , h ₁ , θ ₁ ) is obtained.

By adding the output network corresponding to the auxiliary features in the target detection model, the index relationship between the classification features and the auxiliary features is established, so that the trained target detection model can output more accurate auxiliary features.

Optionally, the marked key points are included in the marked frame, and in the step of adjusting the network parameters of the target detection model according to the metric distance between the sample function distribution and the marked function distribution, it is possible to calculate according to the sample key points of the classification feature map The first loss between the sample key point and the labeled key point; the metric distance is converted into a second loss through a preset conversion function; based on the first loss and the second loss, network parameters are adjusted for the target detection model.

In the embodiment of the present invention, the above-mentioned sample key point is the center point in the sample detection frame, and the calculation method of the above-mentioned artwork sample key point is the same as that of the above-mentioned target key point, both of which are calculated by the maximum pooling kernel. The above-mentioned marked key points are marked to obtain, and the calculation of the first loss between the sample key point and the marked key point can be calculated through the first loss function, and the first loss function is shown in the following formula:

loss _hm ＝Guassian_focal_loss(hm _pre d,hm _target )

Among them, loss _hm is the first loss, hm _pred is the prediction result of the center point of the sample detection frame, and hm _target is the real label corresponding to the center point of the label frame. Guassian_focal_loss() is the first loss function.

The above-mentioned second loss is obtained through a preset conversion function, and the above-mentioned preset conversion function may be a nonlinear function. Specifically, the above-mentioned preset rotation function may be shown in the following formula:

Among them, loss _rbbox is the second loss, d is the metric distance between the sample function distribution and the label function distribution, and τ is an adjustable constant.

Based on the first loss and the second loss, the total loss of classification features and auxiliary features can be obtained, and the total loss is shown in the following formula:

Loss＝loss _hm +λloss _rbbox

The above λ is a priori coefficient, which can be adjusted according to prior knowledge during the training process.

In a possible embodiment, the auxiliary feature also includes a bias feature, and the loss between the bias attribute corresponding to the sample detection frame and the label bias attribute corresponding to the label frame can be calculated as the third loss, and the third loss can be obtained by The third loss function is calculated, and the third loss function can be shown in the following formula:

loss _offset ＝Smooth-L1(offset _pred ,offset _target t)

Among them, loss _offset is the third loss, hm _pred is the prediction result of the bias attribute of the sample detection frame, and hm _target is the real label of the offset attribute of the corresponding annotation frame. Smooth-L1() is the third loss function.

In this embodiment, based on the first loss, the second loss and the third loss, the total loss of classification features and auxiliary features can be obtained, and the total loss is shown in the following formula:

Loss＝loss _hm +λ ₁ loss _offset +λ ₂ loss _rbbox

Wherein, the above-mentioned λ1 is the first priori coefficient, and _λ2 is the _second priori coefficient, and both the first priori coefficient and the second priori coefficient can be adjusted according to the priori knowledge during the training process.

In the embodiment of the present invention, by adding the output network corresponding to the auxiliary features, the auxiliary features are trained through the corresponding first loss and the second loss, so that the trained object detection model can output more accurate auxiliary features.

Optionally, in the step of performing auxiliary processing on the classification features based on the auxiliary features to obtain the detection result of the rotating target to be detected, the obtained coordinates (i, j) of the key points of the target can be indexed on the height and width features Target width attribute w and target height attribute h at the corresponding position, index the x-direction offset dx and y-direction offset dy of the corresponding position on the offset feature, and index the target rotation angle at the corresponding position on the rotation angle feature; Detection of rotating targets can be expressed as (x, y, w, h, θ) in the form of x=i+dx and y=j+dy; the four elements of (x, y, w, h) of all detection results The value is scaled to the original image scale (x', y', w', h') of the image to be detected, so the final detection result representation of the rotating target to be detected is (x', y', w', h' , θ).

In the embodiment of the present invention, by extracting the classification features and auxiliary features of the rotating target to be detected, the auxiliary features are used to perform auxiliary processing on the classification features, so as to obtain the detection result of the rotating target to be detected, and there is no need to design the anchor, so there is no need to Non-maximum suppression is required to improve the detection accuracy of rotating target detection, thereby improving the detection performance of rotating target detection.

It should be noted that the target detection method provided by the embodiment of the present invention can be applied to smart phones, computers, servers and other devices capable of target detection.

Optionally, please refer to FIG. 3. FIG. 3 is a schematic structural diagram of a target detection device provided in an embodiment of the present invention. As shown in FIG. 3, the device includes:

The first acquiring module 301 is configured to acquire an image to be detected, the image to be detected includes a rotating target to be detected;

The extraction module 302 is used to perform feature extraction on the image to be detected by using the trained target detection model to obtain classification features and auxiliary features of the rotating target to be detected;

The processing module 303 is configured to perform auxiliary processing on the classification feature based on the auxiliary feature to obtain a detection result of the rotating object to be detected.

Optionally, the trained target detection model includes a feature extraction network, a feature fusion network, a classification feature output network, and an auxiliary feature output network, and the extraction module 302 includes:

The first extraction submodule is used to perform feature extraction on the image to be detected through the feature extraction network to obtain multi-scale features of the image to be detected;

The fusion sub-module is used to perform feature fusion on the multi-scale features through the feature fusion network to obtain the fusion features of the image to be detected;

The first prediction sub-module is used to predict the fusion feature through the classification feature output network to obtain the classification feature of the rotation target to be detected, the classification feature includes a feature channel, and the rotation target of different categories corresponds to in different feature channels;

The second prediction sub-module is configured to predict the fused feature through the auxiliary feature output network to obtain the auxiliary feature of the rotating target to be detected.

Optionally, the auxiliary features include height-width features and rotation angle features, and the processing module 303 includes:

The second extraction submodule is used to extract the key points of the classification features to obtain the target key points of the rotating target to be detected;

An indexing submodule, configured to index the corresponding target height-width attribute in the height-width feature based on the target key point, and index the corresponding target rotation angle attribute in the rotation angle feature;

The first processing submodule is configured to obtain a detection result of the rotating object to be detected based on the key points of the object, the object height and width attributes, and the object rotation angle attribute.

Optionally, the device also includes:

An acquisition module, configured to acquire a training data set, the training data set includes a sample image and a label frame, the sample image includes a sample rotation target, and the label frame is a label frame of the sample rotation target;

The training module is used to obtain the target detection model, and train the target detection model through the training data set to obtain the trained target detection model. The target detection model includes a feature extraction network, a feature fusion network, and a classification feature output Network, height and width feature output network, and rotation angle feature output network.

Optionally, the training module includes:

The second processing submodule is configured to input the sample image into the target detection model to obtain a sample detection frame corresponding to the sample rotation target;

The encoding sub-module is used to encode the sample detection frame and the label frame respectively through a preset encoding function, and respectively obtain the sample function distribution corresponding to the sample detection frame and the label function distribution corresponding to the label frame ;

An adjustment submodule, configured to adjust the network parameters of the target detection model according to the measured distance between the sample function distribution and the label function distribution;

The iteration sub-module is used to iterate the network parameter adjustment process of the target detection model until the target detection model converges or reaches a preset number of iterations to obtain a trained target detection model.

Optionally, the second processing submodule includes:

The first processing unit is configured to process the sample image through the target detection model to obtain a sample feature map corresponding to the sample image, and construct the sample feature map according to the height and width of the sample feature map A matrix grid, the sample feature image includes a classification feature map, a height and width feature map, and a rotation angle feature map corresponding to the sample image;

An index building unit, configured to create an index of a height-width attribute for each grid point in the height-width feature map, and create an index of a rotation angle attribute for each grid point in the rotation angle feature map;

The second processing unit is configured to obtain a sample detection frame corresponding to the sample rotation target according to each grid point in the sample feature map and its corresponding index attribute.

Optionally, the labeling frame includes labeling key points, and the adjustment submodule includes:

A calculation unit, configured to calculate a first loss between the sample key points and the labeled key points according to the sample key points of the classification feature map;

a conversion unit, configured to convert the metric distance into a second loss through a preset conversion function;

An adjustment unit, configured to adjust network parameters of the target detection model based on the first loss and the second loss.

It should be noted that the object detection device provided by the embodiment of the present invention can be applied to devices such as smart phones, computers, and servers capable of object detection.

The target detection device provided by the embodiment of the present invention can realize each process realized by the target detection method in the above method embodiment, and can achieve the same beneficial effect. To avoid repetition, details are not repeated here.

Referring to FIG. 4, FIG. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As shown in FIG. 4, it includes: a memory 402, a processor 401, and an 401. A computer program for an object detection method run on, wherein:

The processor 401 is used to call the computer program stored in the memory 402, and perform the following steps:

Acquiring an image to be detected, the image to be detected includes a rotating target to be detected;

performing feature extraction on the image to be detected through the trained target detection model to obtain classification features and auxiliary features of the rotating target to be detected;

An auxiliary process is performed on the classification feature based on the auxiliary feature to obtain a detection result of the rotating target to be detected.

Optionally, the trained target detection model includes a feature extraction network, a feature fusion network, a classification feature output network, and an auxiliary feature output network, and the trained target detection model executed by the processor 401 performs a The image is subjected to feature extraction to obtain classification features and auxiliary features of the rotating target to be detected, including:

performing feature extraction on the image to be detected through the feature extraction network to obtain multi-scale features of the image to be detected;

performing feature fusion on the multi-scale features through the feature fusion network to obtain the fusion features of the image to be detected;

The fusion feature is predicted by the classification feature output network to obtain the classification feature of the rotating target to be detected, the classification feature includes a feature channel, and different types of rotating targets to be detected correspond to different feature channels;

The fusion feature is predicted by the auxiliary feature output network to obtain the auxiliary feature of the rotating target to be detected.

Optionally, the auxiliary features include a height-width feature and a rotation angle feature, and the processor 401 performs auxiliary processing on the classification features based on the auxiliary features to obtain a detection result of the rotating target to be detected, including :

Carrying out key point extraction on the classification features to obtain target key points of the rotating target to be detected;

Based on the target key point, indexing a corresponding target height and width attribute in the height and width feature, and indexing a corresponding target rotation angle attribute in the rotation angle feature;

A detection result of the rotating object to be detected is obtained based on the key point of the object, the attribute of the height and width of the object, and the attribute of the rotation angle of the object.

Optionally, before performing feature extraction on the image to be detected by using the trained target detection model to obtain classification features and auxiliary features of the rotating target to be detected, the method executed by the processor 401 further includes:

Obtain a training data set, the training data set includes a sample image and a label frame, the sample image includes a sample rotation target, and the label frame is a label frame of the sample rotation target;

Obtain the target detection model, and train the target detection model through the training data set to obtain the trained target detection model. The target detection model includes a feature extraction network, a feature fusion network, a classification feature output network, and a height and width feature. Output network and rotation angle feature output network.

Optionally, the acquisition of the target detection model executed by the processor 401, and the training of the target detection model of the training data set to obtain the trained target detection model include:

Inputting the sample image into the target detection model to obtain a sample detection frame corresponding to the sample rotation target;

Encoding the sample detection frame and the label frame respectively by a preset encoding function, respectively obtaining a sample function distribution corresponding to the sample detection frame and a label function distribution corresponding to the label frame;

adjusting the network parameters of the target detection model according to the metric distance between the sample function distribution and the label function distribution;

The network parameter adjustment process of the target detection model is iterated until the target detection model converges or reaches a preset number of iterations to obtain a trained target detection model.

Optionally, the inputting the sample image into the target detection model performed by the processor 401 to obtain the sample detection frame corresponding to the sample rotating target includes:

Process the sample image through the target detection model to obtain a sample feature map corresponding to the sample image, and construct a matrix grid for the sample feature map according to the height and width of the sample feature map, the sample The feature image includes a classification feature map, a height-width feature map, and a rotation angle feature map corresponding to the sample image;

Establishing an index of a height and width attribute for each grid point in the height-width feature map, and establishing an index of a rotation angle attribute for each grid point in the rotation angle feature map;

According to each grid point in the sample feature map and its corresponding index attribute, a sample detection frame corresponding to the sample rotation target is obtained.

Optionally, the marked key points are included in the marked frame, and the processor 401 executes performing network parameter adjustment on the target detection model according to the measured distance between the sample function distribution and the marked function distribution, include:

Calculate the first loss between the sample key point and the label key point according to the sample key point of the classification feature map;

converting the metric distance into a second loss through a preset conversion function;

Based on the first loss and the second loss, network parameters are adjusted for the target detection model.

The electronic device provided by the embodiment of the present invention can realize each process realized by the target detection method in the above method embodiment, and can achieve the same beneficial effect. To avoid repetition, details are not repeated here.

The embodiment of the present invention also provides a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the target detection method or the application-side target detection method provided by the embodiment of the present invention is implemented. Each process can achieve the same technical effect, so in order to avoid repetition, it will not be repeated here.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the programs can be stored in a computer-readable storage medium. During execution, it may include the processes of the embodiments of the above-mentioned methods. Wherein, the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM) or a random access memory (Random Access Memory, RAM for short).

The above disclosures are only preferred embodiments of the present invention, and certainly cannot limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention still fall within the scope of the present invention.

Claims (10)

1. A target detection method, characterized in that, the target detection method is used for the detection of a rotating target, comprising the following steps: Acquiring an image to be detected, the image to be detected includes a rotating target to be detected; performing feature extraction on the image to be detected through the trained target detection model to obtain classification features and auxiliary features of the rotating target to be detected; An auxiliary process is performed on the classification feature based on the auxiliary feature to obtain a detection result of the rotating target to be detected. 2. The target detection method according to claim 1, wherein the trained target detection model comprises a feature extraction network, a feature fusion network, a classification feature output network, and an auxiliary feature output network, and the trained The target detection model performs feature extraction on the image to be detected to obtain classification features and auxiliary features of the rotating target to be detected, including: performing feature extraction on the image to be detected through the feature extraction network to obtain multi-scale features of the image to be detected; performing feature fusion on the multi-scale features through the feature fusion network to obtain the fusion features of the image to be detected; The fusion feature is predicted by the classification feature output network to obtain the classification feature of the rotating target to be detected, the classification feature includes a feature channel, and different types of rotating targets to be detected correspond to different feature channels; The fusion feature is predicted by the auxiliary feature output network to obtain the auxiliary feature of the rotating target to be detected. 3. The target detection method according to claim 2, wherein the auxiliary features include a height-width feature and a rotation angle feature, and the auxiliary processing is performed on the classification features based on the auxiliary features to obtain the Detection results for detecting rotating targets, including: Carrying out key point extraction on the classification features to obtain target key points of the rotating target to be detected; Based on the target key point, indexing a corresponding target height and width attribute in the height and width feature, and indexing a corresponding target rotation angle attribute in the rotation angle feature; A detection result of the rotating object to be detected is obtained based on the key point of the object, the attribute of the height and width of the object, and the attribute of the rotation angle of the object. 4. target detection method as claimed in claim 3, is characterized in that, carry out feature extraction to described to-be-detected image in described target detection model by training, obtain the classification feature and auxiliary feature of described to-be-detected rotating target Previously, the method further included: Obtain a training data set, the training data set includes a sample image and a label frame, the sample image includes a sample rotation target, and the label frame is a label frame of the sample rotation target; Obtain the target detection model, and train the target detection model through the training data set to obtain the trained target detection model. The target detection model includes a feature extraction network, a feature fusion network, a classification feature output network, and a height and width feature. Output network and rotation angle feature output network. 5. The target detection method according to claim 4, wherein the acquisition target detection model is trained by the target detection model of the training data set to obtain a trained target detection model, comprising: Inputting the sample image into the target detection model to obtain a sample detection frame corresponding to the sample rotation target; Encoding the sample detection frame and the label frame respectively by a preset encoding function, respectively obtaining a sample function distribution corresponding to the sample detection frame and a label function distribution corresponding to the label frame; adjusting the network parameters of the target detection model according to the metric distance between the sample function distribution and the label function distribution; The network parameter adjustment process of the target detection model is iterated until the target detection model converges or reaches a preset number of iterations to obtain a trained target detection model. 6. The target detection method according to claim 5, wherein the inputting the sample image into the target detection model to obtain the sample detection frame corresponding to the sample rotation target comprises: Process the sample image through the target detection model to obtain a sample feature map corresponding to the sample image, and construct a matrix grid for the sample feature map according to the height and width of the sample feature map, the sample The feature image includes a classification feature map, a height-width feature map, and a rotation angle feature map corresponding to the sample image; Establishing an index of a height and width attribute for each grid point in the height-width feature map, and establishing an index of a rotation angle attribute for each grid point in the rotation angle feature map; According to each grid point in the sample feature map and its corresponding index attribute, a sample detection frame corresponding to the sample rotation target is obtained. 7. The target detection method according to claim 6, wherein the labeling frame includes key points for labeling, and according to the metric distance between the distribution of the sample function and the distribution of the labeling function, the The target detection model adjusts network parameters, including: Calculate the first loss between the sample key point and the label key point according to the sample key point of the classification feature map; converting the metric distance into a second loss through a preset conversion function; Based on the first loss and the second loss, network parameters are adjusted for the target detection model. 8. A target detection device, characterized in that the device comprises: A first acquiring module, configured to acquire an image to be detected, where the image to be detected includes a rotating target to be detected; The extraction module is used to perform feature extraction on the image to be detected through the trained target detection model, and obtain classification features and auxiliary features of the rotating target to be detected; A processing module, configured to perform auxiliary processing on the classification feature based on the auxiliary feature to obtain a detection result of the rotating target to be detected. 9. An electronic device, characterized in that it comprises: a memory, a processor, and a computer program stored on the memory and operable on the processor, when the processor executes the computer program, it realizes the Requires the steps in the target detection method described in any one of 1 to 7. 10. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is realized. Steps in an object detection method.

Images