CN114663760A - Model training method, target detection method, storage medium and computing device - Google Patents

Abstract

The invention relates to the technical field of machine learning, and provides a model training method, a target detection method, a storage medium and computing equipment, wherein the model training method comprises the following steps: acquiring a plurality of target images, wherein the plurality of target images comprise source domain images with labels and target domain images without labels; for each of the plurality of target images: substituting the target image into a feature extractor to perform feature extraction processing to obtain a first feature map; inputting the first feature map into a classifier to obtain a classification result of the target image; inputting the first feature map into a first discriminator to obtain a first probability value of the target image belonging to the source domain; training a feature extractor, a classifier and a first discriminator based on target detection classification results, first probability values and labels of source domain images corresponding to the target images respectively so that the feature extractor learns feature distribution which can be shared by the source domain images and the target domain images, and the classifier can classify the target domain images more accurately.

Description

Model training method, target detection method, storage medium and computing device

technical field

The present invention relates to the technical field of machine learning, and in particular, to a method for model training, a method for target detection, a storage medium and an electronic device.

Background technique

Large-scale tree surveys are a key research question. Today, abundant remote sensing images and the rapid development of deep learning algorithms have brought new opportunities for large-scale detection of forest trees such as oil palms. However, large-scale tree counting and detection may face remote sensing images with different acquisition conditions, such as different sensors, seasons, and environments, resulting in different distributions between images. For example, as shown in Figure 1, Image A and Image B are two different satellite images. Here, we assume that image A is an image with enough labels, here the labels have 4 categories, namely the area between oil palm trees, oil palm trees, other vegetation or bare ground, impervious layer or cloud, and image B is Images without labels; significant differences can be seen between Image A and Image B in the 4-category histograms (used to characterize the distribution of image pixel values) due to differences in sensor, acquisition date, and region location; even if Feature extractor and classifier have excellent detection accuracy and classification accuracy under images with labels, when it is directly applied to images without any labels, such as image B, the performance of feature extractor and classifier may drop sharply.

Therefore, how to improve the performance of feature extractors and classifiers on unlabeled images has become an urgent problem to be solved.

SUMMARY OF THE INVENTION

The present invention provides a model training method, a target detection method, a computer-readable storage medium and an electronic device. By training a feature extractor to make it learn the feature distribution shared by the source domain image and the target domain image, the classifier can The target domain image is more accurately classified, and the label of the source domain image is transferred to the target domain image; in addition, since the forest area corresponding to the source domain image and the target image are different, cross-regional tree detection can be realized.

In a first aspect, the present invention provides a method for model training, including:

Based on the remote sensing images obtained from different forest areas, a plurality of target images are obtained, and the plurality of target images include a labeled source domain image and an unlabeled target domain image, the source domain image and the target domain image The corresponding forest areas are different;

For each image of the plurality of target images:

Substitute the target image into a feature extractor to perform feature extraction processing to obtain a first feature map;

Inputting the first feature map into a classifier to obtain a classification result for the target image;

Inputting the first feature map into the first discriminator to obtain the first probability value that the target image belongs to the source domain;

obtaining a first loss, which indicates the uncertainty of the classification result, based on the target detection classification results and the first probability value corresponding to the plurality of target images;

Obtaining a second loss indicating the classification error of the classifier based on the label and the corresponding classification result of each image of the source domain;

obtaining a third loss based on the respective first probability values of the plurality of target images, which indicates the error of the source domain classification based on the first feature map;

The feature extractor, classifier and first discriminator are trained based on the first loss, the second loss and the third loss.

In a second aspect, the present invention provides a method for target detection, comprising:

Obtain the target image to be detected;

Segmenting the target image to determine a plurality of sub-images;

The feature extractor and the classifier are used to detect and classify the plurality of sub-images respectively, and the detection and classification results are determined; the feature extractor and the classifier are obtained by training any method in the first aspect, and the detection and classification results include multiple a target frame and the respective categories of the plurality of target frames;

The target frames belonging to the same category are merged to determine the target detection result of the target image.

In a third aspect, the present invention provides a device for model training, comprising:

The image acquisition module is used to obtain a plurality of target images based on the remote sensing images obtained by shooting different forest areas respectively, and the plurality of target images include a labeled source domain image and an unlabeled target domain image, and the source domain image is different from the forest area corresponding to the target domain image;

The classification module is used for each image of the plurality of target images: substituting the target image into a feature extractor to perform feature extraction processing to obtain a first feature map; inputting the first feature map into the classifier to obtain a pair of The classification result of the target image; inputting the first feature map into the first discriminator to obtain the first probability value that the target image belongs to the source domain;

a first loss calculation module, configured to obtain a first loss based on the target detection classification results corresponding to the plurality of target images and the first probability value, which indicates the uncertainty of the classification result;

A second loss calculation module, configured to obtain a second loss based on the label and the corresponding classification result of each image in the source domain, which indicates the classification error of the classifier;

a third loss calculation module, configured to obtain a third loss based on the respective first probability values of the plurality of target images, which indicates the error of the source domain classification based on the first feature map;

A training module for training the feature extractor, the classifier and the first discriminator based on the first loss, the second loss and the third loss.

In a fourth aspect, the present invention provides a device for target detection, comprising:

an image acquisition module for acquiring the target image to be detected;

a segmentation module, for segmenting the target image and determining a plurality of sub-images;

A classification module, configured to detect and classify the plurality of sub-images respectively through a feature extractor and a classifier, and determine a detection and classification result; the feature extractor and the classifier are trained by any one of the methods described in the first aspect above. Obtain, the detection and classification result includes multiple target frames and respective categories of the multiple target frames;

The merging module is used for merging each target frame belonging to the same category to determine the target detection result of the target image.

In a fifth aspect, the present invention provides a computer-readable storage medium, comprising execution instructions, when a processor of an electronic device executes the execution instructions, the processor executes any one of the first aspect or the second aspect. method described.

In a sixth aspect, the present invention provides an electronic device, comprising a processor and a memory storing execution instructions. When the processor executes the execution instructions stored in the memory, the processor executes the first aspect. or the method of any one of the second aspects.

The present invention provides a method, device, computer-readable storage medium and electronic equipment for model training. The method obtains multiple target images based on remote sensing images obtained from different forest areas respectively, and the multiple target images include labeled The source domain image and the unlabeled target domain image, the forest area corresponding to the source domain image and the target domain image are different; for each image of multiple target images: Then, the target image is substituted into the feature extractor for feature extraction processing, and the first a feature map; input the first feature map into the classifier to obtain the classification result of the target image; input the first feature map into the first discriminator to obtain the first probability value that the target image belongs to the source domain; then, based on multiple The target detection classification result and the first probability value corresponding to each target image, and the first loss is obtained, which indicates the uncertainty of the classification result; then, based on the label of each image in the source domain and the corresponding classification result, the first loss is obtained. The second loss indicates the classification error of the classifier; then, based on the first probability values corresponding to each of the multiple target images, a third loss is obtained, which indicates the error of the source domain classification based on the first feature map; The first loss, the second loss and the third loss, train the feature extractor, the classifier and the first discriminator. To sum up, the technical solution of the present invention enables the classifier to more accurately classify the target domain images by training the feature extractor to learn the feature distribution that can be shared by the source domain image and the target domain image, and realizes the source domain image. The label of the image is transferred to the target domain image; in addition, since the forest tree regions corresponding to the source domain image and the target image are different, cross-regional tree detection can be realized.

Further effects of the above-mentioned non-conventional preferred mode will be described below in conjunction with specific embodiments.

Description of drawings

In order to illustrate the embodiments of the present invention or the existing technical solutions more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the existing technology. Obviously, the accompanying drawings in the following description are only the For some embodiments described in the invention, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

1 is a schematic diagram of a histogram of images with different acquisition conditions provided by an embodiment of the present invention;

2 is a schematic structural diagram of an identification model provided by an embodiment of the present invention;

3 is a schematic flowchart of a method for model training provided by an embodiment of the present invention;

FIG. 4 is a schematic flowchart 1 of a method for target detection provided by an embodiment of the present invention;

5 is a second schematic flowchart of a method for target detection provided by an embodiment of the present invention;

6 is a schematic structural diagram of an apparatus for model training provided by an embodiment of the present invention;

7 is a schematic structural diagram of an apparatus for target detection provided by an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to specific embodiments and corresponding drawings. Obviously, the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

First, the source domain and target domain are introduced .

Transfer learning specifically applies knowledge or patterns learned on the source domain to a different but related target domain. Among them, the source domain (source domain) represents a field different from the test sample, but has rich supervision information; the target domain (target domain) represents the field where the test sample is located, with no labels or only a few labels. The data distribution of the source and target domains is different, but the tasks are the same. Here, the task: is what to do, such as forest tree identification and classification.

In the embodiment of the present invention, the model training is performed through the source domain and the target domain, so as to realize the task of detection and classification of forest trees, for example, detection and classification of oil palm trees. It should be noted that the forest trees here are trees of a certain type, and in practical applications, various types of forest trees can also be considered; in the embodiment of the present invention, the forest trees of a certain type (for the convenience of description and distinction, referred to as the target type) are example is described. The source domain includes multiple images with labels, and the target domain includes multiple images without labels. Here, due to the high similarity between forest trees and areas between forest trees, in order to improve the accuracy of forest tree identification, the labels in this embodiment of the present invention include areas between trees and forest tree categories. Other labels can be designed in combination with actual conditions, such as other vegetation or Bare ground, impervious layers or clouds, etc.

In practical applications, the images of the source domain and the target domain are usually remote sensing images, and the remote sensing images usually contain large forests. Therefore, the images of the source domain and the target domain are images after segmentation of the remote sensing images. In a specific embodiment, when constructing the source domain, multiple labels are preset; then, remote sensing images obtained from one area or multiple areas (including forest trees of the target category) are selected, and these remote sensing images are processed Manual labeling, for example, label the area and the label to which the area belongs, then segment the labeled remote sensing image, and the segmented image (carrying the label, for example, if the image is part of the labeled area, the labeled area can be The label of the image is used as the label of the image; if the image has two labels, that is, the boundary between the different areas marked, the label of the area with a larger area is used as the label of the image. If the area is similar, it is difficult to determine the image to which the image belongs. label, the image can be discarded) to form the source domain; in practical applications, the label can also be segmented first. Afterwards, segment the remote sensing images taken separately from one or more areas different from the above area (including forest trees of the target category), and the segmented images form the target domain. Illustratively, the size of the images in the source and target domains may be 17×17 pixels.

The construction of the above target domain and source domain is only an example, and does not constitute a specific limitation, as long as the acquisition conditions of the source domain and the target domain are different, such as different sensors, seasons, regions, and preferred regions are different.

It should be noted that the data distribution of the source domain and the target domain in this embodiment of the present invention are different. If a large number of labeled source domains are used for training, the trained classifier will not perform well on the target domain. Based on this, the embodiment of the present invention proposes an adversarial transfer learning method for model training.

Next, we will introduce the adversarial transfer learning .

FIG. 2 shows a structure of a recognition model provided by an embodiment of the present invention. In the embodiment of the present invention, as shown in FIG. 2 , the recognition model includes a feature extractor, a classifier, and a first discriminator.

Adversarial transfer learning is a form of unsupervised deep transfer learning, which uses three parts: feature extractor, classifier and first discriminator to form an adversarial transfer learning network model. The purpose of adversarial transfer learning is how to extract features from the source domain image and the target domain image, so that the first discriminator cannot distinguish whether the extracted features are from the source domain or the target domain.

Among them, the feature extractor is used to map the image to a specific feature space, so that the classifier can distinguish the labels from the source domain image, while the first discriminator cannot distinguish which source domain or target domain the image comes from.

The feature extractor includes N convolution blocks, for example, N=5, which will be described below by taking N=5 as an example. In one example, a convolutional block includes a convolutional layer, a Batch Normalization (BN) layer, an Instance Normalization (IN) layer, and an activation layer. The convolutional layer realizes image processing through the Convolutional Neural Network (CNN); the activation layer is processed through the activation function, for example, the activation function can be ReLU (Rectified Linear Units); it should be noted that although the BN layer can be effective Accelerates model convergence, but makes the CNN insensitive to image changes. Therefore, the IN layer is added to eliminate the differences between different individuals, thereby enhancing the generalization of the network. In addition, the BN layer and the IN layer are in the prior art, and details are not described in this embodiment of the present invention.

示例地，当

大于等于0.5时，代表特征图属于源域，当

小于0.5时，代表特征图属于目标域。进一步地，特征提取器的最后一个卷积块的卷积层输出的特征图通过如下公式(1)示出的公式处理得到新的特征图h_i：Further, the feature extractor further includes a pooling layer located between the jth convolution block and the j+1th convolution block, where j is a positive integer greater than 1. Illustratively, j=2. For example, the pooling method adopted by the pooling layer may be maximum pooling or average pooling. Correspondingly, on the basis that the feature extractor includes a pooling layer, the feature extractor may further include a second discriminator. Among them, the second discriminator is used to output the probability that the image belongs to the source domain

For example, when

When it is greater than or equal to 0.5, it means that the feature map belongs to the source domain.

When it is less than 0.5, the representative feature map belongs to the target domain. Further, the feature map output by the convolution layer of the last convolution block of the feature extractor is processed by the formula shown in the following formula (1) to obtain a new feature map h _i :

表示特征层次注意力值。图像的迁移能力更强的特征能够被赋予更大的特征层次注意力值。Among them, f _i is the feature map output by the last convolutional layer; _hi is the new feature map containing the transfer ability information;

Represents the feature-level attention value. The more transferable features of the image can be assigned a larger feature-level attention value.

It should be noted that the purpose of feature-level attention is to find the features of images that are more mobile between the source and target domains, thereby mapping the source and target domains from the original feature space to the new feature space (source domain and target domain have the same data distribution), so that the second discriminator cannot distinguish whether the image is from the target domain or the source domain. In order to measure this mobility, the embodiment of the present invention describes the uncertainty by the method of information entropy. Information entropy, also called Shannon entropy, is calculated by the following formula (2).

E(p)=-∑ _d P _d ·log(P _d ) (2)

Among them, d=0, P _d represents the probability that the image belongs to the target domain; d=1, P _d represents the probability that the image belongs to the source domain; the embodiment of the present application only considers the case of d=1. According to information theory, the greater the entropy, the greater the amount of information, and the greater the mobility of the image. Correspondingly, the feature extractor can calculate the feature-level attention value (V _i ^F ) by the following formula (3):

Among them, the result of E(·) output is the information entropy.

In this way, the feature extractor can effectively measure the transferability of feature maps, and know which feature maps are more suitable for classification, and which feature maps have a negative effect on classification. Therefore, a connection is established between the feature map output by the convolutional layer of the last convolutional block of the feature extractor and the feature-level attention value, and a new feature map is generated. The newly generated feature map contains the transfer ability information.

Among them, the classifier classifies the source domain images and separates the correct labels as much as possible.

Among them, the first discriminator classifies the images in the feature space, and tries to separate the images from the source domain or the target domain as much as possible.

In addition, the feature extractor and the classifier make up the classification detection part, and the feature extractor and the first discriminator make up the domain discrimination part. The optimization goal of the feature extractor and the first discriminator is opposite. The first discriminator tries to determine whether the image comes from the source domain or the target domain. The feature extractor tries to make the first discriminator unable to determine the source of the image. The optimization goal of the author, through confrontation, finally makes the feature distribution of the source domain image and the target domain image output through the feature extractor similar, that is, map the source domain and target domain with different distributions to the same feature space, and find a certain kind of The metric criterion is to make the "distance" in this space as close as possible; thus, the classifier can accurately classify the source domain image and the target domain image at the same time.

Next, the loss function of the recognition model provided by the embodiment of the present invention is introduced.

Among them, the loss function is shown in the following formula (4).

表示带有标签的源域图像的分类损失。Among them, _LS represents the shallow loss domain loss; _LD represents the deep feature domain loss; _LE represents the entropy loss; μ, α and β represent the hyperparameters that balance the shallow feature domain loss, the deep feature domain loss and the entropy loss.

Represents the classification loss for source domain images with labels.

通过如下公式(5)计算：in,

Calculated by the following formula (5):

Among them, _Ly (·) represents the cross-entropy loss function; G _y (·) represents the classifier, and the output is the predicted probability that the i-th image in the source domain belongs to the _yi category.

Among them, the shallow feature domain loss is used to make the feature extractor learn features that are more transferable between the source domain and the target domain, so that the second discriminator in the feature extractor cannot distinguish whether the image is from the target domain or the source domain. . The shallow feature domain loss is calculated by the following formula (6):

等于1，对于目标域的图像

等于0；F′^S表示第j个卷积块输出的源域的第i个图像的特征图；F′^T来表示表示第j个卷积块输出的目标域的第i个图像的特征图。值得注意的是，L_S表示基于浅层特征进行源域分类的误差。where G _d ( ) represents the second discriminator, L _d ( ) represents the binary cross-entropy loss of G _d ( ), and for the images in the source domain,

equal to 1, for images in the target domain

Equal to 0; F′ ^S represents the feature map of the i-th image in the source domain output by the j-th convolution block; F′ ^T represents the feature map of the i-th image in the target domain output by the j-th convolution block . It is worth noting that _LS represents the error of source domain classification based on shallow features.

It should be noted that the feature map output by the feature extractor is generated through the pooling layer, which leads to the loss of shallow feature information. In addition, considering that the transferability of each image is different, for images that are not similar in the feature space, it will have a negative effect on the transfer of features from the source domain to the target domain, thereby affecting the classification performance of the classifier. Considering that the feature extractor needs to obtain features with good transferability, so that the second discriminator cannot distinguish whether the image is from the target domain or the source domain. Complete the classification task. As shown in Figure 2, the embodiment of the present invention designs a shallow feature domain loss before the pooling layer.

Among them, the deep feature domain loss is used to make the feature extractor learn the deep features of the image with stronger migration between the source domain image and the target domain image, and it is calculated by the following formula (7):

等于1，对于目标域图像

等于0。更进一步解释，g_d(·)的输出即是图像属于源域的概率

这样，我们的域损失就包括了浅层特征域损失L_S和深层特征的域损失L_D。where g _d (·) represents the first discriminator, and L _d (·) represents the binary cross-entropy loss of g _d (·). For source domain images,

equal to 1, for target domain images

equal to 0. Further explanation, the output of g _d ( ) is the probability that the image belongs to the source domain

In this way, our domain loss includes the domain loss LS for shallow features and the domain loss _LD for _deep features.

It should be noted that, considering that the transferability of each image is different, for images that are not similar in the feature space, it will have a negative effect on the transfer of knowledge from the source domain to the target domain, thereby affecting the classification performance of the classifier. . As shown in Figure 2, the embodiment of the present invention designs a deep feature domain loss before classification.

Among them, _LE is calculated by the following formula (8):

Among them, LE represents the entropy loss, which is used to describe the uncertainty of the classification result of the classifier; V _i ^F represents the entropy level attention value; C represents the number of categories, such as 4 categories; p _{i, c} represent the corresponding image of the _i -th image The probability that the predicted class is c.

It should be noted that entropy-level attention is similar to feature-level attention, and is used to describe the degree of attention to the loss of information in images. Images with less transferability may prevent the feature extractor from learning more transferable features and reduce the classification accuracy of the classifier. Smaller images have smaller entropy-level attention values, making the feature extractor pay more attention to more transferable features. Correspondingly, the entropy level attention value (V _i ^E ) can be calculated by the following formula (9):

Among them, the result of E(·) output is the information entropy.

It should be understood that, inspired by the entropy function of information theory, the entropy loss is used to reduce the uncertainty of the output class probability. In the embodiment of the present invention, if the target image is used, the entropy loss has two advantages. On the one hand, since the forest trees in the image and the area between the trees have a large similarity, the entropy loss can improve the confidence of the easily confused sample prediction; on the other hand , for images with poor similarity, if they are forced to increase their confidence, it will have a bad impact on the classifier; therefore, the weight of the entropy-level attention value is assigned to the entropy loss, so that the entropy-level attention-based Minimum entropy regularization makes our classifier's predictions more credible. Entropy loss regularization is used as a penalty term for the loss function, and some restrictions are imposed on some parameters in the loss function.

As shown in FIG. 3 , it is a method for model training provided by an embodiment of the present invention. The method provided by the embodiment of the present invention can be applied to an electronic device, and specifically can be applied to a server or a general computer. In this embodiment, the method specifically includes the following steps:

Step 301, obtaining multiple target images based on the remote sensing images obtained by shooting different forest areas respectively, and the multiple target images include a labeled source domain image and an unlabeled target domain image, and the target domain image and the forest tree corresponding to the source domain image. Regions are different.

According to a feasible implementation manner, the first remote sensing image obtained by photographing the first forest area by the first sensor is segmented to obtain multiple images; then, tags are added to these images to obtain multiple source domain images. Of course, in practical applications, the first sensor usually captures a plurality of first remote sensing images, and the processing process of each first remote sensing image is the same.

According to a feasible implementation manner, the second remote sensing image obtained by photographing the second forest area by the second sensor is segmented to obtain multiple images; there is no need to add labels to these images to obtain multiple target domain images. Of course, in practical applications, the second sensor usually captures multiple second remote sensing images, and the processing process of each second remote sensing image is the same.

Illustratively, the first sensor and the second sensor are different. For example, the first sensor and the second sensor are sensors on different satellites.

For example, the first sensor and the second sensor may also be the same, but the acquisition moments are different.

For example, the first forest area and the second forest area are different, so as to realize cross-area identification.

For example, the size of the above target image may be 17×17 pixels.

Illustratively, the first forest area and the second forest area include forest trees belonging to the target category, such as oil palm trees. Of course, the oil palm tree is only an example, and does not constitute a specific limitation. The source domain image and the target domain image may be determined according to actual requirements, which are not specifically limited in this embodiment of the present invention. The source domain image is the image in the above-mentioned source domain, and the target domain image is the image in the above-mentioned target domain.

For the content related to the construction of the source domain and the target domain, please refer to the above, and will not be repeated here.

Step 302, for each image of a plurality of target images: substituting the target image into a feature extractor to perform feature extraction processing to obtain a first feature map; inputting the first feature map into the classifier to obtain a classification result for the target image; A feature map is input into the first discriminator to obtain a first probability value that the target image belongs to the source domain.

The detailed description of the feature extractor and the classifier is referred to above, and will not be repeated here.

Step 303 , obtaining a first loss, which indicates the uncertainty of the classification result, based on the target detection classification result and the first probability value corresponding to each of the plurality of target images.

According to a feasible implementation manner, for each of the multiple target images, the Shannon entropy is calculated based on the corresponding first probability value, and the attention value is calculated based on the Shannon entropy; The attention value and the probability value of each category in the classification result determine the first loss.

The first loss corresponds to the entropy loss _LE described above. It should be noted that the higher the probability of a certain category in the classification result relative to other categories, the lower the uncertainty of the classification result; if the difference between the probabilities of multiple categories in the classification result is small, the higher the uncertainty of the classification result. . Therefore, the purpose of designing the first loss is to reduce the uncertainty of the classification result output by the classifier.

Step 304, based on the labels of each source domain image and the corresponding classification results, obtain a second loss, which indicates the classification error of the classifier.

需要说明的是，分类损失越小，说明分类器预测出的源域图像的标签的概率越高，分类结果越准确。The second loss corresponds to the above classification loss

It should be noted that the smaller the classification loss, the higher the probability of the label of the source domain image predicted by the classifier, and the more accurate the classification result.

Step 305 , obtaining a third loss based on the first probability values corresponding to each of the multiple target images, which indicates the error of the source domain classification based on the first feature map.

The third loss corresponds to the above-mentioned deep feature domain loss _LD . It should be noted that, the smaller the loss _LD of the deep feature domain, the smaller the error of the source domain classification performed by the feature extractor for the first feature maps respectively output by the source domain image and the target domain image.

In step 306, the feature extractor, the classifier and the first discriminator are trained based on the first loss, the second loss and the third loss.

According to a feasible implementation manner, the feature extractor includes a first extraction layer, a second extraction layer, and a second discriminator; the first extraction layer is used to extract shallow features of the target image to obtain a second feature map; The second discriminator is used to determine the second probability value of the target image belonging to the source domain image based on the second feature map; the second extraction layer is used to extract deep features based on the second feature map and the second probability value to obtain the first feature map. .

In one example, the feature extractor includes a plurality of convolutional blocks and pooling layers, the first advance layer includes a plurality of convolutional blocks before the pooling layer, and the second extraction layer includes the pooling layer and subsequent layers Multiple convolutional blocks; a single convolutional block includes convolutional layers, batch normalization layers, instance normalization layers, and activation layers. For details, refer to the above, and will not be repeated here.

Further, the method further includes: obtaining a fourth loss based on the second probability values corresponding to each of the plurality of target images, which indicates the error of the source domain classification based on the second feature map.

After that, based on the first loss, the second loss, the third loss and the fourth loss, the feature extractor, the classifier and the first discriminator are trained, so as to find that the migration between the source domain image and the target domain image is strong , construct a feature space with strong mobility, so that the feature extractor maps the source domain image and the target domain image into the feature space, and then realizes the classification of the target domain image through the classifier.

It can be known from the above technical solutions that the beneficial effects of the present embodiment are:

By training the feature extractor to learn the feature distribution shared by the source domain image and the target domain image, the classifier can classify the target domain image more accurately, and transfer the label of the source domain image to the target domain image; , since the forest area corresponding to the source image and the target image is different, cross-area forest detection can be realized.

As shown in FIG. 4 , it is a target detection method provided by an embodiment of the present invention. The method provided by the embodiment of the present invention can be applied to an electronic device, and specifically can be applied to a server or a general computer. In this embodiment, the method specifically includes the following steps:

Step 401, acquiring a target image to be detected.

According to a feasible implementation manner, an image captured by the third sensor on the third area is acquired, and the image is used as the target image. Here, the forest tree area corresponding to the target image and the forest tree area corresponding to the above-mentioned multiple target images may be different.

Step 402: Segment the target image to determine multiple sub-images.

According to a feasible implementation manner, overlapping divisions are performed on the target image through a sliding window to obtain multiple sub-images whose size meets the input requirements of the model, for example, 17×17 pixels.

In step 403, the feature extractor and the classifier are used to detect and classify the multiple sub-graphs respectively, and the detection and classification results are determined; the feature extractor and the classifier are obtained by training any of the above methods, and the detection and classification results include multiple a target box and the respective categories of the plurality of target boxes.

As shown in Figure 5, for each of the multiple sub-images, the sub-images are input into the feature extractor and the classifier in turn to obtain the detection and classification results of the classifier for the multiple sub-images, including the coordinates of the target frame of each sub-image and the Category, here, there can be multiple target boxes for each subgraph. In practical applications, the output of the classifier is a probability distribution (including probability values corresponding to multiple categories), and the category corresponding to the maximum probability value in the probability distribution is used as the category of the target frame in the detection classification result.

Step 404: Merge the target frames belonging to the same category to determine the target detection result of the target image.

According to a feasible implementation manner, a criterion based on Intersection-Of-Union (IOU) is used to merge target frames of the same category, and the IOU-based merging method does not require iterative steps, which can improve the merging efficiency. For example, if the IOU of two target boxes with the same category is greater than or equal to a given threshold, the coordinates of the two target boxes are averaged. In practical applications, it is sufficient to merge multiple target frames with an IOU greater than or equal to a given threshold; the calculation formula of the merged target frame is shown in the following formula (10).

Among them, (X _lt , Y _lt ) represents the coordinates of the upper left corner of the combined target frame; (X _rb , Y _rb ) represents the coordinates of the lower right corner of the combined target frame; n represents multiple targets with IOU greater than the threshold The number of boxes; (x _{lt, i} , y _{lt, i} ) represents the coordinates of the upper left corner of the ith target box in multiple target boxes with IOU greater than the threshold; (x _{rb, i} , y _{rb, i} ) represents the IOU The coordinates of the lower right corner of the ith target box in multiple target boxes greater than the threshold.

In practical applications, only the target boxes of the specified category, such as the oil palm tree category, can be merged.

It can be known from the above technical solutions that the beneficial effects of the present embodiment are:

The target image is segmented, and each segmented image is detected and classified separately, and then target frames of the same type are combined to improve the accuracy of the target detection result of the image.

Based on the same concept as the method embodiment of the present invention, please refer to FIG. 6 , the embodiment of the present invention further provides a model training device, including:

The image acquisition module 601 is used to obtain a plurality of target images based on remote sensing images obtained by shooting different forest areas respectively, and the plurality of target images include a labeled source domain image and an unlabeled target domain image. The image and the forest area corresponding to the target domain image are different;

The classification module 602 is used for each image of the plurality of target images: substitute the target image into a feature extractor to perform feature extraction processing to obtain a first feature map; input the first feature map into the classifier to obtain Classification result of the target image; inputting the first feature map into a first discriminator to obtain a first probability value that the target image belongs to the source domain;

a first loss calculation module 603, configured to obtain a first loss based on the respective target detection classification results and the first probability value corresponding to the multiple target images, which indicates the uncertainty of the classification result;

The second loss calculation module 604 is configured to obtain a second loss based on the label and the corresponding classification result of each image in the source domain, which indicates the classification error of the classifier;

A third loss calculation module 605, configured to obtain a third loss based on the respective first probability values corresponding to the multiple target images, which indicates the error of the source domain classification based on the first feature map;

A training module 606, configured to train the feature extractor, the classifier and the first discriminator based on the first loss, the second loss and the third loss.

According to a feasible implementation manner, the multiple target images are obtained by segmenting remote sensing images obtained from different forest areas;

Each source domain image has a variety of labels, including at least the forest tree category and the area between trees;

The different forest tree areas respectively include forest trees belonging to the target forest tree category;

the target domain image and the source domain image are from different sensors; and/or,

The corresponding shooting seasons of the target domain image and the source domain image are different.

According to a feasible implementation manner, the feature extractor includes a first extraction layer, a second extraction layer, and a second discriminator; the first extraction layer is used to extract shallow features on the target image, and obtain the second feature map; the second discriminator is used to determine the second probability value of the target image belonging to the source domain based on the second feature map; the second extraction layer is used to determine the second probability value based on the second feature The deep feature extraction is performed on the map and the second probability value to obtain a first feature map.

In one example, the feature extractor includes a plurality of convolutional blocks and a pooling layer, the first advance layer includes a plurality of convolutional blocks preceding the pooling layer, and the second extraction layer includes the pooling layer layer and multiple convolutional blocks after;

A single convolutional block includes convolutional layers, batch normalization layers, instance normalization layers, and activation layers.

In one example, the second extraction layer is configured to calculate entropy based on the second probability value, determine a feature attention value based on the calculated entropy value, and perform a feature attention value based on the feature attention value and the second feature map. Extraction of deep features to obtain a first feature map.

According to a feasible implementation manner, the apparatus further includes: a third loss calculation module; wherein,

The third loss calculation module is configured to obtain a fourth loss based on the respective second probability values corresponding to the multiple target images, which indicates the error of the source domain classification based on the second feature map.

The training module 606 is configured to train the feature extractor, the classifier and the first discriminator based on the first loss, the second loss, the third loss and the fourth loss.

According to a feasible implementation manner, the first loss calculation module 603 includes: an attention calculation unit and a loss calculation unit; wherein,

The attention calculation unit is configured to, for each of the multiple target images, calculate entropy based on the corresponding first probability value, and obtain an entropy attention value based on the calculated entropy value, where the entropy attention value indicates a the degree of attention to the target image;

The loss calculation unit is configured to determine the first loss based on the respective corresponding entropy attention values of the multiple target images and the probability values of each category in the classification result.

Based on the same concept as the method embodiment of the present invention, please refer to FIG. 7 , the embodiment of the present invention further provides a device for target detection, including:

An image acquisition module 701, configured to acquire a target image to be detected;

A segmentation module 702, configured to segment the target image to determine multiple sub-images;

The classification module 703 is configured to detect and classify the plurality of sub-images respectively through the feature extractor and the classifier, and determine the detection and classification result; the feature extractor and the classifier use any one of the methods described in the first aspect above. Obtained through training, the detection and classification result includes multiple target frames and respective categories of the multiple target frames;

The merging module 704 is used for merging each target frame belonging to the same category to determine the target detection result of the target image.

FIG. 8 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. At the hardware level, the electronic device includes a processor 801 and a memory 802 storing execution instructions, and optionally an internal bus 803 and a network interface 804 . The memory 802 may include a memory 8021, such as a high-speed random-access memory (Random-Access Memory, RAM), and may also include a non-volatile memory 8022 (non-volatile memory), such as at least one disk memory, etc.; the processor 801, the network interface 804 and the memory 802 can be connected to each other through an internal bus 803, and the internal bus 803 can be an ISA (Industry Standard Architecture, industry standard architecture) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or EISA (Extended Industry Standard Architecture, Extended Industry Standard Architecture) bus, etc.; the internal bus 803 can be divided into address bus, data bus, control bus, etc. For convenience, only one bidirectional arrow is used in FIG. 8, but it does not mean that there is only one Root bus or a type of bus. Of course, the electronic equipment may also include hardware required for other services. When the processor 801 executes the execution instructions stored in the memory 802, the processor 801 executes the method in any one of the embodiments of the present invention, and is at least configured to execute the method shown in FIG. 3 or FIG. 4 .

In a possible implementation manner, the processor reads the corresponding execution instruction from the non-volatile memory into the memory and then executes it, and also obtains the corresponding execution instruction from other devices, so as to form a logic level A device for model training or a device for object detection. The processor executes the execution instructions stored in the memory, so as to implement a model training method or a target detection method provided in any embodiment of the present invention through the executed execution instructions.

A processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above-mentioned method can be completed by a hardware integrated logic circuit in a processor or an instruction in the form of software. The above-mentioned processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; it may also be a digital signal processor (Digital Signal Processor, DSP), dedicated integrated Circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Various methods, steps, and logical block diagrams disclosed in the embodiments of the present invention can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

Embodiments of the present invention further provide a computer-readable storage medium, including execution instructions. When a processor of an electronic device executes the execution instructions, the processor executes the method provided in any one of the embodiments of the present invention. Specifically, the electronic device may be the electronic device shown in FIG. 8 ; the execution instruction is a computer program corresponding to an apparatus for model training or an apparatus for target detection.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method or a computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or a combination of software and hardware.

Each embodiment of the present invention is described in a progressive manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for related parts.

It should also be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a series of elements includes not only those elements, but also Other elements not expressly listed, or which are inherent to such a process, method, article of manufacture, or apparatus are also included. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article of manufacture, or device that includes the element.

The above descriptions are merely embodiments of the present invention, and are not intended to limit the present invention. Various modifications and variations of the present invention are possible for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims (10)

1. A method of model training, comprising:

obtaining a plurality of target images based on remote sensing images obtained by respectively shooting different forest regions, wherein the target images comprise source domain images with labels and target domain images without labels, and the forest regions corresponding to the source domain images and the target domain images are different;

for each image of the plurality of target images:

substituting the target image into a feature extractor to perform feature extraction processing to obtain a first feature map;

inputting the first feature map into a classifier to obtain a classification result of the target image;

inputting the first feature map into a first discriminator to obtain a first probability value of the target image belonging to a source domain;

obtaining a first loss indicating an uncertainty of the classification result based on the target detection classification result and the first probability value corresponding to each of the plurality of target images;

obtaining a second loss which indicates the classification error of the classifier based on the label of each source domain image and the corresponding classification result;

obtaining a third loss indicating an error of source domain classification based on the first feature map based on the first probability values corresponding to the target images respectively;

training the feature extractor, classifier, and first discriminator based on the first loss, the second loss, and the third loss.

2. The method according to claim 1, wherein the plurality of target images are obtained by segmenting based on remote sensing images respectively captured in different forest regions;

the source domain images have various labels, and at least comprise the forest type and the forest region;

the different forest regions respectively comprise forests belonging to the target forest category;

the target domain image and the source domain image are from different sensors; and/or the presence of a gas in the atmosphere,

the shooting seasons corresponding to the target domain image and the source domain image are different.

3. The method of claim 1, wherein the feature extractor comprises a first extraction layer, a second discriminator;

the first extraction layer is used for extracting shallow features of the target image to obtain a second feature map;

the second discriminator is used for judging a second probability value of the target image belonging to the source domain based on the second feature map;

the second extraction layer is used for extracting deep features based on the second feature map and the second probability value to obtain a first feature map.

4. The method of claim 3, wherein the feature extractor comprises a plurality of volume blocks and a pooling layer, the first extraction layer comprises a plurality of volume blocks before the pooling layer, and the second extraction layer comprises the pooling layer and a plurality of volume blocks after;

a single volume block includes a volume layer, a batch normalization layer, an instance normalization layer, and an activation layer.

5. The method of claim 3, wherein the second extraction layer is configured to calculate entropy based on the second probability value, determine a feature attention value based on the calculated entropy value, and extract deep features based on the feature attention value and the second feature map to obtain the first feature map.

6. The method of claim 3, further comprising:

obtaining a fourth loss based on the second probability values corresponding to the target images respectively, wherein the fourth loss indicates an error of source domain classification based on a second feature map;

the training the feature extractor, the classifier, and the first discriminator based on the first loss, the second loss, and the third loss includes:

training the feature extractor, classifier, and first discriminator based on the first loss, the second loss, the third loss, and a fourth loss.

7. The method of claim 1, wherein determining a first loss based on the target detection classification result and the first probability value for each of the plurality of target images comprises:

for each image of the plurality of target images, calculating entropy based on the corresponding first probability value, and deriving an entropy attention value based on the calculated entropy value, the entropy attention value indicating a degree of attention to the target image;

and determining a first loss based on the entropy attention values corresponding to the target images and the probability values of the categories in the classification result.

8. A method of target detection, comprising:

acquiring a target image to be detected;

segmenting the target image and determining a plurality of sub-images;

respectively carrying out detection classification on the multiple subgraphs through a feature extractor and a classifier, and determining a detection classification result; the feature extractor and the classifier are trained by the method of any one of claims 1 to 7, and the detection classification result comprises a plurality of target frames and respective categories of the target frames;

and merging the target frames belonging to the same category, and determining a target detection result of the target image.

9. A computer-readable storage medium comprising executable instructions that, when executed by a processor of an electronic device, cause the processor to perform the method of any of claims 1 to 7, or the method of claim 8.

10. A computing device comprising a processor and a memory storing execution instructions, the processor performing the method of any of claims 1 to 7, or the method of claim 8, when the processor executes the execution instructions stored by the memory.

Images