JP7563241B2 - Neural network and training method thereof - Google Patents

Description

The present invention relates to a neural network and a training method thereof, and in particular to a neural network model for object detection and a corresponding training method.

At present, with the widespread use of smartphones, people are beginning to enjoy the convenience brought by science and technology. This has also, to a certain extent, stimulated the continuous development of artificial intelligence technology. Artificial intelligence often requires a lot of computing power to achieve certain effects. However, many mature algorithms cannot be deployed and applied on smartphones because the processing power of the hardware of mobile platforms is usually limited.

Therefore, researchers have begun to explore the realization of small models. In recent years, many efficient architectures have been proposed, such as Pelee described in Non-Patent Document 1, ShuffleNetV2 described in Non-Patent Document 2, and MobileNetV3 described in Non-Patent Document 3. Although these models can meet the real-time requirements of mobile platforms, there is still room for improving the accuracy. In particular, for detection tasks, the accuracy loss of small models is much larger than that for classification tasks.

As we all know, detection tasks are fundamental research in computer vision, which has been widely studied and applied in practice. Most traditional target detection models use networks designed for image classification as their backbone, and developers have developed various feature representations for detectors.

Generally speaking, traditional detection models usually suffer from the following shortcomings:

1) The detection model relies on a large amount of manual effort and prior knowledge, and although good detection accuracy can be obtained, it is not suitable for real-time tasks;
2) Although artificially designed small models or pruned models can address real-time problems, the accuracy is often not high because the backbone of these models is from networks designed for classification tasks.

Wang, R. J. , Li, X. , Ao, S. , Ling, C. X. : Pelee: A real-time object detection system on mobile devices. arXiv preprint arXiv:1804.06882 (2018) Ninning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018 Howard, A. , Sandler, M. , Chu, G. , Chen, L. -C. , Chen, B. , Tan, M. , Wang, W. , Zhu, Y. , Pang, R. , Vasudevan, V. , et al. Searching for mobilenetv3. arXiv preprint arXiv:1905.02244, 2019

The inventors of the present invention have discovered that an end-to-end detection model is necessary and efficient for the detection task. Based on this, the objective of the present invention is to provide an end-to-end detection model and a corresponding training method suitable for resource-limited platforms.

According to one aspect of the present invention, there is provided a method for training a neural network, wherein the neural network is used for detecting an object in an image and includes a backbone network, a feature network and a prediction module, the feature network includes a first module and a second module, and the method includes:
The backbone network processes the sample image and outputs N first features of different sizes;
A first module of the feature network performs N-1 deconvolution operations based on the smallest first feature output from the backbone network, and outputs N second features of different sizes;
A second module of the feature network performs merging on the N first features output from the backbone network, and outputs N third features of different sizes;
generating N fourth features of different sizes by combining each of the N second features with a corresponding one of the N third features having the same size, and performing a different number of convolution operations on the N fourth features;
the prediction module making a prediction based on the N fourth features and calculating a first loss;
the prediction module making a prediction based on the features obtained after the convolution operation and calculating a second loss; and training the neural network by optimizing settings of the backbone network, the feature network and the prediction module based on a combination of the first loss and the second loss.

According to another aspect of the present invention, a neural network for detecting an object in an image is provided, the neural network including a backbone network, a feature network, and a prediction module. The feature network includes a first module and a second module. The backbone network processes a sample image and outputs N first features of different sizes. The first module of the feature network performs N-1 deconvolution operations based on the smallest first feature output from the backbone network, and outputs N second features of different sizes. The second module of the feature network performs merging on the N first features output from the backbone network, and outputs N third features of different sizes. Each of the N second features is combined with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and the N fourth features are convolved a different number of times. The prediction module makes a prediction based on the N fourth features and calculates a first loss, and makes a prediction based on the features obtained after the convolution operation and calculates a second loss. The neural network is trained based on a combination of the first loss and the second loss.

According to another aspect of the present invention, there is provided an apparatus for training a neural network, the neural network being used for detecting an object in an image, the apparatus including a backbone network, a feature network, and a prediction module, the feature network including a first module and a second module. The backbone network performs processing on a sample image and outputs N first features of different sizes. The first module of the feature network performs N-1 deconvolution operations based on the smallest first feature output from the backbone network, and outputs N second features of different sizes. The second module of the feature network performs merging on the N first features output from the backbone network, and outputs N third features of different sizes. Each of the N second features is combined with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and the N fourth features are subjected to different convolution operations. The prediction module performs prediction based on the N fourth features and calculates a first loss, and performs prediction based on the features obtained after the convolution operation and calculates a second loss. The apparatus includes one or more processors, the processors configured to train the neural network by optimizing settings of the backbone network, the feature network, and the prediction module based on a combination of the first loss and the second loss.

According to another aspect of the present invention, a storage medium is provided that stores a program, which, when executed, causes a computer to execute a method for training a neural network. The neural network is used for detecting an object in an image, and includes a backbone network, a feature network, and a prediction module, and the feature network includes a first module and a second module. The backbone network processes a sample image and outputs N first features of different sizes. The first module of the feature network performs N-1 deconvolution operations based on the smallest first feature output from the backbone network, and outputs N second features of different sizes. The second module of the feature network performs merging on the N first features output from the backbone network, and outputs N third features of different sizes. Each of the N second features is combined with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and different convolution operations are performed on the N fourth features. The prediction module makes predictions based on the N fourth features and calculates a first loss, and makes predictions based on features obtained after the convolution operation and calculates a second loss. The method includes training the neural network by optimizing settings of the backbone network, the feature network, and the prediction module based on a combination of the first loss and the second loss.

FIG. 2 illustrates the architecture of an object detection model according to the present invention. A diagram showing the architecture of a backbone network. FIG. 2 illustrates a convolution unit. FIG. 2 illustrates the architecture of the feature network and target prediction module. FIG. 13 illustrates the merging process performed by the second module of the feature network. 1 is a flowchart of a method for training an object detection model according to an embodiment of the present invention. FIG. 2 is an exemplary block diagram of computer hardware in which the present invention may be implemented.

Below, a preferred embodiment for carrying out the present invention will be described in detail with reference to the attached drawings. Note that the following embodiment is merely illustrative and does not limit the present invention.

Figure 1 is a diagram showing the architecture of an object detection model according to the present invention. The object detection model can be realized by a neural network. As shown in Figure 1, the object detection model includes a backbone network 110, a feature network 120, and a prediction module 130. The backbone network 110 constitutes the basic network of the detection model, the feature network 120 is used to extract feature representations, and the target prediction module 130 performs object detection using the extracted feature representations.

Also, the images input to the detection model may be scaled to have a uniform size, e.g., 320x320. The prediction module 130 may, for example, consist of a 3x3 convolution operation, and finally output an output image with a bounding box and a class label, where the bounding box indicates the location of the detected object and the class label indicates the class to which the object belongs.

Figure 2 is an exemplary diagram illustrating the architecture of a backbone network. As shown in Figure 2, the backbone network according to the present invention includes, for example, 17 layers, but the present invention is not limited thereto. The first layer is the backbone layer, which performs a convolution operation of a predetermined step length, for example, a 3x3 convolution operation with a step length of 2. Each of the second to seventeenth layers is composed of the convolution unit shown in Figure 3, but the second to seventeenth layers may be different from each other in terms of the configuration of the convolution unit.

As shown in FIG. 3, the current layer receives the output of the previous layer, performs random mixing of channels on it, and then divides the channels into two equal parts to form two branches. The same processing is performed in these two branches, and the calculations can be performed in parallel, so that the processing time can be saved compared with a single-branch convolutional layer. Each branch sequentially performs the following operations: (1) 1×1 convolution operation, followed by an increase in the number of channels by a certain factor, i.e., channel expansion; (2) K×K deep convolution operation; (3) 1×1 convolution operation, followed by a decrease in the number of channels by the same factor, i.e., channel reduction, so that the number of channels is the initial number in operation (1). Finally, the outputs of the two branches are cascaded and output to the next layer.

The convolution unit shown in FIG. 3 may have different settings in different layers. Specifically, at least one of the following may be set: channel expansion factor, deep convolution kernel size, whether to add processed and unprocessed channels, and whether to add a squeeze-and-excitation block. As an example, an optimal combination can be obtained based on prior knowledge.

For example, the channel expansion factor may be 1, 3, or 6, and the size of the deep convolution kernel may be 3x3 or 5x5. Also, although not shown in FIG. 3, an SE block may be added to the convolution units of a certain layer or some layers to obtain higher accuracy. In such a case, an SE block may be added between the deep convolution and the 1x1 convolution (channel reduction).

Also, as shown in Figure 3, in each branch, the output of that branch is generated by taking the sum of the convoluted channel and the unprocessed channel, as indicated by the "shortcut" arrow and the "addition" sign (+) in the figure. However, this setting can be changed. For example, such addition processing is not performed when there is a change in channel or feature size between the current layer and the previous layer. In other words, in such a case, the "shortcut" arrow and the "addition" sign (+) in Figure 3 are removed, and only the convoluted channel is used as the output of the corresponding branch.

The specific parameters of the backbone network are shown in Table 1 below, in which "unit" represents the convolution unit shown in FIG. 3, and "Y" represents an additional SE block.

As shown in FIG. 1, the output of the backbone network 110 is the input of the feature network 120. Since the feature pyramid network (FPN) is suitable for processing objects of different sizes in an image, the present invention uses the FPN as the basic feature structure. Therefore, the present invention generates a first feature based on the output of each layer (layer 1 to layer 17) of the backbone network 110, and the first feature is input to the feature network 120 as the output of the backbone network 110.

The generation of the first feature will be described below with an example. In this example, features output by the 5th, 12th, and 17th layers of the backbone network 110 are selected, and these features are defined as {f1, f2, f3}. The features f1, f2, and f3 have different sizes, for example, 40×40, 20×20, and 10×10, respectively, and among them, the size of the feature f3 is the smallest. Then, by performing max pooling operations with step lengths of 2 and 4 on the feature f3, respectively, two other features f4 and f5 with smaller sizes are obtained. The five obtained features of different sizes {f1, f2, f3, f4, f5} are used to construct the first feature, which is then input to the feature network 120. Note that the present invention is not limited to this example, and other methods can also be used to generate the first feature. For example, features output from layers other than the 5th, 12th, and 17th layers may be selected, or the number of features included in the first feature may be changed.

Figure 4 is an exemplary diagram illustrating the architecture of the feature network and the target prediction module. As shown in Figure 4, the feature network includes a first module 410 and a second module 420, and receives a first feature F1 from a backbone network. The first feature F1 includes N features of different sizes (five are shown in the figure as an example).

The first module 410 performs N-1 (four) deconvolution operations based on the feature f5 that has the smallest size among the first features F1, to generate a set of new features (represented as second features F2). Specifically, it first performs a first deconvolution operation on feature f5, and then performs a second deconvolution operation on the features obtained after the first deconvolution operation, and performs a total of N-1 deconvolution operations in this manner. For example, the generated second feature F2 includes feature f5 and four features generated by four deconvolution operations.

The second module 420 performs merging on the features included in the first feature F1 to generate a set of new features (denoted as the third feature F3). FIG. 5 illustrates a method of performing the merging process. As shown in FIG. 5, a merging process is performed between the i+1th feature and the ith feature, where i=1, 2, ..., N-1, and the size of the i+1th feature is smaller than that of the ith feature. First, a process including bilinear interpolation and convolution operation is performed on the i+1th feature. The bilinear interpolation is used to change the size of the i+1th feature to be the same as that of the ith feature by upsampling it. The convolution operation is used to realize channel normalization of the two features. Then, the ith feature is merged with the processed i+1th feature to obtain a new ith feature. After that, the new i-1th feature is obtained by performing the merging process shown in FIG. 5 on the obtained new i-1th feature and the original i-1th feature.

N-1 new features can be generated by the above method. Then, the second module 420 performs a max pooling operation on the N-1th new feature with the smallest size among these new features to obtain the Nth new feature, whose size is smaller than the N-1th new feature. Up to this point, N new features have been generated, which constitute the output of the second module 420, i.e., the third feature F3.

The first module 410 and the second module 420 can improve efficiency by performing processes in parallel. Then, the second feature F2 output by the first module 410 and the third feature F3 output by the second module 420 are cascaded to obtain an enhanced multi-scale feature, i.e., a fourth feature F4. More specifically, the fourth feature F4 including N features is generated by cascading each feature included in the second feature F2 with a feature having the same size in the third feature F3.

Figure 4 shows a fourth feature F4 that includes five features {p1, p2, p3, p4, p5}. The prediction module 130 performs prediction based on the fourth feature F4 and calculates the first loss. Note that this is known to those skilled in the art, so the detailed process is omitted in this document.

Meanwhile, different numbers of convolution operations (e.g., 3×3 convolutions) are performed on features p1, p2, p3, p4, and p5 included in the fourth feature F4, and the prediction module 130 performs prediction based on the features obtained after the convolution operations and calculates the second loss. More specifically, as the size of a feature increases, the number of convolution operations performed thereon decreases. In this respect, FIG. 4 exemplarily shows that no convolution operation is performed on feature p1, which has the largest size, and one to four convolutions are performed on features p2 to p5, which have gradually decreasing sizes. Note that the present invention is not limited to the example shown in FIG. 4.

Performing convolution operations on features with relatively large scales can result in a relatively large amount of calculations, which can lead to a decrease in efficiency. Therefore, in the present invention, a good balance between accuracy and efficiency can be achieved by performing a relatively small number of convolution calculations on features with relatively large sizes and a relatively large number of convolution calculations on features with relatively small sizes.

The obtained first loss and second loss are used to train the object detection model shown in FIG. 1. Specifically, the prediction module 130 calculates the regression representation and the sum classification representation, respectively, using, for example, 3×3 convolution. In terms of model evaluation, for example, the regression loss method and the focal loss method can be adopted. The regression loss (RLOSS) represents the loss related to the detection of the bounding box, and the focal loss (FLOSS) represents the loss related to the class label. Note that the focal loss is one type of classification loss, and although the present invention focuses on the focal loss, other classification losses may be used in the present invention.

The first loss includes a regression loss RLOSS ₁ and a focus loss FLOSS ₁ , and the second loss includes a regression loss RLOSS ₂ and a focus loss FLOSS _2. Therefore, when training the detection model m, the loss function expressed by the following equation (1) can be adopted.

By minimizing the loss function LOSS(m), the setting parameters of the detection models of the backbone network 110, the feature network 120 and the prediction module 130 can be optimized to obtain good detection accuracy.

Figure 6 is a flowchart of a method for training an object detection model according to the present invention. As shown in Figure 6, in step S610, the backbone network 110 processes an input image to generate and output a first feature F1. The first feature F1 includes N features of different sizes, for example, {f1, f2, f3, f4, f5} as described above.

In step S620, the first module 410 of the feature network 120 performs N-1 deconvolution operations based on the smallest feature (e.g., feature f5) among the first features F1 output by the backbone network 110 to generate a second feature F2. The second feature F2 includes the smallest feature and the feature obtained after each deconvolution operation.

In step S630, the second module 420 of the feature network 120 performs merging on N features among the first features F1 output by the backbone network 110 to generate a third feature F3. For the method of merging, please refer to the explanation of FIG. 5 above.

In step S640, the second feature F2 and the third feature F3 are combined according to the size grade to generate a fourth feature F4. The fourth feature F4 includes N features of different sizes, for example, {p1, p2, p3, p4, p5} as described above.

In step S650, a different number of convolution operations are performed on each feature in the fourth feature F4. In particular, as the size of a feature increases, the number of convolution operations performed on it decreases.

In step S660, the prediction module 130 makes a prediction based on the fourth feature F4 that has not undergone the convolution operation and calculates a first loss. The prediction module 130 also makes a prediction based on the features obtained after the convolution operation and calculates a second loss.

In step S670, the object detection model is trained based on the loss function shown in the above equation (1), and the settings of the backbone network 110, the feature network 120, and the prediction module 130 are optimized.

The end-to-end object detection model and its training method proposed in the present invention have been described above based on specific examples. Compared to conventional models, the detection model according to the present invention has at least the following advantages:

This detection model is specifically designed for detection tasks on small platforms;
The model architecture design takes into full account the parallel processing capabilities of the device and can process data streams simultaneously;
In each layer of the backbone network (each layer other than the backbone layer), the size of the convolution kernel and the number of channels are flexibly set. This allows for higher accuracy than models that use fixed convolution kernel sizes and fixed numbers of channels in all layers;
A good balance between accuracy and efficiency can be achieved.

The methods described in the above-described embodiments can be realized by software, hardware, or a combination of software and hardware. The programs included in the software can be stored in advance on a storage medium installed inside or outside the device. As one example, at run time, these programs can be written into a ROM and executed by a processor (e.g., a CPU) to realize the various methods and processes described herein.

Figure 7 is a block diagram of computer hardware (general-purpose PC 700) that executes the method of the present invention based on a program. The computer hardware is an example of an apparatus for training a detection model in the present invention. In addition, the backbone network, feature network, and prediction module in the detection model of the present invention can also be realized using the computer hardware.

The general-purpose PC 700 may be, for example, a computer system. Note that the general-purpose PC 700 is merely an example, and the scope or functionality of the method and device according to the present invention is not limited thereto. Furthermore, the general-purpose PC 700 does not depend on any module, assembly, etc., or combination thereof, in the above-mentioned method and device.

In FIG. 7, a central processing unit (CPU) 701 can perform various processes based on a program stored in a ROM 702 or a program loaded from a storage unit 708 to a RAM 703. The RAM 703 can also store data required when the CPU 701 performs various processes, depending on the needs. The CPU 701, the ROM 702, and the RAM 703 can be connected to each other via a bus 704. An input/output interface 705 can also be connected to the bus 704.

The input/output interface 705 is further connected to the following components: an input section 706 including a keyboard, an output section 707 including a display such as a liquid crystal display (LCD) and a speaker, a storage section 708 including a hard disk, and a communication section 709 including a network interface card such as a LAN card and a modem. The communication section 709 performs communication processing via a network such as the Internet or a LAN. A drive 710 may be connected to the input/output interface 705 according to needs. A removable medium 711, such as a semiconductor memory, can be set in the drive 710 as needed, and a computer program read from the medium can be installed in the storage section 708.

The present invention further provides a program product including machine-readable instruction code. Such instruction code, when read and executed by a machine, can execute the method in the above-described embodiment of the present invention. Accordingly, various storage media that carry such a program product, such as magnetic disks (including floppy disks (registered trademark)), optical disks (including CD-ROMs and DVDs), magneto-optical disks (including MDs (registered trademark)), and semiconductor memory devices, are also included in the present invention.

The above-mentioned storage media may include, for example, but are not limited to, magnetic disks, optical disks, magneto-optical disks, semiconductor memory devices, etc.

Furthermore, each operation (process) in the above-described method can be realized in the form of a computer-executable program stored on various machine-readable storage media.

Furthermore, the above examples are disclosed in the following addendum.

(Appendix 1)
1. A method for training a neural network, comprising:
The neural network is used for detecting an object in an image, and includes a backbone network, a feature network, and a prediction module, the feature network includes a first module and a second module, and the method includes:
The backbone network operates on the sample image and outputs N first features of different sizes;
A first module of the feature network performs N-1 deconvolution operations based on a first feature with the smallest size output from the backbone network, and outputs N second features with different sizes;
A second module of the feature network performs merging on the N first features output from the backbone network, and outputs N third features of different sizes;
generating N fourth features of different sizes by combining each of the N second features with a corresponding one of the N third features having the same size, and performing a different number of convolution operations on the N fourth features;
the prediction module making a prediction based on the N fourth features and calculating a first loss;
the prediction module making a prediction based on the features obtained after the convolution operation and calculating a second loss; and training the neural network by optimizing settings of the backbone network, the feature network, and the prediction module based on a combination of the first loss and the second loss.

(Appendix 2)
2. The method according to claim 1, comprising:
The N second features output by the first module of the feature network include the first feature with the smallest size and a feature obtained after performing a deconvolution operation each time.

(Appendix 3)
The method according to claim 1, further comprising:
A second module of the feature network performs processing on the i+1th first feature, including bilinear interpolation and convolution operations, and merges the ith first feature with the processed i+1th first feature;
Among them, the method where i=1, 2, . . . , N-1.

(Appendix 4)
The method according to claim 1, further comprising:
a second module of the feature network performing a process including a bilinear interpolation and a convolution operation on the i+1th first feature, and performing a merging of the ith first feature with the processed i+1th first feature to obtain an ith third feature, where i=1, 2, ..., N-1; and a second module performing a max pooling operation on the N-1th third feature to obtain an Nth third feature.

(Appendix 5)
2. The method according to claim 1, comprising:
A method wherein as the size of the fourth feature increases, the number of convolution operations performed thereon decreases.

(Appendix 6)
2. The method according to claim 1, comprising:
wherein each of the first loss and the second loss comprises a regression loss and a classification loss.

(Appendix 7)
2. The method according to claim 1, comprising:
the backbone network includes a plurality of layers, and the channels of each layer are divided into two equal parts;
The method further includes performing processing including convolution, channel expansion, and channel reduction on the channels of each of the two parts, and combining the processed channels of the two parts for input to a next layer.

(Appendix 8)
The method according to claim 7, further comprising:
For each partial channel of each layer, summing a processed channel with an unprocessed channel to obtain an output of the portion; and combining the two partial outputs as input to the next one layer.

(Appendix 9)
9. The method of claim 8, further comprising:
The method, wherein the processing performed on the channels of each of the two portions further comprises adding an SE block.

(Appendix 10)
10. The method of claim 9, further comprising:
The method includes setting at least one of a channel expansion factor, a convolution kernel size, whether to sum processed and unprocessed channels, and whether to add an SE block for each layer of the backbone network.

(Appendix 11)
The method according to claim 7, further comprising:
generating a plurality of features of different sizes through a plurality of layers of the backbone network;
selecting a subset of the plurality of features, and determining a particular feature within the subset that is smallest in size;
performing processing on the particular feature to generate one or more features smaller in size than the particular feature; and constituting N first features output from the backbone network with the features in the subset and the one or more features smaller in size than the particular feature.

(Appendix 12)
1. A neural network for use in detecting objects in an image, comprising:
The neural network comprises:
a backbone network, the backbone network operating on a sample image and outputting N first features of different sizes;
A feature network, the feature network including a first module and a second module, the first module performing N-1 deconvolution operations based on a first feature with a smallest size output from the backbone network, and outputting N second features of different sizes, the second module performing a merge on the N first features output from the backbone network, and outputting N third features of different sizes, in which each of the N second features is combined with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and performing different numbers of convolution operations on the N fourth features; and a prediction module, the prediction module performing a prediction based on the N fourth features and calculating a first loss, and performing a prediction based on a feature obtained after the convolution operation and calculating a second loss,
Wherein, the neural network is trained based on a combination of the first loss and the second loss.

(Appendix 13)
1. An apparatus for training a neural network, comprising:
The neural network is used for detecting an object in an image, and includes a backbone network, a feature network and a prediction module, the feature network includes a first module and a second module;
The backbone network processes the sample image and outputs N first features of different sizes;
The first module of the feature network performs N-1 deconvolution operations based on the smallest first feature output from the backbone network, and outputs N second features of different sizes;
A second module of the feature network performs merging on the N first features output from the backbone network, and outputs N third features of different sizes;
combining each of the N second features with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and performing a different number of convolution operations on the N fourth features;
The prediction module performs prediction based on the N fourth features and calculates a first loss; and performs prediction based on features obtained after the convolution operation and calculates a second loss;
The apparatus includes one or more processors configured to train the neural network by optimizing settings of the backbone network, the feature network, and the prediction module based on a combination of the first loss and the second loss.

(Appendix 14)
A storage medium storing a program,
The program, when executed, causes a computer to perform a method for training a neural network,
Wherein, the neural network is used for detecting an object in an image, and includes a backbone network, a feature network and a prediction module;
the feature network includes a first module and a second module;
The backbone network processes the sample image and outputs N first features of different sizes;
The first module of the feature network performs N-1 deconvolution operations based on the smallest size first feature output from the backbone network, and outputs N second features of different sizes;
A second module of the feature network performs merging on the N first features output from the backbone network, and outputs N third features of different sizes;
each of the N second features is combined with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and a different number of convolution operations are performed on each of the N fourth features;
The prediction module performs prediction based on the N fourth features and calculates a first loss; and performs prediction based on features obtained after the convolution operation and calculates a second loss;
wherein the method includes training the neural network by optimizing settings of the backbone network, the feature network, and the prediction module based on a combination of the first loss and the second loss.

Although a preferred embodiment of the present invention has been described above, the present invention is not limited to this embodiment, and any modification to the present invention falls within the technical scope of the present invention as long as it does not deviate from the spirit of the present invention.

Claims (10)

1. A method for training a neural network, comprising:
The neural network is used for detecting an object in an image, and includes a backbone network, a feature network, and a prediction module;
the feature network includes a first module and a second module;
The method comprises:
The backbone network processes the sample image and outputs N first features of different sizes;
A first module of the feature network performs N-1 deconvolution operations based on the smallest first feature output from the backbone network, and outputs N second features of different sizes;
A second module of the feature network performs a merging process on the N first features output from the backbone network to output N third features of different sizes;
combining each of the N second features with a corresponding one of the N third features having the same size to generate N fourth features of different sizes, and performing a different number of convolution operations on the N fourth features;
the prediction module making a prediction based on the N fourth features and calculating a first loss;
the prediction module makes a prediction based on the features obtained after the convolution operation and calculates a second loss; and training the neural network by optimizing settings of the backbone network, the feature network, and the prediction module based on a combination of the first loss and the second loss. 2. The method of claim 1 ,
The N second features output by the first module of the feature network include the first feature with the smallest size and a feature obtained after performing a deconvolution operation each time. 2. The method of claim 1 ,
The second module of the feature network further includes performing a process on the i+1th first feature, including a bilinear interpolation and a convolution operation, and performing a merge process on the i-th first feature and the processed i+1th first feature;
The method, where i=1, 2, . . . , N-1. 2. The method of claim 1 ,
The method, wherein the number of convolution operations performed on the fourth feature decreases as the size of the fourth feature increases. 2. The method of claim 1 ,
wherein each of the first loss and the second loss comprises a regression loss and a classification loss. 2. The method of claim 1 ,
the backbone network includes a plurality of layers, and the channels of each layer are divided into two equal parts;
The method comprises:
The method further comprises performing processing on the channels of each of the two parts, including a convolution operation, a channel expansion, and a channel reduction, and combining the processed channels of the two parts for input to a next layer. 7. The method of claim 6,
For each partial channel of each layer, summing the processed channel with the unprocessed channel to obtain an output of the portion; and combining the outputs of the two portions as input to the next one layer. 8. The method of claim 7,
The method, wherein the processing performed on the channels of each of the two portions further comprises the addition of a Squeeze-and-Excitation block (SE block). 9. The method of claim 8,
For each layer of the backbone network,
The method includes setting at least one of a channel expansion factor, a convolution kernel size, whether to sum processed and unprocessed channels, and whether to add an SE block. 7. The method of claim 6,
generating a plurality of features of different sizes through a plurality of layers of the backbone network;
selecting a subset of the plurality of features and determining a particular feature from the subset that is smallest in size;
performing processing on the particular feature to generate one or more features whose size is smaller than a size of the particular feature; and constituting N first features output from the backbone network with the features in the subset and the one or more features whose size is smaller than a size of the particular feature.

Images