CN111639751A - Non-zero padding training method for binary convolutional neural network - Google Patents

Abstract

The invention discloses a non-zero filling training method for a binary convolutional neural network. First, the loss function in ordinary neural network training is improved, and a joint loss function is constructed by using the knowledge distillation theory, so that the zero-filling binary network can be used for Guided training of nonzero-padded binary networks. Then, the progressive training method is used to train the non-zero-filled binary network. Based on the zero-filled binary network, the number of non-zero-filled binary activations is gradually increased, thereby reducing the training difficulty of the non-zero-filled binary network. The invention corrects the pseudo-binary activation problem in the zero-filled binary network, and effectively reduces the training difficulty of the non-zero-filled binary network by combining the joint loss function and the progressive training method, and greatly reduces the need for correcting and filling values. It brings the problem of performance degradation of non-zero-filled binary networks.

Description

A Non-Zero Padding Training Method for Binary Convolutional Neural Networks

technical field

The invention belongs to the field of image processing, in particular to a non-zero filling training method for a binary convolutional neural network.

Background technique

In the past ten years, deep learning has attracted more and more researchers' attention due to its huge advantages over shallow models in feature extraction and model construction, and has achieved rapid development in the fields of computer vision and text recognition. Deep learning takes the deep neural network as the main form, and the Convolutional Neural Network (CNN) is the pioneering research result inspired by biological neurology. Compared with the traditional method, the convolutional neural network has the characteristics of weight sharing, local connection, pooling operation, etc., so it can effectively reduce the global optimization of training parameters, reduce the complexity of the model, and make the network model better for input scaling, translation, and distortion. One has some degree of invariance. Under the advantage of this characteristic, convolutional neural networks have shown excellent performance in many computer vision tasks including image classification, object detection and tracking.

Although convolutional deep networks have shown reliable results in many vision tasks, the huge storage and computational overhead limit their application on popular portable devices. In order to expand the application of convolutional neural networks, the model's Compression and acceleration have become hot issues in the field of computer vision. At present, the compression methods for convolutional neural networks are mainly divided into three categories:

One is the network pruning method. The basic idea of this method is: Convolutional neural networks with better performance often have more complex structures, but some of the parameters do not contribute much to the final output result and appear redundant. Therefore, for an existing convolutional neural network, An effective method for evaluating the importance of convolution kernel channels can be found, and the corresponding redundant convolution kernel parameters can be cut off to improve the efficiency of the neural network. In this method, the evaluation method has a very important influence on the model performance.

The second is the neural network structure search method. This method can automatically search out a network with high speed and high precision through a good search algorithm within a certain range of search space, so as to achieve the purpose of network compression. The key to this approach is to build a large space of network architectures, explore the space through some efficient network search algorithms, and search for the optimal volume under a specific combination of training data and computational constraints (e.g., network size and latency) Integral Neural Network Architecture.

The third is the network quantification method. This method obtains a low-bit network by quantizing the 32-bit full-precision network weight parameters into lower-bit parameters (such as 8-bit, 4-bit, 1-bit, etc.). This method can effectively reduce parameter redundancy, thereby reducing storage occupancy, communication bandwidth and computational complexity, which is helpful for the application of deep networks in lightweight application scenarios such as artificial intelligence chips.

In the network quantization method, the network binarization method is a quantization method that can maximize the compression. This method quantizes the activation and weight in the network model into 1-bit, so that the model storage can theoretically obtain about 32 times. However, since the operations in the network are changed from the addition and multiplication of floating-point numbers to bit-level bit operations, the operating speed can theoretically be improved by about 58 times. Therefore, after binary quantization, the network can often be greatly improved. compression. At present, in the binary convolutional neural network, the sign function (sign( )) is often used to quantize the network weights and activations into two values {-1, +1}. For a specific convolutional layer, in order to keep the size of the feature map unchanged during the convolution process, it is often necessary to introduce a "padding" operation, that is, the feature map input to the convolutional layer is numerically padded. , and then perform a convolution operation on the filled feature map. For the current mainstream deep learning frameworks, such as Pytorch, the default value of the padding is often set to 0, which causes the activation of the convolutional layer to be quantized into {-1, +1} by the sign function. After the filling operation, there is another value - "0", the filling process can be seen in Figure 1. Since the current field of binary network research has not paid attention to this problem, in fact, various results reported in various studies are based on the "pseudo-binary network" with mixed activation of {-1, +1, 0}. ”, which cannot enjoy the extremely high compression ratio of the binary network in practical applications. It has been verified by experiments that when the padding is 1, it is a non-zero padding binary network with activation of {-1, +1} after quantization. Compared with the zero-padding binary network of the same structure, its performance is very obvious. Decline. How to make up for the performance degradation of the non-zero-padded binary network caused by correcting the padded value, so that it can run with high precision and high compression ratio in actual deployment in the future, has become a problem to be further studied.

SUMMARY OF THE INVENTION

Considering the "pseudo-binary" problem caused by the wrongly filled values of the convolution layer in the current binary network research, the present invention constructs a non-zero filled binary convolutional neural network whose activation is {-1, +1}, and A non-zero padding training method for binary convolutional neural networks is proposed to compensate for the performance degradation of binary networks after correcting the padding values.

The present invention is used for the non-zero padding training method of the binary convolutional neural network, which specifically includes the following steps:

Step 1: Prepare the training dataset to be applied to the vision task.

Step 2: Construct a deep convolutional neural network for vision tasks, binarize the weights and activations in the network into 1-bit values, and initialize it with pre-trained zero-padded binary network weights that are padded to 0.

Step 3: Use knowledge distillation theory to construct a joint loss function for non-zero-filled binary networks.

Step 4: Perform progressive training on the non-zero-padded binary network on the training data set, gradually increasing the number of channels padded to 1.

Step 5: Use the completely non-zero filled binary network obtained in Step 4 for the test set of the task to test its classification effect.

The advantages of the present invention are:

(1) The non-zero padding training method used in the present invention for binary convolutional neural network, taking into account the contradiction between binarization and padding values neglected in the past research, the activation is constructed as {-1, +1 }'s non-zero-padded binary deep network, which corrects the pseudo-binary activation problem in zero-padded binary networks.

(2) The present invention is used for the non-zero padding training method of the binary convolutional neural network. The knowledge distillation theory is used to construct a joint loss function for the training of the non-zero padding binary network. The guided training of the padded binary network compensates for the performance degradation of the non-zero padded binary network caused by correcting the padded values.

(3) The present invention is used for the non-zero padding training method of the binary convolutional neural network, and the progressive training method is used to train the non-zero padding binary network, and the evolution of the zero padding binary network to the non-zero padding binary network Provides a transition period that reduces the loss in accuracy of the non-zero padding network.

Description of drawings

Figure 1 is a schematic diagram of the filling process in the zero-padding binary network convolution operation;

Fig. 2 is the overall flow chart of the non-zero padding training method for binary convolutional neural network according to the present invention;

3 is a process diagram of the construction of a joint loss function in the non-zero padding training method for binary convolutional neural networks according to the present invention;

Fig. 4 is the progressive training process diagram in the non-zero padding training method used for binary convolutional neural network of the present invention;

FIG. 5 is the training and testing curves of the training method of the present invention and the common training method on the CIFAR100 test set.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings.

The present invention is used for the non-zero padding training method of the binary convolutional neural network, as shown in Figure 1, and the specific method includes the following steps:

Step 1: Prepare training samples to be applied to a specific vision task.

The invention takes the image classification task as the training task, and takes the training set in the CIFAR-100 data set as the training sample. There are 100 categories of training samples, 500 images in each category, and the total number of samples is 50,000, of which the ith training sample can be represented as ( _xi , _yi ), where: x _i represents image data, y _i represents The category label corresponding to this image. The image data of each sample is randomly cropped and flipped during training to perform data augmentation and alleviate the problem of network overfitting.

Step 2: And initialize.

First, a full-precision CNN structure with L-layer convolution is defined in the form of triples: <I, W, Conv>. Among them, I and W are a collection of tensors, both of which are 32-bit floating-point numbers, that is, full-precision values. Let I1 represent the input feature map (or activation) of the _lth layer; W1 represent the convolution kernel of the _lth layer; then the operator _Conv represents the volume between the input feature map _I1 and all convolution kernels W1 in the same layer accumulation operation.

To construct a binary convolutional neural network, the full-precision feature map and convolution kernel need to be binarized using a symbolic function, which is specifically expressed as:

与

表示二值化后的特征图与卷积核，对于任意输入a，符号函数sign(·)的表达式为：in,

and

Represents the feature map and convolution kernel after binarization. For any input a, the expression of the sign function sign( ) is:

进行填补操作，可以以下式描述：When performing convolution operations, due to some network structure needs, it is necessary to

The filling operation can be described as follows:

即为填补数值value后的特征图。in,

That is, the feature map after filling the numerical value.

中value值设置为0。After the construction of the binary convolutional neural network is completed, the network is initialized, which is specifically described as follows: the weight of the binary convolutional neural network constructed by initializing the weight of the binary network with the pre-trained padding (padding) of 0 is filled. time formula

The value value is set to 0.

Step 3: Construct the joint loss function.

Based on the knowledge distillation theory, the loss function in the network training process is improved by a joint training method, and the network with 0 padding is implemented to guide the training of the network with 1 padding, so as to reduce the gap of classification accuracy between the two. The construction process of the joint loss function of the present invention is shown in Figure 3. The input image is input into the non-zero padding and zero padding binary networks, wherein the non-zero padding network is processed by quantization, non-zero padding binary convolution, and batch normalization. It is stacked with the basic module composed of activation, and the zero-padding network is composed of quantization, zero-padding binary convolution, batch normalization, and activation. Losses based on knowledge distillation theory. At the same time, the loss based on the hard target is obtained by calculating the cross-entropy loss between the output of the non-zero padding binary network and the real image annotation (hard target). The weighted summation of the cross-entropy loss based on the knowledge distillation theory and the cross-entropy loss based on the hard target can obtain the total loss in the training process, which is the joint loss function constructed by the present invention.

For the loss function based on knowledge distillation theory, the specific description is that the pre-trained binary network filled with 0 is used as the teacher network, the binary network to be trained with the filling of 1 is used as the student network, and the input images are input into the two networks respectively, The output of the teacher network is used as a soft target to calculate the cross entropy loss with the output of the student network. The specific calculation formula is:

Among them, x _i is the image data in the ith training sample, W _s is the weight parameter of the student network, L _st ( _xi ; W _s ) is the cross-entropy loss between the output of the student network and the teacher network, and n is the classification The total number of categories of tasks, S _j and T _j correspond to the jth output of the student network and the teacher network respectively, and the calculation formula of the total output S of the student network is S=F _s (x; W _s ), that is, the input image is input The output generated in the student network F _s with the parameter W _s ; the calculation formula of the total output T of the teacher network is T=F _t (x; W _t ), that is, the input picture is input to the teacher network F with the parameter W _t The output generated in _t can also be called a soft target.

For the joint loss function in the network training process, its specific calculation formula is:

L(x; W _s )=L _s (x; W _s )+θL _st (x; W _s )

The left side of the equation is the improved joint loss function of this method. In the right side of the equation, L _s (x; W _s ) is the cross loss between the student network and the hard target, and θ is the weighting coefficient of the distillation loss. It shows that the more training depends on the contribution of the teacher network, it helps the student network to identify simple samples more easily, but if it is too large, it will reduce the impact of real annotations, making it difficult for the network to identify difficult samples. Make joint training work better.

Step 4: Perform progressive network training.

Considering that in some quantization-related works, gradually quantizing the network or gradually reducing the number of bits represented by the network can play a positive role in the training of low-bit networks, this method fills the binary network with non-zero values in step 3 according to this idea. Based on the initialized student network (that is, the feature maps of all channels are initially filled with 0), the number of feature map channels filled with 1 in the network is gradually increased. The progressive network training process of the present invention is shown in Figure 4, where c represents the ratio of feature map channels filled with 1 in the training process, epoch is defined as the number of iterations of the current training process for the data set, and c will increase with the increase of epoch during training. Gradually increase, its function can be specifically described as:

c=tanh(epoch/50)

where for any input a, the output produced by the tanh function can be expressed as:

Under a certain number of iterations, the number of feature map channels padded to 1 by the binary network is expressed as:

cnum=[c ₁ ×c]

c ₁ is the total number of channels of the feature map, [ ] is the rounding operation. When the number of iterations is 0, that is, the initial moment, cnum=0, which means that the network is actually zero-filled binary network. As the number of iterations increases, cnum gradually increases, and the network gradually evolves to a non-zero filled binary network. When the number of iterations is a certain large value, the network eventually evolves into a completely non-zero filled binary network. Table 1 shows a convolutional layer with a total number of input feature map channels c ₁ =5, the feature map channel ratio c padded to 1, the number cnum, and the features padded to 0 with the change of the number of iterations of the dataset epoch number of pictures.

Table 1 Changes in cnum0

Step 5: Use the finally obtained non-zero filled binary deep network for the test set of the task to test its classification effect.

Figure 5 shows the training and test accuracy curves of using the method described in the present invention to train a non-zero-padded binary network, and the training method of the present invention and directly training a non-zero-padded binary network (with zero-padded weights of the binary network parameters to initialize) the training and test curves on the CIFAR100 test set, wherein train1 and test1 refer to the training method of the present invention, train2 and test2 refer to the ordinary training method, and the ordinate accuracy is the precision value, it can be seen that this method effectively maintains The performance of the zero-padded binary network can be improved, the test results of the non-zero-padded binary network can be improved, and the performance degradation of the non-zero-padded binary network caused by the correction of the padded value can be compensated.

Claims (6)

1. a non-zero padding training method for binary convolutional neural network, is characterized in that: specifically comprise the steps: Step 1: Prepare a training dataset for vision tasks; Step 2: Build a deep convolutional neural network for vision tasks, binarize the weights and activations in the network into 1-bit values, and initialize it with the pre-trained zero-padding binary network weights with 0 padding; Step 3: Use knowledge distillation theory to construct a joint loss function for non-zero filled binary network; Step 4: On the training data set, the non-zero-padded binary network is gradually trained to gradually increase the number of channels filled with 1; Step 5: Use the completely non-zero filled binary network obtained in Step 4 for the test set of the task to test its classification effect. 2. a kind of non-zero padding training method for binary convolutional neural network as claimed in claim 1, is characterized in that: the concrete method of constructing binary convolutional neural network in step 2 is: First, a full-precision CNN structure with L-layer convolution is defined in the form of triples: <I, W, Conv>; where I and W are sets of tensors; let I _l represent the input feature map of the lth layer; W _l represents the convolution kernel of the _lth layer; then the operator Conv represents the convolution operation between the input feature map _I1 and all the convolution kernels W1 of the same layer; To construct a binary convolutional neural network, the full-precision feature map and convolution kernel need to be binarized using a symbolic function, which is specifically expressed as:

in,

and

Represents the feature map and convolution kernel after binarization. For any input a, the expression of the sign function sign( ) is:

When performing convolution operations, due to some network structure needs, it is necessary to

The filling operation is performed, described by the following formula:

in,

That is, the feature map after filling the numerical value. 3. a kind of non-zero padding training method for binary convolutional neural network as claimed in claim 1, is characterized in that: in step 2, with pre-trained padding as 0 zero padding binary network weight initialization constructed by The weights of the binary convolutional neural network. 4. a kind of non-zero filling training method for binary convolutional neural network as claimed in claim 1 is characterized in that: in step 2, the concrete method for the joint loss function of non-zero filling binary network is: The pre-trained binary network filled with 0 is used as the teacher network, and the binary network to be trained with the filling of 1 is used as the student network. The cross entropy loss of , the specific calculation formula is:

Among them, x _i is the image data in the ith training sample, W _s is the weight parameter of the student network, L _st ( _xi ; W _s ) is the cross-entropy loss between the output of the student network and the teacher network, and n is the classification The total number of categories of tasks, S _j and T _j correspond to the jth output of the student network and the teacher network respectively, and the calculation formula of the total output S of the student network is S=F _s (x; W _s ), that is, the input image is input The output generated in the student network F _s with the parameter W _s ; the calculation formula of the total output T of the teacher network is T=F _t (x; W _t ), that is, the input picture is input to the teacher network F with the parameter W _t The output generated in _t is called the soft target; For the joint loss function in the network training process, its specific calculation formula is: L(x; W _s )=L _s (x; W _s )+θL _st (x; W _s ) The left side of the equation is the improved joint loss function of this method. In the right side of the equation, L _s (x; W _s ) is the cross loss between the student network and the hard target, and θ is the weighting coefficient of the distillation loss. 5. A non-zero padding training method for a binary convolutional neural network according to claim 4, wherein the design θ value is 0.25. 6. a kind of non-zero padding training method for binary convolutional neural network as claimed in claim 4, is characterized in that: in step 4, progressive training method is: Let c represent the ratio of feature map channels filled with 1 during the training process, and epoch is the number of iterations of the current training process for the dataset. During training, c will gradually increase with the increase of epoch. Its function can be specifically described as: c=tanh(epoch/50) where for any input a, the output produced by the tanh function can be expressed as:

Under a certain number of iterations, the number of feature map channels padded to 1 by the binary network is expressed as: cnum=[c ₁ ×c] c ₁ is the total number of channels of the feature map, [ ] is the rounding operation. When the number of iterations is 0, that is, the initial moment, cnum=0, which means that the network actually fills the binary network with zero. Evolves into a completely non-zero filled binary network.

Images