CN114139678A - Convolutional neural network quantization method and device, electronic equipment and storage medium - Google Patents
Convolutional neural network quantization method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114139678A CN114139678A CN202111265219.0A CN202111265219A CN114139678A CN 114139678 A CN114139678 A CN 114139678A CN 202111265219 A CN202111265219 A CN 202111265219A CN 114139678 A CN114139678 A CN 114139678A
- Authority
- CN
- China
- Prior art keywords
- quantization
- weight
- layer
- neural network
- convolutional neural
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013139 quantization Methods 0.000 title claims abstract description 440
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 176
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000012549 training Methods 0.000 claims abstract description 39
- 238000012545 processing Methods 0.000 claims abstract description 30
- 210000002569 neuron Anatomy 0.000 claims description 45
- 230000008569 process Effects 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 101100129590 Schizosaccharomyces pombe (strain 972 / ATCC 24843) mcp5 gene Proteins 0.000 claims description 5
- 238000005265 energy consumption Methods 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 15
- 238000004891 communication Methods 0.000 description 5
- 230000011218 segmentation Effects 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a convolutional neural network quantization method, a convolutional neural network quantization device, electronic equipment and a storage medium, wherein the method comprises the following steps: respectively acquiring a first quantization weight range of each layer in a pre-training convolutional neural network according to a preset important weight priority principle; the important weight priority principle comprises a principle of determining the frequency of occurrence of power exponents based on the original weight of each layer; respectively carrying out weight quantity expansion processing on the first quantization weight range to obtain a second quantization weight range of each layer; determining a target quantization weight range of the pre-trained convolutional neural network based on the second quantization weight range. The invention can realize the purposes of increasing the quantization range and reducing the precision loss under the condition of the same quantization bit width, not only can reduce the storage space of a network model, reduce the calculated amount and reduce the energy consumption, but also can greatly improve the quantization precision.
Description
Technical Field
The invention relates to the technical field of machine learning, in particular to a convolutional neural network quantization method and device, electronic equipment and a storage medium.
Background
Convolutional Neural Networks (CNNs) are a type of feed-forward Neural Network that includes Convolutional calculation and has a deep structure, are one of the representative algorithms for deep learning, and have been well applied to the aspects of image classification, target detection, semantic segmentation, and the like. However, as the number of CNNs increases, the accuracy of CNNs is improved, and meanwhile, the resource consumption of CNNs such as computation, energy, and storage is also greatly increased, and thus, the method is not suitable for embedded application scenarios with limited resources, such as mobile phones and wearable devices.
The idea of the conventional Incremental Network Quantization (INQ) method is to convert an arbitrarily given pre-trained full-precision floating point CNN model into a lossless low-precision model with a weight of 2 to the power or 0 through three steps of parameter grouping, grouping Quantization and retraining, so that the purpose of simplifying operation is achieved by using shift operation instead of multiplication operation.
However, in the conventional INQ method, when determining the weight quantization range, the quantization range is selected by using the maximum weight of each layer as a reference and one bit is used to represent the weight 0, which not only reduces the quantization weight representation range, but also causes the weight quantization range to deviate when the maximum weight is far away from other weights, thereby causing a precision loss.
Disclosure of Invention
The invention provides a convolutional neural network quantization method, a convolutional neural network quantization device, electronic equipment and a storage medium, which are used for overcoming the defects of precision loss caused by the fact that an INQ method in the prior art selects a quantization range by taking the maximum weight of each layer as a reference basis and independently adopts one bit to represent 0 weight, and achieving the purposes of improving the CNN quantization precision, reducing the CNN quantization error and reducing the storage space, the calculated amount and the energy consumption.
The invention provides a convolutional neural network quantization method, which comprises the following steps:
respectively acquiring a first quantization weight range of each layer in a pre-training convolutional neural network according to a preset important weight priority principle; the important weight priority principle comprises a principle of determining the frequency of occurrence of power exponents based on the original weight of each layer;
respectively carrying out weight quantity expansion processing on the first quantization weight range to obtain a second quantization weight range of each layer;
determining a target quantization weight range of the pre-trained convolutional neural network based on the second quantization weight range.
According to the quantization method of the convolutional neural network provided by the invention, the first quantization weight range of each layer in the pre-trained convolutional neural network is respectively obtained according to a preset important weight priority principle, and the method comprises the following steps:
respectively setting a quantization weight value range and a quantization bit width of each layer in a pre-training convolutional neural network; the quantization weight value domain comprises a plurality of quantization weights, the power base numbers of the absolute values of the quantization weights are respectively the same, the power indexes are different, and the power indexes of the quantization weights at the end points in the quantization weight value domain are unknown;
respectively acquiring a first power exponent with the highest frequency of occurrence in the original weight of each layer and a second power exponent with the highest frequency of occurrence adjacent to the first power exponent according to a preset important weight priority principle;
calculating a first endpoint power exponent of the quantization weights at the endpoints in the quantization weight value domain based on the first power exponent, the second power exponent and the quantization bit width, respectively;
and respectively determining a first quantization weight range of each layer based on a first quantization weight value range corresponding to the first endpoint power exponent.
According to the convolutional neural network quantization method provided by the present invention, the performing weight number expansion processing on the first quantization weight range to obtain the second quantization weight range of each layer includes:
based on the principle of weight 0 removal, the first endpoint power exponents are respectively adjusted to obtain second endpoint power exponents;
and determining a second quantization weight range of each layer based on a second quantization weight value range corresponding to the second endpoint power exponent respectively.
According to a quantization method of a convolutional neural network provided by the present invention, the determining a target quantization weight range of the pre-trained convolutional neural network based on the second quantization weight range includes:
and carrying out unified quantization processing on the second quantization weight range of each layer to obtain a target quantization weight range of the pre-training convolutional neural network.
According to the quantization method of the convolutional neural network provided by the invention, according to a preset important weight priority principle, a first power exponent with the highest occurrence frequency and a second power exponent with the highest occurrence frequency and close to the first power exponent in the original weight of each layer are respectively determined, and the method comprises the following steps:
determining the power exponent S of the original weight of each layer respectivelylAnd respectively determining the first power exponent maximum with the highest frequency in the original weight of each layer according to a preset important weight priority principlelAnd with the first power exponent maxinumlThe next highest power index 1lWherein:
Sl＝floor(log2(4abs(Wl)/3))
maxinuml＝Max_num1(Sl)
maxinum1l＝Max_neighboor(Sl，maxinuml)
wherein floor (. cndot.) represents rounding-down, abs (. cndot.) represents absolute value, WlRepresenting the original weights of the l-th layer of the pre-trained convolutional neural network, SlThe power exponent of the original weight of the l-th layer of the pre-trained convolutional neural network is shown, Max _ num1(·) shows the value with the highest frequency of occurrence is taken, Max _ neighbor (S)l，maxinuml) Is shown at SlTaking out the herb and maxinumlAdjacent to and most frequently occurring value of (a), maxinumlRepresents the original weight W of the l-th layerlThe highest frequency of occurrence of the first power exponent, maxinum1lRepresents the sum of maxinumlAnd L is more than or equal to 1 and less than or equal to L, wherein L represents the total number of learnable layers in the pre-training convolutional neural network.
According to the convolutional neural network quantization method provided by the present invention, the first endpoint power exponent of the quantization weight at the endpoint in the quantization weight value domain is calculated based on the first power exponent, the second power exponent and the quantization bit width, respectively, and the calculation formula includes:
correspondingly, the first quantization weight range of each layer is respectively determined based on the first quantization weight value range corresponding to the first endpoint power exponent, wherein the determination process of each first quantization weight of each layer comprises the following steps:
wherein sgn (x) is a sign function,
b denotes the quantization bit width, nl1Maximum first-endpoint power exponent, n, representing quantization weight at endpoint in quantization weight value domain of layer ll2Minimum first endpoint power exponent, n, representing quantization weight at endpoint in quantization weight value domain of layer ll1≥nl2And n isl1And nl2Are all integers; wl(i, j) represents the original weight between the ith and jth neurons of the l-th layer of the pre-trained convolutional neural network, αlAnd betalTwo adjacent first quantization weights, beta, after the absolute value of the first quantization weight value range of the first layer is orderedl＞αl，
Representing a first quantization weight between the ith neuron and the jth neuron of the ith layer, I is more than or equal to 1 and less than or equal to I, I represents the total number of neurons input to the ith layer of the pre-trained convolutional neural network, J is more than or equal to 1 and less than or equal to J, and J represents the number of the neurons input to the jth layer of the pre-trained convolutional neural networkTotal number of neurons outputted from layer i.
According to the convolutional neural network quantization method provided by the invention, based on the weight 0-removing principle, the first endpoint power exponents are respectively adjusted to obtain second endpoint power exponents, and the adjustment processing formula comprises;
correspondingly, the second quantization weight range of each layer is determined based on the second quantization weight value range corresponding to the second endpoint power exponent, wherein the determination process of each second quantization weight of each layer comprises:
wherein sgn (x) is a sign function,
a maximum second endpoint power exponent representing the quantization weight at the endpoint in the quantization weight value domain of the l-th layer,
a minimum second endpoint power exponent representing the quantization weight at the endpoint in the quantization weight value domain of the l-th layer,
and
two second quantization weights adjacent to each other after the absolute value of the second quantization weight value range of the l-th layer is sorted,
Wl(i, j) represents the original weight between the ith neuron and the jth neuron of the l-th layer of the pre-trained convolutional neural network,
representing second quantization weight between the ith neuron and the jth neuron of the ith layer, I is more than or equal to 1 and less than or equal to I, I represents the total number of neurons input to the ith layer of the pre-trained convolutional neural network, J is more than or equal to 1 and less than or equal to J, J represents the total number of neurons output from the ith layer of the pre-trained convolutional neural network, maxinumlRepresents the original weight W of the l-th layerlThe highest frequency of occurrence of the first power exponent, maxinum1lRepresents the sum of maxinumlAnd L is more than or equal to 1 and less than or equal to L, wherein L represents the total number of learnable layers in the pre-training convolutional neural network.
The present invention also provides a convolutional neural network quantization apparatus, comprising:
the acquisition module is used for respectively determining a first quantization weight range of each layer in the pre-training convolutional neural network according to a preset important weight priority principle; the important weight priority principle comprises a principle of determining the frequency of occurrence of power exponents based on the original weight of each layer;
the processing module is used for respectively carrying out weight quantity expansion processing on the first quantization weight range to obtain a second quantization weight range of each layer;
a determining module to determine a target quantization weight range of the pre-trained convolutional neural network based on the second quantization weight range.
The present invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the convolutional neural network quantization method as described in any of the above when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the convolutional neural network quantization method as described in any one of the above.
The invention provides a convolutional neural network quantization method, a convolutional neural network quantization device, electronic equipment and a storage medium, wherein the convolutional neural network quantization method comprises the steps of firstly respectively obtaining a first quantization weight range of each layer in a pre-trained convolutional neural network according to a preset important weight priority principle, then respectively carrying out weight quantity expansion processing on the first quantization weight of each layer to obtain a second quantization weight range of each layer, and finally determining a target quantization weight range of the pre-trained convolutional neural network based on the second quantization weight range of each layer; for the convolution neural network quantization, a frequency diagram of the frequency of occurrence of the power exponent corresponding to the weight absolute value is a bell-shaped distribution diagram, that is, when the difference between the frequency of occurrence in the bell-shaped distribution diagram is small, the difference between the corresponding weights is also small, therefore, when the important weight priority principle is a principle that the frequency of occurrence of the power exponent of each layer of weights is determined, a first quantization weight range with small difference between the quantization weights in each layer can be obtained, and further, the purposes of increasing the quantization range and reducing the precision loss under the condition of the same quantization bit width are achieved by performing weight quantity expansion processing on the first quantization weight range of each layer, so that not only can the storage space of a network model be reduced, the calculation quantity be reduced, the energy consumption be reduced, but also the quantization precision can be greatly improved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a convolutional neural network quantization method provided by the present invention;
FIG. 2 is a distribution diagram of the frequency of occurrence of the power exponent corresponding to the weight in the Tiny-yolov3 convolutional layer according to the present invention;
FIG. 3 is a schematic diagram of a process for determining positive first quantization weights in a first quantization weight range according to the present invention;
FIG. 4 is a diagram illustrating a quantization process for determining each of the first quantization weights provided by the present invention;
FIG. 5 is a schematic structural diagram of a convolutional neural network quantization apparatus provided in the present invention;
fig. 6 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In recent years, CNN obtains results superior to traditional algorithms on computer vision tasks such as image classification, target detection and semantic segmentation. The accuracy of the convolutional neural network can be improved with the increase of the number of layers, but the resource consumption of calculation, energy, storage and the like is greatly increased, so that the convolutional neural network is not suitable for embedded application scenes with limited resources such as mobile phones and wearable equipment. Therefore, how to compress the network under the condition that the accuracy of the full-precision convolutional neural network model is unchanged or is reduced less, so as to reduce the storage space of the network model, reduce the calculation amount and reduce the energy consumption, is a problem to be solved urgently.
The proposal of the INQ method brings hope for the convolution neural network model to be deployed on hardware. The idea of INQ is to convert any given one of the pre-trained full-precision (i.e., 32-bit floating point) CNN models into a lossless low-precision model with a weight of 2 to the power of 2 or 0 through three steps of parameter grouping, grouping quantization, and retraining, thereby replacing the multiplication operation with a shift operation. However, in the INQ method, when determining the weight quantization range, the quantization range is selected by using the maximum weight of each layer as a reference, and when the maximum weight is far away from other weights, the weight quantization range is deviated, thereby causing accuracy loss; meanwhile, in the INQ method, a single bit is used to represent the weight 0, which may reduce the representation range of the quantization weight and cause the loss of precision.
Based on the above problems, the present invention provides a convolutional neural Network Quantization method, that is, a Total Incremental Network Quantization (TINQ) method obtained by improving an INQ method, where an execution main body of the convolutional neural Network Quantization method may be a terminal device having functions of target detection and segmentation, image classification, semantic segmentation, and the like, and the terminal device may be a mobile phone, a wearable device, and other embedded devices.
Fig. 1 is a diagram illustrating a convolutional neural network quantization method according to the present invention, as shown in fig. 1, the convolutional neural network quantization method includes:
step S110, respectively obtaining a first quantization weight range of each layer in the pre-training convolutional neural network according to a preset important weight priority principle.
The important weight priority principle may include a principle of determining the frequency of occurrence of power exponents based on the original weight of each layer; the first quantization weight range may include a plurality of first quantization weights and 0 weight, and the power base numbers of the absolute values of the first quantization weights are respectively the same and the power exponents are different.
Specifically, the pretrained convolutional neural network may be a pretrained full-precision floating point CNN, such as a pretrained 32-bit floating point CNN, the pretrained convolutional neural network may include L learnable layers, each learnable layer may include each convolutional layer and a full-link layer, and an original weight of the L-th layer of the pretrained convolutional neural network may be represented as WlL is not less than 1 and not more than L, and the original weight W of the L-th layerlThe second layer is a high-dimensional tensor, for example, a four-dimensional tensor which may be C × K × R × S, where C represents the total number of output channels of the first layer, K represents the total number of input channels of the first layer, and R × S represents the dimensional information of the first layer.
Since, for the pre-trained 32-bit floating point number CNN, the histogram of the frequency of occurrence of the power exponent corresponding to the absolute value of the original weight thereof has a certain distribution characteristic, if exemplified by Tiny-yolov3 trained on the voc data set, the histogram as shown in fig. 2 can be obtained, in fig. 2, (a) to (m) respectively represent the histogram of the frequency of occurrence of the power exponent corresponding to the original weights of the 1 st to 13 th layers of convolution layers in the pre-trained 32-bit floating point number CNN, and (n) represent the histogram of the frequency of occurrence of the power exponent corresponding to the original weights of the 13 th layers of convolution layers in the pre-trained 32-bit floating point number CNN, in which the abscissa represents the power exponent corresponding to the original weight of the convolution layer, and the ordinate represents the frequency of occurrence of the power exponent corresponding to the original weight of the convolution layer in the pre-trained 32-bit floating point number CNN.
As can be seen from fig. 2, the distribution diagram of the frequency of occurrence of the power exponents corresponding to the absolute values of the original weights of the pre-trained 32-bit floating point numbers CNN is substantially bell-shaped, the frequency of occurrence of the middle power exponents is high, the frequency of occurrence of the power exponents of the maximum value and the minimum value in the original weights is low, and the power exponents with the most frequency of occurrence in each layer are substantially close to each other, so that the original weights corresponding to the power exponents are also distributed in this feature.
Therefore, based on the bell-shaped distribution characteristics of the distribution diagram, an important weight priority principle can be set based on the power exponent occurrence frequency of each layer of original weight, and the important weight priority principle is stored in the terminal equipment, so that when a pre-training convolutional neural network with the same property as that of the pre-training full-precision floating point CNN is obtained subsequently, the first quantization weight range of each layer in the pre-training convolutional neural network can be directly obtained according to the important weight priority principle.
Step S120, performing weight quantity expansion processing on the first quantization weight ranges respectively to obtain second quantization weight ranges of each layer.
Specifically, since the first quantization weight range includes a plurality of first quantization weights and 0 weight, that is, when one bit is used to represent 0 weight, the quantization representation range is reduced greatly, for example, when the quantization bit width is b, the first quantization weight range includes at most 2b-1+1 first quantization weights. Therefore, in order to increase the quantization expression range under the same bit width, the first quantization weight range may be respectively subjected to weight number expansion processing toThis enables the number of quantization weights in the quantization weight range to be up to 2bAnd thus a second quantization weight range for each layer. The second quantization weight range may include a plurality of second quantization weights and may be symmetrically distributed on the coordinate axis, for example, the plurality of second quantization weights may be composed of positive second quantization weights and negative quantization weights, and the number of the positive second quantization weights and the number of the negative second quantization weights are equal and correspond to each other.
And S130, determining a target quantization weight range of the pre-training convolutional neural network based on the second quantization weight range.
Specifically, when the terminal device obtains the second quantization weight range of each layer, the terminal device may compare and statistically analyze the second quantization weight ranges of each layer, where the second quantization weights with the highest frequency of occurrence in each layer are all within the same quantization range, for example, as shown in (n) in fig. 2, which is obtained by taking Tiny-yolov3 as an example, the second quantization weights with the highest frequency of occurrence in 13 convolutional layers are all [2 ]-5,2-10]Therefore, the second quantization weight range of each layer can be uniformly quantized, that is, the second quantization weight range of each layer is uniformly quantized into a target quantization weight range of the pre-trained convolutional neural network, and the process may include:
the first power exponent maximum appearing frequently in the original weight of each layerlAs the first target power exponent maximum which appears most frequently in the pre-training convolutional neural network, and the original weight of each layer is compared with the first power exponent maximumlThe next highest power index 1lAs the second target power exponent maxinum1 with the highest frequency of occurrence in the pre-trained convolutional neural network immediately adjacent to the first target power exponent maxinum. As shown in (n) of fig. 2, when maxinum is-7 and maxinum1 is-6, and when quantization bit width b is 4, a target end point power exponent n of the target quantization weight at the end point of the target quantization weight range of the pre-trained convolutional neural network can be obtained1＝-3，n2-10, the calculation formula comprises:
the target quantization weight range from which the pre-trained convolutional neural network can be derived may be [ -2 [-3,-2-10]And [2 ]-10，2-3]The target quantization weight value range of the pre-trained convolutional neural network may be p { ± 2 { (2) { (m {)-10,…,±2-3}. Therefore, the target quantization weight range can fully cover the target quantization weights with high frequency in the whole pre-training convolutional neural network (namely all the target quantization weights which are important for the pre-training convolutional neural network).
The invention provides a convolutional neural network quantization method, a convolutional neural network quantization device, electronic equipment and a storage medium, wherein the convolutional neural network quantization method comprises the steps of firstly respectively obtaining a first quantization weight range of each layer in a pre-trained convolutional neural network according to a preset important weight priority principle, then respectively carrying out weight quantity expansion processing on the first quantization weight of each layer to obtain a second quantization weight range of each layer, and finally determining a target quantization weight range of the pre-trained convolutional neural network based on the second quantization weight range of each layer; for the convolutional layer, a frequency diagram of the frequency of occurrence of the power exponent corresponding to the weight absolute value is a bell-shaped distribution diagram, that is, when the difference between the frequency of occurrence in the bell-shaped distribution diagram is small, the difference between the corresponding weights is also small, so when the important weight priority principle is a principle that the frequency of occurrence of the power exponent of each layer of weights is determined, a first quantization weight range with small difference between the quantization weights in each layer can be obtained, and further, by means of performing weight quantity expansion processing on the first quantization weight range of each layer, the purposes of increasing the quantization range and reducing precision loss under the condition of the same quantization bit width can be achieved, and not only can the storage space of a network model be reduced, the calculation amount be reduced, the energy consumption be reduced, but also the quantization precision can be greatly improved.
Optionally, the specific implementation process of step S110 may include:
firstly, respectively setting a quantization weight value range and a quantization bit width of each layer in a pre-training convolutional neural network; then according to a preset important weight priority principle, respectively acquiring a first power exponent with the highest occurrence frequency and a second power exponent with the highest occurrence frequency adjacent to the first power exponent in the original weight of each layer; then, based on the first power exponent, the second power exponent and the quantization bit width, respectively calculating a first endpoint power exponent of the quantization weights at endpoints in the quantization weight value domain; and finally, respectively determining a first quantization weight range of each layer from a first quantization weight value domain corresponding to the first endpoint power exponent.
The quantization weight value domain comprises a plurality of quantization weights, the power base numbers of the absolute values of the quantization weights are respectively the same, the power exponents are different, and the power exponents of the quantization weights at the end points in the quantization weight value domain are unknown.
Specifically, for the pre-training convolutional neural network, the terminal device may first obtain the original weight W of each layer of the pre-training convolutional neural networklL is more than or equal to 1 and less than or equal to L, L represents the total number of learnable layers in the pre-training convolutional neural network, and the quantization weight value range and the quantization bit width of each layer of the pre-training convolutional neural network can be set based on the goal of quantizing the original weight of each layer of the pre-training convolutional neural network into low-precision quantization weight, wherein the quantization weight value range of each layer can be set to be
nl1≥nl2，nl1≤0，nl1And nl2Are all integers; the quantization bit width can be set as b, and the low-precision quantization weight of each layer is from a quantization weight value range Pl. Thus, n isl1And nl2Assisting in quantizing a weight value field PlDefining the quantization weight range of each layer, and constraining the quantization weight at
Or
And (4) the following steps.
Then, the terminal device may determine the power exponent S of the original weight of each layer as followslAnd respectively determining the first power exponent most frequently appearing in the original weight of each layer according to a preset important weight priority principlelAnd with the first power exponent maxinumlThe next highest power index 1lWherein:
Sl＝floor(log2(4abs(Wl)/3)) (2)
maxinuml＝Max_num1(Sl) (3)
maxinum1l＝Max_neighboor(Sl，maxinuml) (4)
wherein floor (. cndot.) represents rounding-down, abs (. cndot.) represents absolute value, WlRepresenting the original weights of the l-th layer of the pre-trained convolutional neural network, SlThe power exponent of the original weight of the l-th layer of the pre-trained convolutional neural network is shown, Max _ num1(·) shows the value with the highest frequency of occurrence is taken, Max _ neighbor (S)l，maxinuml) Is shown at SlTaking out the herb and maxinumlAdjacent to and most frequently occurring value of (a), maxinumlRepresents the original weight W of the l-th layerlThe highest frequency of occurrence of the first power exponent, maxinum1lRepresents the sum of maxinumlAnd L is more than or equal to 1 and less than or equal to L, wherein L represents the total number of learnable layers in the pre-training convolutional neural network.
In addition, since the quantization weight range includes 2 excluding 0 weightb-1Quantized weight, and 2b-1The quantization weight comprises 2b-12 positive quantization weights and 2b-1The/2 negative quantization weights, i.e. the number of quantization weights in the quantization weight range, is an even number. Therefore, in order to more uniformly select the quantization weight with the highest frequency of occurrence in the pre-training convolutional neural network, the first power exponent maximum with the highest frequency of occurrence in each layer can be obtainedlThen, the most frequent occurrence frequency of the adjacent frequency is selectedHigh second power exponent maxinum1lTherefore, the purpose of quickly and efficiently acquiring the first quantization weight range of each layer can be achieved.
Finally, the terminal device is further based on the first power exponent maximumlSecond power exponent maxinum1lAnd a quantization bit width b, calculating a quantization weight value field PlThe first endpoint power exponent of the quantified weight at the endpoint in (1) is calculated as:
in obtaining the quantization weight value field PlThe first endpoint power exponent n of the quantized weight at endpoint in (1)l1And nl2Then, the first endpoint power exponent n can be obtained correspondinglyl1And nl2Corresponding first quantization weight value range Pl1，
Based on the first quantization weight value range Pl1A first quantization weight range for each layer is determined. Such as a first quantization weight value field Pl1Is composed of
The first quantization weight range may be
And
therefore, it can be considered that when the first quantization weight range of each layer is determined, the first quantization weight range of each layer is determined.
The absolute value of each original weight in the pre-trained convolutional neural network can be obtained, then the first quantization weight of each original weight after the absolute value is calculated based on the formula (6), then the positive and negative of each original weight are further judged according to the property of a sign function, and the positive and negative attributes of each original weight are based, and the positive and negative signs are correspondingly added before each first quantization weight, so that the first quantization weight of each original weight is obtained; and finally, obtaining each first quantization weight of each layer according to the position relation of each original weight in the pre-training convolutional neural network, wherein the determination process of each first quantization weight of each layer comprises the following steps:
wherein the sign function may be sgn (x),
floor (. cndot.) denotes rounding-down, abs (. cndot.) denotes absolute value, WlRepresenting the original weights of the l-th layer of the pre-trained convolutional neural network, SlThe power exponent of the original weight of the l-th layer of the pre-trained convolutional neural network is shown, Max _ num1(·) shows the value with the highest frequency of occurrence is taken, Max _ neighbor (S)l，maxinuml) Is shown at SlTaking out the herb and maxinumlAdjacent to and most frequently occurring value of (a), maxinumlRepresents the original weight W of the l-th layerlThe highest frequency of occurrence of the first power exponent, maxinum1lRepresents the sum of maxinumlThe second power exponent with the highest frequency of occurrence is adjacent, L is more than or equal to 1 and less than or equal to L, L represents the total number of layers which can be learnt in the pre-training convolutional neural network, b represents the quantization bit width, nl1Maximum first-endpoint power exponent, n, representing quantization weight at endpoint in quantization weight value domain of layer ll2Minimum first endpoint power exponent, n, representing quantization weight at endpoint in quantization weight value domain of layer ll1≥nl2And n isl1And nl2Are all integers; wl(i, j) represents the original weight between the ith and jth neurons of the l-th layer of the pre-trained convolutional neural network, αlAnd betalTwo first quantization weights, β, representing the first quantization weight range of the l-th layer after the absolute value orderingl＞αl，
And expressing a first quantization weight between the ith neuron and the jth neuron of the ith layer, wherein I is more than or equal to 1 and less than or equal to I, I represents the total number of neurons input to the ith layer of the pre-trained convolutional neural network, J is more than or equal to 1 and less than or equal to J, and J represents the total number of neurons output from the ith layer of the pre-trained convolutional neural network.
It should be noted that the quantization bit width b and the 0 weight each occupy one bit, and when determining the maximum of each layer according to the preset important weight priority principlelAnd maxinum1lThen, maxinum can be given separatelylAnd maxinum1lDetermining the number of values from the left side and the right side of each central point for the central point according to the quantization bit width b, thereby determining the first quantization weight range of each layer, namely, solving nl1And nl2Since the first quantization weight range of each layer includes the positive first quantization weight and the negative first quantization weight, and the positive first quantization weight and the negative first quantization weight are symmetrically distributed on the coordinate axis, the negative first quantization weight can be obtained when the positive first quantization weight is obtained. The determination of the positive first quantization weight may be as shown in fig. 3.
In addition, since the pre-trained convolutional neural network is pre-trained full-precision floating-point CNN, its quantization weight is large (e.g. 2)1) Occur less frequently and therefore n can be substitutedl1The upper limit value of (2) is set to 0, and as can be seen from equation (6), the near rule can be used as the quantization rule, and the quantization weight value range P can be quantized from the first quantization weight value range P in accordance with the near rulel1Respectively find out and Wl(i, j) similar quantization weights as the first quantization weight
The quantization weight alternative W to be foundl(i, j), the process of which can be shown in fig. 4, the data marked by the filled circles in fig. 4 represents each first quantization weight of the ith layer of the pre-trained convolutional neural network, and the data marked by the cross signs represents each original weight of the ith layer of the pre-trained convolutional neural network.
The convolutional neural network quantization method provided by the invention comprises the steps of firstly calculating a first endpoint power exponent of a quantization weight at an endpoint in a quantization weight value domain of each layer by using a first power exponent with the highest frequency of occurrence in an original weight of each layer and a second power exponent with the highest frequency of occurrence adjacent to the first power exponent, and then respectively determining a first quantization weight range of each layer from the first quantization weight value domain corresponding to the first endpoint power exponent, so that the problems of reduction of the quantization weight range and precision loss caused when the quantization weight range of each layer of a network is obtained by taking the maximum quantization weight as a standard in the traditional INQ are solved, the aim of rapidly and effectively improving the quantization precision on the premise of reducing the storage space of a network model, reducing the calculation amount and reducing the energy consumption is fulfilled, the quantization weight range is enlarged, and the quantization error is reduced.
Optionally, step S120 may be implemented by the following process:
firstly, based on the principle of weight 0 removal, respectively adjusting the first endpoint power exponent to respectively obtain second endpoint power exponents; and then determining a second quantization weight range of each layer from a second quantization weight value domain corresponding to the second endpoint power exponent.
Specifically, the terminal device may obtain, by comparing and analyzing the second quantization weight in each convolutional layer of the pre-trained convolutional neural network, that the second quantization weight with the highest frequency of occurrence in the pre-trained convolutional neural network is the same as the second quantization weight with the highest frequency of occurrence in each convolutional layer, as shown in (n) in fig. 2. Moreover, when the quantization bit width is b and one bit is used alone to represent the 0 weight, the first quantization weight range of each layer can only include 2b-1+1 first quantization weights, therefore, in order to increase the quantization weight representation range under the same bit width, 0 weight may not be used as the weight quantization result any more, and 2 may be represented at this timebThe quantization weights for which equation (5) can be optimized, thus achieving a quantization weight value range PlThe first endpoint power exponent of the quantified weight at the endpoint is adjusted, and the optimization formula or the adjustment formula is as follows:
the power exponent n to the first endpoint can be obtained based on equation (7)l1And nl2Further optimizing the results or adjusting the processing results
And
a maximum second endpoint power exponent representing the quantization weight at the endpoint in the quantization weight value domain of the l-th layer,
the minimum second endpoint power index of the quantization weight at the endpoint in the quantization weight value domain of the ith layer is expressed, and then quantization can be performed again based on the principle of proximity, namely the 0 weight in the first quantization weight range of each layer is quantized to the quantization weight nearest to the 0 weight
Therefore, the purpose of obtaining the second quantization weight range of each layer based on the weight de-0 principle is achieved.
In addition, in the quantization weight value domain PlSecond endpoint power exponent of the quantized weights at the endpoints in (1)
And
then, a second quantization weight value range corresponding to the second endpoint power exponent can be obtained correspondingly
Based on the second quantization weight value range
A second range of quantization weights for each layer is determined. Such as a second quantization weight value range
Is composed of
The second quantization weight range may be
And
therefore, it can be considered that when the second quantization weight value range of each layer is determined, the second quantization weight range of each layer is determined.
The absolute value of each original weight in the pre-trained convolutional neural network can be obtained, then a second quantization weight of each original weight after the absolute value is calculated based on the formula (8), and then the positive and negative of each original weight are further judged according to the property of a sign function, so that the positive and negative attributes of each original weight are based, and the positive and negative signs are correspondingly added before each second quantization weight, so that the second quantization weight of each original weight is obtained; and finally, obtaining each second quantization weight of each layer according to the position relation of each original weight in the pre-training convolutional neural network, wherein the determination process of each second quantization weight of each layer comprises the following steps:
wherein the sign function may be sgn (x),
a maximum second endpoint power exponent representing the quantization weight at the endpoint in the quantization weight value domain of the l-th layer,
a minimum second endpoint power exponent representing the quantization weight at the endpoint in the quantization weight value domain of the l-th layer,
and
two second quantization weights adjacent to each other after the absolute value of the second quantization weight value range of the l-th layer is sorted,
Wl(i, j) represents the original weight between the ith neuron and the jth neuron of the l-th layer of the pre-trained convolutional neural network,
representing second quantization weight between the ith neuron and the jth neuron of the ith layer, I is more than or equal to 1 and less than or equal to I, I represents the total number of neurons input to the ith layer of the pre-trained convolutional neural network, J is more than or equal to 1 and less than or equal to J, J represents the total number of neurons output from the ith layer of the pre-trained convolutional neural network, maxinumlRepresents the original weight W of the l-th layerlThe highest frequency of occurrence of the first power exponent, maxinum1lRepresents the sum of maxinumlAnd L is more than or equal to 1 and less than or equal to L, wherein L represents the total number of learnable layers in the pre-training convolutional neural network.
According to the convolutional neural network quantization method provided by the invention, the purpose of increasing the second quantization weight range of each layer under the same bit width is realized by determining the second endpoint power exponent of the quantization weight at the endpoint in the second quantization weight value domain of each layer through the weight de-0 principle, so that the storage space of the convolutional neural network is reduced, the calculated amount is reduced, the energy consumption is reduced, the precision of the quantization weight range of each layer is improved, and the quantization error is reduced.
The following describes the convolutional neural network quantization apparatus provided by the present invention, and the convolutional neural network quantization apparatus described below and the convolutional neural network quantization method described above may be referred to correspondingly.
The present invention provides a convolutional neural network quantization apparatus, as shown in fig. 5, the convolutional neural network quantization apparatus 500 includes:
an obtaining module 510, configured to determine a first quantization weight range of each layer in a pre-trained convolutional neural network according to a preset important weight priority principle; wherein the important weight priority rule comprises a rule of determining the frequency of occurrence of power exponent based on the original weight of each layer.
A processing module 520, configured to perform weight quantity expansion processing on the first quantization weight ranges respectively to obtain second quantization weight ranges of each layer.
A determining module 530 configured to determine a target quantization weight range of the pre-trained convolutional neural network based on the second quantization weight range.
Optionally, the obtaining module 510 may be specifically configured to set a quantization weight value range and a quantization bit width of each layer in the pre-training convolutional neural network respectively; the quantization weight value domain comprises a plurality of quantization weights, the power base numbers of the absolute values of the quantization weights are respectively the same, the power indexes are different, and the power indexes of the quantization weights at the end points in the quantization weight value domain are unknown; respectively acquiring a first power exponent with the highest frequency of occurrence in the original weight of each layer and a second power exponent with the highest frequency of occurrence adjacent to the first power exponent according to a preset important weight priority principle; calculating a first endpoint power exponent of the quantization weights at the endpoints in the quantization weight value domain based on the first power exponent, the second power exponent and the quantization bit width, respectively; and respectively determining a first quantization weight range of each layer based on a first quantization weight value range corresponding to the first endpoint power exponent.
Optionally, the processing module 520 may be specifically configured to perform adjustment processing on the first endpoint power exponents based on a weight-to-0 principle, so as to obtain second endpoint power exponents respectively; and determining a second quantization weight range of each layer based on a second quantization weight value range corresponding to the second endpoint power exponent respectively.
Optionally, the determining module 530 may be specifically configured to perform unified quantization processing on the second quantization weight range of each layer to obtain a target quantization weight range of the pre-trained convolutional neural network.
Optionally, the obtaining module 510 may be further configured to determine the power exponent S of the original weight of each layer respectivelylAnd respectively determining the first power exponent maximum with the highest frequency in the original weight of each layer according to a preset important weight priority principlelAnd with the first power exponent maxinumlThe next highest power index 1lWherein:
Sl＝floor(log2(4abs(Wl)/3))
maxinuml＝Max_num1(Sl)
maxinum1l＝Max_neighboor(Sl，maxinuml)
wherein floor (. cndot.) represents rounding-down, abs (. cndot.) represents absolute value, WlRepresenting the original weights of the l-th layer of the pre-trained convolutional neural network, SlThe power exponent of the original weight of the l-th layer of the pre-trained convolutional neural network is shown, Max _ num1(·) shows the value with the highest frequency of occurrence is taken, Max _ neighbor (S)l，maxinuml) Is shown at SlTaking out the herb and maxinumlAdjacent to and most frequently occurring value of (a), maxinumlRepresents the original weight W of the l-th layerlThe highest frequency of occurrence of the first power exponent, maxinum1lRepresents the sum of maxinumlAnd L is more than or equal to 1 and less than or equal to L, wherein L represents the total number of learnable layers in the pre-training convolutional neural network.
Optionally, the obtaining module 510 may be further specifically configured to calculate a first endpoint power exponent of the quantization weight at the endpoint in the quantization weight value domain, where the calculation formula includes:
correspondingly, the first quantization weight range of each layer is respectively determined based on the first quantization weight value range corresponding to the first endpoint power exponent, wherein the determination process of each first quantization weight of each layer comprises the following steps:
wherein sgn (x) is a sign function,
b denotes the quantization bit width, nl1Maximum first-endpoint power exponent, n, representing quantization weight at endpoint in quantization weight value domain of layer ll2Minimum first endpoint power exponent, n, representing quantization weight at endpoint in quantization weight value domain of layer ll1≥nl2And n isl1And nl2Are all integers; wl(i, j) represents the original weight between the ith and jth neurons of the l-th layer of the pre-trained convolutional neural network, αlAnd betalTwo adjacent first quantization weights, beta, after the absolute value of the first quantization weight value range of the first layer is orderedl＞αl，
And expressing a first quantization weight between the ith neuron and the jth neuron of the ith layer, wherein I is more than or equal to 1 and less than or equal to I, I represents the total number of neurons input to the ith layer of the pre-trained convolutional neural network, J is more than or equal to 1 and less than or equal to J, and J represents the total number of neurons output from the ith layer of the pre-trained convolutional neural network.
Optionally, the processing module 520 may be further specifically configured to perform adjustment processing on the first endpoint power exponents respectively based on a weight-to-0 principle, so as to obtain second endpoint power exponents respectively, where an adjustment processing formula includes;
correspondingly, the second quantization weight range of each layer is determined based on the second quantization weight value range corresponding to the second endpoint power exponent, wherein the determination process of each second quantization weight of each layer comprises:
wherein sgn (x) is a sign function,
a maximum second endpoint power exponent representing the quantization weight at the endpoint in the quantization weight value domain of the l-th layer,
a minimum second endpoint power exponent representing the quantization weight at the endpoint in the quantization weight value domain of the l-th layer,
and
two second quantization weights adjacent to each other after the absolute value of the second quantization weight value range of the l-th layer is sorted,
Wl(i, j) denotes the primitive between the ith neuron and the jth neuron of the l-th layer of the pretrained convolutional neural networkThe weight of the beginning of the process is weighted,
representing second quantization weight between the ith neuron and the jth neuron of the ith layer, I is more than or equal to 1 and less than or equal to I, I represents the total number of neurons input to the ith layer of the pre-trained convolutional neural network, J is more than or equal to 1 and less than or equal to J, J represents the total number of neurons output from the ith layer of the pre-trained convolutional neural network, maxinumlRepresents the original weight W of the l-th layerlThe highest frequency of occurrence of the first power exponent, maxinum1lRepresents the sum of maxinumlAnd L is more than or equal to 1 and less than or equal to L, wherein L represents the total number of learnable layers in the pre-training convolutional neural network.
Fig. 6 illustrates a physical structure diagram of an electronic device, and as shown in fig. 6, the electronic device 600 may include: a processor (processor)610, a communication interface (communication interface)620, a memory (memory)630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 are communicated with each other via the communication bus 640. The processor 610 may invoke logic instructions in the memory 630 to perform a convolutional neural network quantization method, the method comprising: respectively acquiring a first quantization weight range of each layer in a pre-training convolutional neural network according to a preset important weight priority principle; the important weight priority principle comprises a principle of determining the frequency of occurrence of power exponents based on the original weight of each layer; respectively carrying out weight quantity expansion processing on the first quantization weight range to obtain a second quantization weight range of each layer; determining a target quantization weight range of the pre-trained convolutional neural network based on the second quantization weight range.
In addition, the logic instructions in the memory 630 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, which when executed by a computer, enable the computer to perform the convolutional neural network quantization method provided by the above methods, the method comprising: respectively acquiring a first quantization weight range of each layer in a pre-training convolutional neural network according to a preset important weight priority principle; the important weight priority principle comprises a principle of determining the frequency of occurrence of power exponents based on the original weight of each layer; respectively carrying out weight quantity expansion processing on the first quantization weight range to obtain a second quantization weight range of each layer; determining a target quantization weight range of the pre-trained convolutional neural network based on the second quantization weight range.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the convolutional neural network quantization method provided above, the method comprising: respectively acquiring a first quantization weight range of each layer in a pre-training convolutional neural network according to a preset important weight priority principle; the important weight priority principle comprises a principle of determining the frequency of occurrence of power exponents based on the original weight of each layer; respectively carrying out weight quantity expansion processing on the first quantization weight range to obtain a second quantization weight range of each layer; determining a target quantization weight range of the pre-trained convolutional neural network based on the second quantization weight range.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A convolutional neural network quantization method, comprising:
respectively acquiring a first quantization weight range of each layer in a pre-training convolutional neural network according to a preset important weight priority principle; the important weight priority principle comprises a principle of determining the frequency of occurrence of power exponents based on the original weight of each layer;
respectively carrying out weight quantity expansion processing on the first quantization weight range to obtain a second quantization weight range of each layer;
determining a target quantization weight range of the pre-trained convolutional neural network based on the second quantization weight range.
2. The convolutional neural network quantization method of claim 1, wherein the obtaining the first quantization weight range of each layer in the pre-trained convolutional neural network according to a preset important weight priority principle comprises:
respectively setting a quantization weight value range and a quantization bit width of each layer in a pre-training convolutional neural network; the quantization weight value domain comprises a plurality of quantization weights, the power base numbers of the absolute values of the quantization weights are respectively the same, the power indexes are different, and the power indexes of the quantization weights at the end points in the quantization weight value domain are unknown;
respectively acquiring a first power exponent with the highest frequency of occurrence in the original weight of each layer and a second power exponent with the highest frequency of occurrence adjacent to the first power exponent according to a preset important weight priority principle;
calculating a first endpoint power exponent of the quantization weights at the endpoints in the quantization weight value domain based on the first power exponent, the second power exponent and the quantization bit width, respectively;
and respectively determining a first quantization weight range of each layer based on a first quantization weight value range corresponding to the first endpoint power exponent.
3. The convolutional neural network quantization method of claim 2, wherein the performing weight number expansion processing on the first quantization weight range to obtain a second quantization weight range of each layer comprises:
based on the principle of weight 0 removal, the first endpoint power exponents are respectively adjusted to obtain second endpoint power exponents;
and determining a second quantization weight range of each layer based on a second quantization weight value range corresponding to the second endpoint power exponent respectively.
4. The convolutional neural network quantization method of claim 1, wherein the determining a target quantization weight range for the pre-trained convolutional neural network based on the second quantization weight range comprises:
and carrying out unified quantization processing on the second quantization weight range of each layer to obtain a target quantization weight range of the pre-training convolutional neural network.
5. The convolutional neural network quantization method of claim 2, wherein the determining a first power exponent with the highest frequency of occurrence and a second power exponent with the highest frequency of occurrence next to the first power exponent in the original weight of each layer according to a preset importance weight priority rule comprises:
determining the power exponent S of the original weight of each layer respectivelylAnd respectively determining the first power exponent maximum with the highest frequency in the original weight of each layer according to a preset important weight priority principlelAnd with the first power exponent maxinumlThe next highest power index 1lWherein:
Sl＝floor(log2(4abs(Wl)/3))
maxinuml＝Max_num1(Sl)
maxinum1l＝Max_neighboor(Sl，maxinuml)；
wherein floor (. cndot.) represents rounding-down, abs (. cndot.) represents absolute value, WlRepresenting the original weights of the l-th layer of the pre-trained convolutional neural network, SlThe power exponent of the original weight of the l-th layer of the pre-trained convolutional neural network is shown, Max _ num1(·) shows the value with the highest frequency of occurrence is taken, Max _ neighbor (S)l，maxinuml) Is shown at SlTaking out the herb and maxinumlAdjacent to and most frequently occurring value of (a), maxinumlRepresents the original weight W of the l-th layerlThe highest frequency of occurrence of the first power exponent, maxinum1lRepresents the sum of maxinumlAnd L is more than or equal to 1 and less than or equal to L, wherein L represents the total number of learnable layers in the pre-training convolutional neural network.
6. The convolutional neural network quantization method of claim 5, wherein the first endpoint power exponent of the quantization weight at the endpoint in the quantization weight value domain is calculated based on the first power exponent, the second power exponent and the quantization bit width, respectively, and the calculation formula comprises:
correspondingly, the first quantization weight range of each layer is respectively determined based on the first quantization weight value range corresponding to the first endpoint power exponent, wherein the determination process of each first quantization weight of each layer comprises the following steps:
wherein sgn (x) is a sign function,
b denotes the quantization bit width, nl1Maximum first-endpoint power exponent, n, representing quantization weight at endpoint in quantization weight value domain of layer ll2Minimum first endpoint power exponent, n, representing quantization weight at endpoint in quantization weight value domain of layer ll1≥nl2And n isl1And nl2Are all integers; wl(i, j) represents the original weight between the ith and jth neurons of the l-th layer of the pre-trained convolutional neural network, αlAnd betalTwo adjacent first quantization weights, beta, after the absolute value of the first quantization weight value range of the first layer is orderedl＞αl，
And expressing a first quantization weight between the ith neuron and the jth neuron of the ith layer, wherein I is more than or equal to 1 and less than or equal to I, I represents the total number of neurons input to the ith layer of the pre-trained convolutional neural network, J is more than or equal to 1 and less than or equal to J, and J represents the total number of neurons output from the ith layer of the pre-trained convolutional neural network.
7. The convolutional neural network quantization method of claim 3, wherein the first endpoint power exponents are respectively adjusted based on the weight-off-0 principle to obtain second endpoint power exponents, and an adjustment formula includes;
correspondingly, the second quantization weight range of each layer is determined based on the second quantization weight value range corresponding to the second endpoint power exponent, wherein the determination process of each second quantization weight of each layer comprises:
wherein sgn (x) is a sign function,
a maximum second endpoint power exponent representing the quantization weight at the endpoint in the quantization weight value domain of the l-th layer,
a minimum second endpoint power exponent representing the quantization weight at the endpoint in the quantization weight value domain of the l-th layer,
and
two second quantization weights adjacent to each other after the absolute value of the second quantization weight value range of the l-th layer is sorted,
Wl(i, j) represents the original weight between the ith neuron and the jth neuron of the l-th layer of the pre-trained convolutional neural network,
representing second quantization weight between the ith neuron and the jth neuron of the ith layer, I is more than or equal to 1 and less than or equal to I, I represents the total number of neurons input to the ith layer of the pre-trained convolutional neural network, J is more than or equal to 1 and less than or equal to J, J represents the total number of neurons output from the ith layer of the pre-trained convolutional neural network, maxinumlRepresents the original weight W of the l-th layerlThe highest frequency of occurrence of the first power exponent, maxinum1lRepresents the sum of maxinumlAnd L is more than or equal to 1 and less than or equal to L, wherein L represents the total number of learnable layers in the pre-training convolutional neural network.
8. A convolutional neural network quantization apparatus, comprising:
the acquisition module is used for respectively determining a first quantization weight range of each layer in the pre-training convolutional neural network according to a preset important weight priority principle; the important weight priority principle comprises a principle of determining the frequency of occurrence of power exponents based on the original weight of each layer;
the processing module is used for respectively carrying out weight quantity expansion processing on the first quantization weight range to obtain a second quantization weight range of each layer;
a determining module to determine a target quantization weight range of the pre-trained convolutional neural network based on the second quantization weight range.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of the convolutional neural network quantization method of any one of claims 1 to 7.
10. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the steps of the convolutional neural network quantization method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111265219.0A CN114139678A (en) | 2021-10-28 | 2021-10-28 | Convolutional neural network quantization method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111265219.0A CN114139678A (en) | 2021-10-28 | 2021-10-28 | Convolutional neural network quantization method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114139678A true CN114139678A (en) | 2022-03-04 |
Family
ID=80395730
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111265219.0A Pending CN114139678A (en) | 2021-10-28 | 2021-10-28 | Convolutional neural network quantization method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114139678A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114898108A (en) * | 2022-03-30 | 2022-08-12 | 哈尔滨工业大学 | CNN model lightweight method based on FPGA, target detection method and system |
-
2021
- 2021-10-28 CN CN202111265219.0A patent/CN114139678A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114898108A (en) * | 2022-03-30 | 2022-08-12 | 哈尔滨工业大学 | CNN model lightweight method based on FPGA, target detection method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109754066B (en) | Method and apparatus for generating a fixed-point neural network | |
| CN108510067B (en) | Convolutional neural network quantification method based on engineering realization | |
| CN107636697A (en) | The fixed point neutral net quantified based on floating-point neutral net | |
| CN110555450A (en) | Face recognition neural network adjusting method and device | |
| CN111105017B (en) | Neural network quantization method and device and electronic equipment | |
| WO2020238237A1 (en) | Power exponent quantization-based neural network compression method | |
| CN111178514A (en) | Neural network quantification method and system | |
| CN110874625A (en) | Deep neural network quantification method and device | |
| CN110647974A (en) | Network layer operation method and device in deep neural network | |
| Qi et al. | Learning low resource consumption cnn through pruning and quantization | |
| CN111240746A (en) | Floating point data inverse quantization and quantization method and equipment | |
| CN110188877A (en) | A kind of neural network compression method and device | |
| CN110503182A (en) | Network layer operation method and device in deep neural network | |
| CN112561050B (en) | Neural network model training method and device | |
| CN112686384A (en) | Bit-width-adaptive neural network quantization method and device | |
| CN114139678A (en) | Convolutional neural network quantization method and device, electronic equipment and storage medium | |
| CN112613604A (en) | Neural network quantification method and device | |
| CN115759192A (en) | Neural network acceleration method, device, equipment, chip and storage medium | |
| CN112488291A (en) | Neural network 8-bit quantization compression method | |
| CN116956997A (en) | LSTM model quantization retraining method, system and equipment for time sequence data processing | |
| CN117348837A (en) | Quantization method and device for floating point precision model, electronic equipment and storage medium | |
| CN113177627B (en) | Optimization system, retraining system, method thereof, processor and readable medium | |
| CN110852361B (en) | Image classification method and device based on improved deep neural network and electronic equipment | |
| CN112200275B (en) | Artificial neural network quantification method and device | |
| CN115034389A (en) | Neural network quantization and processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |