KR20190013162A - Method for convolution operation redution and system for performing the same - Google Patents

Abstract

Embodiments in the present invention relate to a hardware accelerator for a convolution neural network and a method for reducing a convolution operation and, more specifically, to a method for reducing a convolution operation by stopping an ongoing convolution operation when it is predicted that an output value of each convolution is to be included in a set range and performing subordinated convolution and a hardware accelerator performing the same.

Description

{METHOD FOR CONVOLUTION OPERATION REDUCTION AND SYSTEM FOR PERFORMING THE SAME}

Embodiments disclosed herein relate to a hardware accelerator and convolutional arithmetic reduction method for convolutional neural networks, and more particularly to a method and apparatus for halting a convolution operation whose output value is expected to be included in a set range, And a hardware accelerator for performing the convolution operation.

Deep neural networks are generally divided into convolutional neural networks (CNNs) using convolution and recurrent neural networks (RNN) using matrix-vector multiplication. CNN is known to be suitable for image processing, and RNN is suitable for continuous data processing. In CNN, most of the computation time is occupied by the convolution operation. At each layer of CNN, the output is obtained by convoluting between the input data and the kernel, and the output of the corresponding layer is obtained by applying an active function (generally, ReLU).

In the case of CNN, the convolution takes up most of the computation, so it is necessary to improve the efficiency of convolution computation to shorten the overall computation time.

Korean Patent No. 10-1563569, which is a prior art document, discloses a system and method for recognizing a dynamic visual image pattern through preliminary training of an exemplary pattern set. However, the convolution method as disclosed in the prior art is insufficient in improving the efficiency of the convolution operation and needs to be improved.

On the other hand, the background art described above is technical information acquired by the inventor for the derivation of the present invention or obtained in the derivation process of the present invention, and can not necessarily be a known technology disclosed to the general public before the application of the present invention .

Embodiments disclosed herein are aimed at disclosing a hardware accelerator and convolutional computational complexity reduction method for convolutional neural networks.

Also, in performing the convolutional neural network, the embodiments attempt to reduce the convolution operation amount by predicting the output value of the convolution operation in advance and stopping the meaningless convolution operation.

Embodiments also attempt to reduce the amount of convolutional computation by predicting the convolution operation early in performing the convolutional neural network.

Also, in performing the convolutional neural network, the embodiments attempt to find a proper point of the output value and the reduction rate of the computation amount to effectively reduce the convolution operation amount.

According to an embodiment of the present invention, a hardware accelerator for a convolutional neural network (CNN) performs a plurality of convolution operations on input data, Estimating whether or not an output value of the output signal is included in the setting range; Stopping a convolution operation that is expected to include an output value in the setting range, and performing a subordinate convolution.

According to another embodiment, in a hardware accelerator for a convolutional neural network (CNN), in performing a plurality of convolutions on input data, it is predicted whether or not the output value of each convolution is included in the setting range, Wherein the convolution operation is predicted to be included in the set range, and performs a subordinate convolution.

According to yet another embodiment, a computer-readable recording medium on which a program for performing a convolutional operation amount reduction method is recorded is disclosed. At this time, the method of reducing the convolutional operation amount is performed by a hardware accelerator for the convolutional neural network (CNN). In performing a plurality of convolution operations on the input data, whether the output value of each convolution is included in the set range Predicting; Stopping the convolution operation in which the output value is predicted to be included in the setting range, and performing subordinate convolution.

According to yet another embodiment, a computer program stored on a medium for performing a convolutional computational load reduction method, which is performed by a hardware accelerator for a neural network, is disclosed. At this time, the method of reducing the convolutional operation amount is performed by a hardware accelerator for the convolutional neural network (CNN). In performing a plurality of convolution operations on the input data, whether the output value of each convolution is included in the set range Predicting; Stopping the convolution operation in which the output value is predicted to be included in the setting range, and performing subordinate convolution.

According to any one of the above-described task solutions, embodiments disclosed herein may disclose a hardware accelerator and a convolutional operation amount reduction method for a convolutional neural network.

Also, in performing the convolutional neural network, the embodiments can reduce the convolution operation amount by predicting the output value of the convolution operation in advance and interrupting the meaningless convolution operation.

Embodiments can also reduce the amount of convolutional computation by predicting the convolution operation early in performing the convolutional neural network.

Also, in performing the convolutional neural network, the embodiments can find a proper point of the output value and the reduction rate of the computation amount and effectively reduce the convolution operation amount.

The hardware acceleration and convolutional computation reduction methods for such convolutional neural networks can effectively reduce the computational complexity of deep-run-based servers and mobile systems.

The effects obtained in the disclosed embodiments are not limited to the effects mentioned above, and other effects not mentioned are obvious to those skilled in the art to which the embodiments disclosed from the following description belong It can be understood.

1 is an exemplary view for explaining a neural network according to an embodiment.
2 is a block diagram illustrating a configuration of a neural network system according to an embodiment.
3 is a block diagram illustrating a hardware accelerator according to an embodiment of the present invention.
FIGS. 4 and 5 are exemplary diagrams illustrating a convolution operation performed in a hardware accelerator according to an embodiment.
6 and 7 are flowcharts for explaining a convolution operation amount reduction method according to an embodiment.

Various embodiments are described in detail below with reference to the accompanying drawings. The embodiments described below may be modified and implemented in various different forms. In order to more clearly describe the features of the embodiments, detailed descriptions of known matters to those skilled in the art are omitted. In the drawings, parts not relating to the description of the embodiments are omitted, and like parts are denoted by similar reference numerals throughout the specification.

Throughout the specification, when a configuration is referred to as being "connected" to another configuration, it includes not only a case of being directly connected, but also a case of being connected with another configuration in between. In addition, when a configuration is referred to as "including ", it means that other configurations may be included, as well as other configurations, as long as there is no specially contradicted description.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 is an exemplary diagram illustrating a neural network performed in a neural network system according to an exemplary embodiment.

The convolutional neural network convolves input data with each kernel to derive output data. At this time, the convolution includes a process of applying a weight (w) corresponding to each kernel in processing the input data.

Referring to FIG. 1, convolution includes a multiplication operation performed on a plurality of input data and a plurality of kernels, and a process of synthesizing a plurality of results.

On the other hand, the convolution as shown in Fig. 1 can be repeated for each layer. That is, the output data derived through the convolution is input to the input data of the next layer, and the convolution is performed once again. Such convolution is repeated in multiple layers until the final output value is derived. More specific embodiments will be described later in the relevant part.

2 is a block diagram illustrating a configuration of a neural network system 100 according to an embodiment.

The neural network system 100 according to one embodiment disclosed herein may include a hardware accelerator, in particular, a system for driving a Convolution Neural Network (CNN). A specific configuration will be described with reference to Fig.

Referring to FIG. 2, the neural network system 100 according to an embodiment may include an input / output device 110, a storage device 120, and a computing device 130.

The input / output device 110 according to an embodiment may include an input device for receiving input from a user, and an output device for displaying information such as a result of performing a job or a status of the neural network system 100. [ For example, the input / output device 110 may include an input device for receiving an instruction of data processing and an output device for outputting a result processed according to the received instruction.

Meanwhile, the storage device 120 may store data for performing a neural network. For example, store the input data that is the object of the convolutional neural network and store the output data as a result of the convolution neural network.

The storage device 120 may include a solid state drive (SSD), a flash memory, a magnetic random access memory (MRAM), a phase change RAM (PRAM), a ferroelectric RAM (FeRAM) hard disk, And may include synchronous random access memory (SRAM), dynamic random access memory (DRAM), and the like.

In addition, the computing device 130 controls the overall operation of the neural network system 100 and may include a processor, such as a CPU. In addition, the computing device 130 may include a hardware accelerator. At this time, the hardware accelerator may perform the convolution operation for the convolutional neural network and may be implemented by the GPU.

Referring now to FIG. 3, there is shown a block diagram of an embodiment of a portion of a neural network system 100. FIG.

3, the neural network system 100 may include an off-chip DRAM 121 as a storage device 120 and may include a hardware accelerator 131 as a computing device 130. The off-chip DRAM 121 and the hardware accelerator 131 may be connected to each other by a DMA unit (141) and a global buffer (142). The off-chip DRAM 121 and the hardware accelerator 131 transmit and receive data in the DMA scheme through the DMA unit 141 and temporarily store the transmitted and received data in the wide area buffer 142.

3, the hardware accelerator 131 may comprise a plurality of arrays of processing elements (PEs) 132 (PE arrays). At this time, the hardware accelerator 131 can read the data stored in the wide area buffer 142 and perform convolution through each calculation unit 132.

Referring now to FIG. 4 in conjunction with FIG. 4, there is shown an exemplary diagram of an embodiment for performing convolution in each calculation unit. According to the embodiment of FIG. 3, each calculation unit can store input data to be convoluted in an RF (Register File). In addition, the Fetch Controller can select the input data from the RF, apply the kernel weight and the activation function to the convolution, and then store it again in the RF. At this time, the fetch controller can repeat the convolution process.

Meanwhile, in performing the convolution described above, the hardware accelerator 131 according to the embodiment may perform a convolution operation amount reduction method to reduce the amount of convolution operation. The hardware accelerator 131 may perform convolution with respect to a plurality of layers, and the calculation amount reduction method according to the embodiment may be applied to each layer. Hereinafter, a description will be made on the basis of one layer, assuming a plurality of layers as necessary.

According to the embodiment, the hardware accelerator 131 for the convolutional neural network can predict whether or not the output value of each convolution is included in the setting range when performing a plurality of convolutions with respect to the input data and the kernel. At this time, the hardware accelerator 131 can stop the calculation of the convolution which is expected to include the output value in the set range, and can perform the remaining convolution. At this time, the setting range can be set to a negative value.

In relation to the convolutional neural network, ReLU (rectified linear unit) can be applied as an activation function after convolution. At this time, if the convolution output value is negative, the result of the ReLU operation is output as 0.

Therefore, if the convolution output value is predicted as a negative value, the hardware accelerator 131 can reduce the convolution operation amount by stopping the convolution, outputting the output value of the ReLU operation as 0, and performing the subordinate convolution.

Meanwhile, in predicting whether or not the output value of the convolution is included in the setting range, the hardware accelerator 131 may derive the intermediate result of the output value and predict whether or not the output value is included in the setting range based on the derived intermediate result have. For example, the intermediate result may be compared with a threshold value to predict whether or not the output value is included in the setting range.

In this regard, the input data may include a plurality of input feature maps. At this time, the hardware accelerator 131 sequentially performs a multiplication operation of a plurality of input property maps and a kernel, and combines the results of the multiplication operations to derive an output value of the convolution.

That is, the multiplication operation may be performed a plurality of times by performing a multiplication operation with a plurality of input property maps with respect to the kernel. In this case, the intermediate result can be derived by synthesizing the result of the multiplication operation performed until the end of each multiplication operation .

For example, at the end of the first product operation, the result of the first product operation is the intermediate result, and at the end of the second product operation, the result of the first product operation and the result of the second product operation are derived as intermediate results At the end of the Nth product operation, a result obtained by combining the results from the first product operation to the Nth product operation can be derived as an intermediate result of the convolution on the input property map.

At this time, the hardware accelerator 131 predicts whether the output value of the convolution is included in the setting range, for example, whether it is negative or not based on the intermediate result derived as described above, and if the output value of the convolution is negative , You can stop the convolution.

Further, when the input data includes a plurality of input characteristic maps, the hardware accelerator 131 sequentially performs a multiplication operation of a plurality of input characteristic maps and a kernel, and arranges the operation order of a plurality of input characteristic maps . At this time, the hardware accelerator 131 may sort the operation order of the input characteristic map based on the weights of the kernels corresponding to the plurality of input characteristic maps.

In this regard, according to the embodiment, the hardware accelerator 131 may obtain a plurality of input property maps from the convolution operation. In other words, when the hardware accelerator 131 has a plurality of layers in performing the convolutional neural network, the input characteristic map input to one layer is an output characteristic map of the convolution operation performed in the previous layer map.

Also, in performing convolution on each layer, there may be one or more kernels to be operated on with input data in each layer. That is, the hardware accelerator 131 may perform many-to-many convolution operations on a plurality of input characteristic maps and a plurality of kernels, and combine them to derive an output characteristic map. Therefore, each output characteristic map can have a weight according to the corresponding kernel. That is, each input characteristic map for a predetermined layer may have a corresponding weight according to the convolution operation performed in the layer of the previous layer.

Using this, the hardware accelerator 131 can sort the operation order of the input characteristic map based on the weight of the kernel of the preceding layer corresponding to the input characteristic map.

At this time, the hardware accelerator 131 arranges the operation order of the input property map according to the weight of the corresponding kernel, and arranges the operation order of the input property map in the order of the largest absolute value of the weights. The larger the absolute value of the weight, the larger the influence on the output value of the convolution becomes. Thus, by sorting the operation sequence in this way, the output value can be predicted early and the meaningless operation can be stopped early.

Meanwhile, according to the embodiment, the hardware accelerator 131 can set a threshold value. The amount of reduction of the operation and the accuracy of the output value have a negative correlation. That is, if the amount of computation is reduced a lot, the accuracy decreases. If the accuracy is increased, the computation amount increases. Accordingly, the hardware accelerator 131 can set a threshold value that can reduce the amount of calculation to the maximum within the tolerance of the accuracy.

For example, the hardware accelerator 131 analyzes the accuracy of the output value and the reduction amount of the calculation derived by performing convolution by variously substituting the sample value serving as the threshold value, and calculates the correlation between the accuracy and the reduction amount of the calculation The threshold value can be set based on the threshold value.

For example, when the hardware accelerator 131 performs the neural network having the learned N layers, the N-1 layers are set to perform all without stopping the convolution, and for each layer, Converts the sample value from small to large values. At this time, it is possible to calculate the correlation (A) between the output quality value and the threshold value by analyzing the output value. At this time, if the tolerance is set for the accuracy, the threshold value at the lowest accuracy among the allowed accuracy can be obtained. In the same way, it is possible to calculate the correlation (B) between the reduction amount of the operation and the threshold value. This process can be performed in each layer.

At this time, the correlation (C) between the accuracy and the reduction amount of the calculation can be obtained based on the correlation between A and B. [

In addition, tolerance of loss can be set for the final output value of the neural network obtained by performing convolution in all layers. At this time, the allowable loss can be distributed to each layer to obtain the most effective threshold value at each layer.

Specifically, some values of the allowed loss are distributed for the layer with the best efficiency of the reduction amount compared with the accuracy. At this time, the reduction amount of the divided layer is calculated based on the correlation (C) between the accuracy and the reduction amount, and the temporary threshold value is calculated based on the reduction amount.

In this state, the predetermined value of the allowed loss is distributed to the layer with the highest efficiency, and the temporary threshold value is calculated in the same manner. If all the allowed losses are distributed by repeating this process, the temporary threshold value set for each layer at that time can be set as a threshold value to be applied.

On the other hand, referring to Fig. 5, there is shown an example of the amount of reduction (horizontal axis) of calculation and the loss of accuracy (vertical axis). As shown in FIG. 5, the greater the amount of reduction of the calculation, the greater the loss of accuracy. According to the above-described embodiment, the most effective threshold value for the allowed loss can be derived and applied.

6 and 7 are flowcharts for explaining a method of decreasing the convolutional operation amount performed by the hardware accelerator 131. FIG. The method of reducing the convolutional operation amount according to the embodiment shown in FIGS. 6 and 7 includes steps that are processed in a time-series manner in the hardware accelerator 131 according to the embodiment of FIGS. 1-5. Therefore, even if omitted from the following description, the above description of the hardware accelerator 131 according to the embodiment related to Figs. 1 to 5 can be applied to the case where the convolution operation amount reduction according to the embodiments shown in Figs. 6 and 7 Method.

6, the hardware accelerator 131 for the convolutional neural network predicts whether or not the output value of the convolution with respect to the input data is included in the setting range (S61). If the output value is predicted to be included in the setting range, The routine operation can be stopped (S62).

In other words, in performing a plurality of convolutions on the input data, the hardware accelerator 131 predicts whether or not the output value of each convolution is included in the setting range, and outputs a convolution operation . ≪ / RTI > Also, subordinate convolution of the interrupted convolution can be performed. According to the embodiment, the setting range may be negative.

FIG. 7 shows a flow chart embodying the step S61 described above. Referring to FIG. 7, the hardware accelerator 131 predicts an output value, derives an intermediate value of the output value (S71), and predicts whether or not the output value is included in the setting range based on the intermediate result obtained (S72) .

At this time, the hardware accelerator 131 may compare the intermediate result with the threshold based on the intermediate result, and predict whether or not the output value is included in the set range.

When the input data includes a plurality of input characteristic maps, the hardware accelerator 131 sequentially performs a multiplication operation of a plurality of input characteristic maps and a kernel in deriving the intermediate result of the output values, The intermediate result can be derived by synthesizing the result of the multiplication operation performed until the end of the operation. The hardware accelerator 131 can predict whether or not the output value is included in the setting range based on the intermediate result thus derived.

When the input data includes a plurality of input characteristic maps, the hardware accelerator 131 sequentially performs a multiplication operation of a plurality of input characteristic maps and a kernel in predicting an output value, The order can be sorted and the product operation can be performed sequentially.

At this time, the hardware accelerator 131 may sort the operation order of the input characteristic map based on the weights of the kernels corresponding to the respective input characteristic maps. Then, the hardware accelerator 131 can arrange the order of operations in the descending order of the weights or the absolute values of the weights. As the absolute value of the weight or weight has a larger influence on the output value, it is possible to quickly determine whether or not the output value is included in the set range by performing the convolution in the ascending order of the weight or the absolute value of the weight.

The above embodiments can be applied to various neural network models. At this time, it is possible to derive a threshold value for each layer according to the weight of the kernel and the number of layers, thereby performing a calculation amount reduction method.

The term " device " used in the above embodiments means a hardware component such as software or a field programmable gate array (FPGA) or an ASIC, and the 'device' performs certain roles. However, "device" is not limited to software or hardware. The 'device' may be configured to be in an addressable storage medium and configured to play one or more processors. Thus, by way of example, 'to device' may include components such as software components, object-oriented software components, class components and task components, and processes, functions, attributes, , Subroutines, segments of program patent code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

The functions provided in the components and 'devices' may be combined into a smaller number of components and 'devices' or may be separated from additional components and 'devices'.

In addition, the components and " devices " may be implemented to play back one or more CPUs in a device or a secure multimedia card.

The method of reducing the convolutional operation amount according to the embodiment described with reference to FIGS. 6 and 7 may also be implemented in the form of a computer-readable medium storing instructions and data executable by a computer. At this time, the command and data may be stored in the form of program code, and when executed by the processor, a predetermined program module may be generated to perform a predetermined operation. In addition, the computer-readable medium can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. The computer-readable medium can also be a computer storage medium, which can be volatile and non-volatile, implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, For example, computer recording media may include magnetic storage media such as HDDs and SSDs, optical recording media such as CD, DVD and Blu-ray discs, or other types of media accessible via a network. May be the memory included in the server.

Also, the method for reducing the convolutional operation amount according to the embodiment described with reference to FIGS. 6 and 7 may be implemented by a computer program (or a computer program product) including instructions executable by the computer. A computer program includes programmable machine instructions that are processed by a processor and can be implemented in a high-level programming language, an object-oriented programming language, an assembly language, or a machine language . The computer program may also be recorded on a computer readable recording medium of a type (e.g., memory, hard disk, magnetic / optical medium or solid-state drive).

Therefore, the method of reducing the convolutional operation amount according to the embodiment described with reference to FIGS. 6 and 7 can be realized by the computer program as described above being executed by the computing device. The computing device may include a processor, a memory, a storage device, a high-speed interface connected to the memory and a high-speed expansion port, and a low-speed interface connected to the low-speed bus and the storage device. Each of these components is connected to each other using a variety of buses and can be mounted on a common motherboard or mounted in any other suitable manner.

Where the processor may process instructions within the computing device, such as to display graphical information to provide a graphical user interface (GUI) on an external input, output device, such as a display connected to a high speed interface And commands stored in memory or storage devices. As another example, multiple processors and / or multiple busses may be used with multiple memory and memory types as appropriate. The processor may also be implemented as a chipset comprised of chips comprising multiple independent analog and / or digital processors.

The memory also stores information within the computing device. In one example, the memory may comprise volatile memory units or a collection thereof. In another example, the memory may be comprised of non-volatile memory units or a collection thereof. The memory may also be another type of computer readable medium such as, for example, a magnetic or optical disk.

And the storage device can provide a large amount of storage space to the computing device. The storage device may be a computer readable medium or a configuration including such a medium and may include, for example, devices in a SAN (Storage Area Network) or other configurations, and may be a floppy disk device, a hard disk device, Or a tape device, flash memory, or other similar semiconductor memory device or device array.

It will be apparent to those skilled in the art that the above-described embodiments are for illustrative purposes only and that those skilled in the art will readily understand that various changes and modifications can be made without departing from the spirit and scope of the present invention. You will understand. It is therefore to be understood that the above-described embodiments are to be considered in all respects only as illustrative and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

It is to be understood that the scope of the present invention is defined by the appended claims rather than the foregoing description and should be construed as including all changes and modifications that come within the meaning and range of equivalency of the claims, .

100: Neural network system 110: Input / output device
120: storage device 130: computing device
131: Hardware Accelerator

Claims (16)

A hardware accelerator for convolutional neural networks (CNN)
The method comprising: predicting whether or not an output value of each convolution is included in a setting range in performing a plurality of convolution operations on input data; And
Stopping a convolution operation that is predicted to include an output value in the setting range, and performing a subordinate convolution. The method according to claim 1,
Wherein the set range is a negative number. The method according to claim 1,
Wherein the predicting comprises:
Deriving an intermediate result of the output value; And
And predicting whether the output value is included in the setting range based on the intermediate result. The method of claim 3,
Wherein the step of predicting whether the output value is included in the setting range based on the intermediate result includes:
And comparing the intermediate result with a threshold value to predict whether or not the output value is included in the set range. The method of claim 3,
Wherein the input data includes:
Comprising a plurality of input feature maps,
Wherein deriving an intermediate result of the output value comprises:
Wherein the intermediate result is derived by sequentially performing a multiplication operation of the plurality of input property maps and a kernel and synthesizing the result of the multiplication operation performed until the end of each multiplication operation. The method according to claim 1,
Wherein the input data includes:
Comprising a plurality of input feature maps,
Wherein the predicting comprises:
Wherein the multiplication operation is sequentially performed by sequentially performing a multiplication operation of the plurality of input property maps and a kernel, and arranging the operation order of the plurality of input property maps. The method according to claim 6,
Wherein the predicting comprises:
Wherein the arithmetic procedure of the input characteristic map is arranged based on a weight of a kernel corresponding to each input characteristic map, . A computer-readable recording medium on which a program for carrying out the method according to claim 1 is recorded. A computer program stored on a medium for performing the method of claim 1, the computer program being performed by a hardware accelerator for a convolutional neural network (CNN). A hardware accelerator for a convolutional neural network (CNN)
In performing a plurality of convolutions on input data, it is possible to predict whether or not the output value of each convolution is included in the setting range, to stop the convolution operation in which the output value is predicted to be included in the setting range, and to perform subordinate convolution Wherein the hardware accelerator for convolutional neural networks is implemented as a hardware accelerator for convolutional neural networks. 11. The method of claim 10,
Characterized in that the setting range is a negative number. 11. The method of claim 10,
And estimating whether the output value is included in the setting range,
Derives an intermediate result of the output value, and predicts whether the output value is included in the setting range based on the intermediate result. 13. The method of claim 12,
Predicting whether the output value is included in the setting range based on the intermediate result,
Wherein the intermediate result is compared with a threshold value to predict whether the output value is included in the setting range. 13. The method of claim 12,
Wherein the input data includes:
Comprising a plurality of input feature maps,
Wherein the intermediate result is obtained by sequentially performing a multiplication operation of the plurality of input property maps and a kernel to synthesize a result of a multiplication operation performed until the end of each multiplication operation, . 11. The method of claim 10,
Wherein the input data includes:
Comprising a plurality of input feature maps,
Characterized in that the multiplication operation is sequentially performed by sequentially performing a multiplication operation of the plurality of input property maps and a kernel and arranging the operation order of the plurality of input property maps, . 16. The method of claim 15,
Wherein the operation order of the input characteristic map is sorted based on a weight of a kernel corresponding to each input characteristic map. Hardware accelerator for.

Images