KR102718583B1 - Method and apparatus of data processing for neural network - Google Patents

Abstract

A data processing method and device for a neural network are disclosed. According to one embodiment, the data processing method includes the steps of accumulating multiplication results between at least some of input elements in an input plane and at least some of weight elements in a weight plane to generate accumulated data, and generating an output plane corresponding to an output channel among output planes of output feature maps each corresponding to an output channel based on the accumulated data.

Description

METHOD AND APPARATUS OF DATA PROCESSING FOR NEURAL NETWORK

The following embodiments relate to a data processing method and device for a neural network.

Technical automation of the recognition process is implemented, for example, by neural network models implemented as processors as special computational structures, which, after considerable training, can provide computationally intuitive mappings between input and output patterns. The trained ability to generate such mappings can be referred to as the learning ability of the neural network. Furthermore, due to specialized training, such specialized trained neural networks can have generalization abilities, for example, generating relatively accurate outputs for untrained input patterns.

According to one embodiment, a data processing method for a neural network includes the steps of: obtaining a first input plane corresponding to a first input channel among input planes of input feature maps each corresponding to input channels; obtaining a first weight plane corresponding to the first input channel among weight planes of weight kernels each corresponding to the input channels; accumulating multiplication results between at least some of the first input elements in the first input plane and at least some of the first weight elements in the first weight plane to generate first accumulated data; and generating a first output plane corresponding to the first output channel among output planes of output feature maps each corresponding to the output channels based on the first accumulated data.

The step of generating the first output plane may include the step of generating the first output plane based on the sum of each accumulated data for each input channel including the first accumulated data. The data processing method may further include the step of obtaining a second input plane corresponding to a second input channel among the input planes; the step of obtaining a second weight plane corresponding to the second input channel among the weight planes; and the step of accumulating multiplication results between at least some of the second input elements in the second input plane and at least some of the second weight elements in the second weight plane to generate second accumulated data. The step of generating the first output plane may include the step of generating the first output plane based on the sum of the first accumulated data and the second accumulated data.

The step of generating the first accumulated data may include the steps of extracting first input element vectors corresponding to at least some of the first weight elements in the first input plane; generating first weighted input element vectors corresponding to a multiplication result between the first input element vectors and the at least some of the first weight elements; and accumulating the first weighted input element vectors to generate the first accumulated data. The step of extracting the first input element vectors may include the steps of determining offsets corresponding to the first input element vectors based on indices of the at least some of the first weight elements; and extracting the first input element vectors in the first input plane based on the offsets. The sizes of the first input element vectors and the first weighted input element vectors may correspond to a single instruction multiple data (SIMD) operation unit.

When the first accumulated data is generated, a multiplication operation between the zero weight elements corresponding to 0 among at least some of the first weight elements and the at least some of the first input elements may be omitted. The data processing method may further include a step of determining the number of non-zero weight elements not corresponding to 0 among the first weight elements; and a step of selecting an operation type corresponding to the determined number of non-zero weight elements from among operation types each set to perform an operation in a predetermined manner. The step of generating the first accumulated data may include a step of accumulating multiplication results between the at least some of the first input elements and the non-zero weight elements corresponding to the at least some of the first weight elements based on the selected operation type to generate the first accumulated data. The step of generating the first accumulated data may include a step of extracting first input element vectors corresponding to the non-zero weight elements in the first input plane based on indices of the non-zero weight elements; The method may include generating first weighted input element vectors corresponding to a multiplication result between the first input element vectors and the non-zero weight elements corresponding to at least some of the first weight elements; and generating the first accumulated data by accumulating the first weighted input element vectors.

According to one embodiment, a data processing device for a neural network includes a processor; and a memory including instructions executable by the processor, wherein when the instructions are executed by the processor, the processor obtains a first input plane corresponding to a first input channel among input planes of input feature maps corresponding to input channels, respectively, obtains a first weight plane corresponding to the first input channel among weight planes of weight kernels corresponding to the input channels, respectively, accumulates multiplication results between at least some of the first input elements in the first input plane and at least some of the first weight elements in the first weight plane to generate first accumulated data, and generates a first output plane corresponding to the first output channel among output planes of output feature maps corresponding to output channels, respectively, based on the first accumulated data.

According to one embodiment, a data processing method for a neural network includes the steps of: obtaining a first input plane corresponding to a first input channel among input planes of input feature maps each corresponding to input channels; obtaining a first weight plane corresponding to the first input channel among weight planes of weight kernels each corresponding to the input channels; accumulating multiplication results between at least some of the first input elements in the first input plane and at least some of the first weight elements in the first weight plane to generate first accumulated data; and generating a first output plane corresponding to the first output channel among output planes of output feature maps each corresponding to the output channels based on the first accumulated data.
The step of generating the first output plane may include the step of generating the first output plane based on the sum of each accumulated data for each input channel including the first accumulated data. The data processing method may further include the step of obtaining a second input plane corresponding to a second input channel among the input planes; the step of obtaining a second weight plane corresponding to the second input channel among the weight planes; and the step of accumulating multiplication results between at least some of the second input elements in the second input plane and at least some of the second weight elements in the second weight plane to generate second accumulated data. The step of generating the first output plane may include the step of generating the first output plane based on the sum of the first accumulated data and the second accumulated data.
The step of generating the first accumulated data may include the steps of extracting first input element vectors corresponding to at least some of the first weight elements in the first input plane; generating first weighted input element vectors corresponding to a multiplication result between the first input element vectors and the at least some of the first weight elements; and accumulating the first weighted input element vectors to generate the first accumulated data. The step of extracting the first input element vectors may include the steps of determining offsets corresponding to the first input element vectors based on indices of the at least some of the first weight elements; and extracting the first input element vectors in the first input plane based on the offsets. The sizes of the first input element vectors and the first weighted input element vectors may correspond to a single instruction multiple data (SIMD) operation unit.
When the first accumulated data is generated, a multiplication operation between the zero weight elements corresponding to 0 among at least some of the first weight elements and the at least some of the first input elements may be omitted. The data processing method may further include a step of determining the number of non-zero weight elements not corresponding to 0 among the first weight elements; and a step of selecting an operation type corresponding to the determined number of non-zero weight elements from among operation types each set to perform an operation in a predetermined manner. The step of generating the first accumulated data may include a step of accumulating multiplication results between the at least some of the first input elements and the non-zero weight elements corresponding to the at least some of the first weight elements based on the selected operation type to generate the first accumulated data. The step of generating the first accumulated data may include a step of extracting first input element vectors corresponding to the non-zero weight elements in the first input plane based on indices of the non-zero weight elements; The method may include generating first weighted input element vectors corresponding to a multiplication result between the first input element vectors and the non-zero weight elements corresponding to at least some of the first weight elements; and generating the first accumulated data by accumulating the first weighted input element vectors.
According to one embodiment, a data processing device for a neural network includes a processor; and a memory including instructions executable by the processor, wherein when the instructions are executed by the processor, the processor obtains a first input plane corresponding to a first input channel among input planes of input feature maps corresponding to input channels, respectively, obtains a first weight plane corresponding to the first input channel among weight planes of weight kernels corresponding to the input channels, respectively, accumulates multiplication results between at least some of the first input elements in the first input plane and at least some of the first weight elements in the first weight plane to generate first accumulated data, and generates a first output plane corresponding to the first output channel among output planes of output feature maps corresponding to output channels, respectively, based on the first accumulated data.

The specific structures or functions disclosed below are merely exemplified for the purpose of explaining technical concepts and may be implemented in various forms other than those disclosed below and do not limit the embodiments of the present specification.

Although the terms first or second may be used to describe various components, these terms should be understood only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, the terms "comprises" and the like are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but should be understood to not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

Hereinafter, embodiments are described in detail with reference to the attached drawings. The same reference numerals presented in each drawing represent the same components.

FIG. 1 is a diagram illustrating a processing device for processing data for a neural network according to one embodiment. Referring to FIG. 1, a processing device (100) may include a neural network (110) and may process operations related to the neural network (110). For example, operations related to the neural network (110) may include object recognition operations and user authentication operations.

The neural network (110) can perform object recognition or user authentication operations by mapping input data and output data in a nonlinear relationship based on deep learning. Deep learning is a machine learning technique for solving problems such as image or voice recognition from a big data set. Deep learning can be understood as an optimization problem-solving process that finds a point where energy is minimized while training the neural network (110) using prepared training data. Through supervised or unsupervised learning of deep learning, the structure of the neural network (110) or the weight corresponding to the model can be obtained, and the input data and the output data can be mapped to each other through these weights.

The neural network (110) may correspond to a deep neural network (DNN) including a plurality of layers. The plurality of layers may include an input layer, at least one hidden layer, and an output layer. The first layer, the second layer, and the nth layer illustrated in FIG. 1 may be at least some of these plurality of layers. The neural network (110) may include at least one of a fully connected network, a convolutional neural network (CNN), and a recurrent neural network (RNN). For example, at least some of the plurality of layers in the neural network (110) may correspond to a CNN, and other some may correspond to a fully connected network.

In CNN, the data input to each layer can be referred to as an input feature map, and the data output from each layer can be referred to as an output feature map. The input feature map and the output feature map can also be referred to as activation data. In the input layer, the input feature map can correspond to the input data.

In order to process an operation regarding a neural network (110), the processing device (100) can perform a convolution operation between an input feature map and a weight kernel for each convolutional layer, and generate an output feature map based on the result of the convolution operation. If the width and depth of the neural network (110) are sufficiently large, it can have a capacity sufficient to implement an arbitrary function. If the neural network (110) learns a sufficiently large amount of training data through an appropriate training process, it can achieve optimal performance.

The weight kernel may be expressed as being determined 'in advance', where 'in advance' may indicate before the neural network (110) is 'started'. The neural network (110) being 'started' may mean that the neural network (110) is ready for inference. For example, the neural network (110) being 'started' may include the neural network (110) being loaded into memory, or input data for inference being input into the neural network (110) after the neural network (110) is loaded into memory.

As will be described in more detail below, the convolution operation according to the embodiments is performed by accumulating intermediate results of the convolution operation in an output feature map, and does not require a buffering operation that transforms a weight kernel or an input feature map into a form suitable for convolution and stores it in a buffer. In other words, the convolution operation according to the embodiments can use the data of the input feature map stored in a planar form as it is. Accordingly, the efficiency of the convolution operation can be significantly improved. In addition, in the convolution operation according to the embodiments, multiplying one weight element corresponding to a scalar and one input plane corresponding to a matrix can correspond to one unit operation. Accordingly, zero-skip can be efficiently processed through software for weight elements having a value of 0.

FIG. 2 is a diagram illustrating a convolution operation process according to one embodiment. Referring to FIG. 2, an output feature map (230) is generated according to a convolution operation between a weight kernel (210) and an input feature map (220). The form in which data of the weight kernel (210), the input feature map (220), and the output feature map (230) are stored in a memory space may each be expressed as a plane. For example, each of the weight kernels (1) to (D) may include C weight planes, the input feature map (220) may include C input planes, and the output feature map (230) may include D output planes. The C weight planes and the C input planes may each correspond to input channels, and the D output planes may each correspond to output channels. In addition, C may correspond to the number of input channels, and D may correspond to the number of output channels.

Each plane may include elements of a predetermined bit width. For example, each weight plane may have a size of K*K, and each input plane and each output plane may have a size of W*H, where W, K, and H may each represent the number of elements. An element of a weight plane may be referred to as a weight element, an element of an input plane may be referred to as an input element, and an element of an output plane may be referred to as an output element. A convolution operation according to an embodiment may be performed on an element-by-element basis.

For convenience of explanation, the width and height of the weight plane are assumed to be equal to K, and the sizes of the input plane and the output plane are assumed to be equal to W*H. However, depending on the embodiment, the width and height of the weight plane may be different from each other, or the sizes of the input plane and the output plane may be different from each other.

Figure 3 is a diagram illustrating a convolution operation using a sliding window method. According to the convolution operation using a sliding window method, a weight kernel (310) slides on an input feature map (320) and a convolution operation is performed to generate an output feature map (330).

The sliding window method is generally used for convolution operations, and can be distinguished from the accumulation method according to embodiments. For example, in the case of the sliding window method, a buffering operation may be performed on the input feature map (320) to generate column vectors. In the case of the accumulation method according to the embodiment, since the intermediate result of the convolution operation is accumulated in the output feature map (330), a buffering operation is not required like the sliding window method.

According to the convolution operation of the sliding window method, in the process of the weight kernel (310) sliding on the input feature map (320), the weight kernel (310) performs an operation with data stored in a non-consecutive address of the input feature map (320), so that the input feature map (320) can be transformed into continuous data of an appropriate form in order to improve the operation processing speed. For example, in FIG. 3, it is assumed that the sliding stride is 1 and zero padding using two rows of zero element vectors is applied to each of the horizontal and vertical directions of the input feature map (320). In this case, K ² * C row vectors corresponding to the weight kernel (310) can be defined, and the input feature map (320) can be transformed into K ² * C column vectors.

The column vector can be buffered in the column buffer from the input feature map (320) of the planar structure or the interleaved structure. In the case of the planar structure, in the process of buffering the input feature map (320) into the column vector, the number of non-consecutive memory accesses may be as many as the product of the height (K) of the kernel and the number (C) of input channels to determine one output element. In the case of the interleaved structure, in the process of buffering the input feature map (320) into the column vector, the number of non-consecutive memory accesses may be as many as the height (K) of the kernel to determine one output element.

In the case of the cumulative convolution according to the embodiment, since the intermediate result of the convolution operation is accumulated in the output feature map (330), a separate buffering operation for transforming the input feature map (320) into a structure such as a planar or interleaved structure is not required. Therefore, the cumulative convolution can minimize memory access and maximize the processing speed of the convolution operation.

FIGS. 4 and 5 are diagrams illustrating a process of generating one output plane through a convolution operation in an accumulative manner according to one embodiment. For example, an output feature map may include D output planes, and FIGS. 4 and 5 may correspond to a process of generating one of the D output planes. The process illustrated in FIGS. 4 and 5 may be repeated for the D output planes to generate an output feature map.

Referring to FIG. 4, an output plane (430) is generated through a convolution operation between an input feature map (410) and a weight kernel (420). For example, the weight kernel (420) may correspond to the d-th among D weight kernels, and the output plane (430) may correspond to the d-th among D output planes. The input feature map (410) may include the input planes (510) of FIG. 5, and the weight kernel (420) may include the weight planes (520) of FIG. 5. The output plane (430) may correspond to the output plane (540) of FIG. 5.

Referring to FIG. 5, input planes (511, 512, 513) correspond to input planes (510). The number of input planes (511, 512, 513) corresponds to the number of input channels (C). Below, it is assumed that the number of input channels (C) is 3. However, this is for convenience of explanation, and the number of input channels (C) may have various values other than 3. Weight planes (521, 522, 523) correspond to weight planes (520), and accumulation planes (531, 532, 533) correspond to accumulation planes (530).

An accumulation plane (531) is generated through a MAC operation between an input plane (511) and a weight plane (521), an accumulation plane (532) is generated through a MAC operation between an input plane (512) and a weight plane (522), and an accumulation plane (533) is generated through a MAC operation between an input plane (513) and a weight plane (523). The MAC operation process will be described in detail later. When the accumulation planes (531, 532, 533) are generated, an output plane (540) can be generated based on the accumulation planes (531, 532, 533). For example, the output plane (540) can be generated through the sum of the accumulation planes (531, 532, 533).

FIGS. 6 and 7 are diagrams illustrating a MAC (multiply and accumulate) operation between an input plane and a weight plane for a convolution operation in an accumulative manner according to one embodiment.

Referring to FIG. 6, an accumulation plane (630) is generated based on a MAC operation between each input element of the input plane (610) and each weight element of the weight plane (620). The weight plane (620) includes weight elements of w ₁ to w _9. The weight plane (620) is described as having a size of 3*3, but this is for convenience of description, and the weight plane (620) may have various sizes other than 3*3. Although omitted in FIG. 6, the input plane (610) and the accumulation plane (630) may also each include a plurality of elements, and a convolution operation may be performed on an element-by-element basis.

Referring to FIG. 7, the input plane (711) corresponds to the input plane (610) of FIG. 6, the weight elements (w ₁ to w ₉ ) of FIG. 7 correspond to the weight elements (w ₁ to w ₉ ) of the weight plane (620) of FIG. 6, and the accumulation plane (740) corresponds to the accumulation plane (630) of FIG. 6. Zero padding may be performed on the input plane (711) based on the sliding stride to generate the input plane (712). For example, if the size of the input plane (711) is W*H and the sliding stride is 1, the input plane (712) may have a size of (W+2)*(H+2).

Assuming that a sliding window-style convolution operation is performed between a weight plane including weight elements (w ₁ to w ₉ ) and an input plane (712), response areas (721 to 729) to which each of the weight elements (w ₁ to w ₉ ) reacts can be defined on the input plane (712). For example, when a sliding window-style convolution operation is performed, input elements in the response area (721) can react to the weight element (w ₁ ), input elements in the response area (722) can react to the weight element (w ₂ ), and input elements in the response area (729) can react to the weight element (w ₉ ).

The size of the reaction areas (721 to 729) is the same as the size of the input plane (711), and the offset of each of the reaction areas (721 to 729) can be determined based on the index of each of the weight elements (w ₁ to w ₉ ). For example, when the width of the input plane (711) is W+2, the offset of each of the reaction areas (721 to 729) can be defined as (W+2)*a+b. The offset can be determined based on the input plane (e.g., the origin of the input plane to which padding is applied). Here, a represents the quotient of (i-1) divided by K, and b represents the remainder of (i-1) divided by K. i represents the index of the weight elements (w ₁ to w ₉ ), and K represents the width of the weight kernel. According to this, the offset of the reaction area (721) is 0, the offset of the reaction area (722) is 1, and the offset of the reaction area (729) is (W+2)*2+2.

Multiplication results (731 to 739) may be generated according to the product between the input elements in each reaction region (721 to 729) and each weight element (w ₁ to w ₉ ), and an accumulation plane (740) may be generated according to the accumulation of the multiplication results (731 to 739). For example, the output plane may be generated through the sum of C accumulation planes, and FIG. 7 may correspond to a process in which an accumulation plane (740) corresponding to one of the C accumulation planes is generated. Each element in the multiplication results (731 to 739) may be referred to as a multiplication result element. The process illustrated in FIG. 7 may be repeated for the C accumulation planes to generate the output plane. In addition, when the output feature map includes D output planes, the output feature map may be determined through the D output planes generated based on accumulation.

According to the embodiments, the output feature map is generated by accumulating the multiplication results (731 to 739) corresponding to the intermediate results of the convolution operation, and it is not required to transform the input feature map into continuous data and store it in a buffer. Accordingly, the time required to transform the input feature map into continuous data and store it in a buffer is reduced, thereby enabling the speed of the convolution operation to be increased, and the memory space for storing the transformed data can be saved.

FIGS. 8 to 10 are diagrams illustrating a convolution operation using an accumulation method using SIMD (single instruction multiple data) processing according to one embodiment. SIMD refers to an operation processing method of a processor that processes multiple data with a single instruction. As described in detail below, a convolution operation using an accumulation method according to an embodiment can be performed through SIMD.

Referring to FIG. 8, a weight plane (810) may slide on a sliding region (821) of an input plane (820), and a MAC operation may be performed to determine an accumulation region (831) of an accumulation plane (830). Similarly, a weight plane (810) may slide on a sliding region (822), and a MAC operation may be performed to determine an accumulation region (832), and a weight plane (810) may slide on a sliding region (823), and a MAC operation may be performed to determine an accumulation region (833). The heights of the sliding regions (821 to 823) correspond to the height of the weight plane (810), and the heights of the accumulation regions (831 to 833) correspond to one element. In this manner, a relationship between the sliding regions and the accumulation regions may be formed.

Referring to FIG. 9, the sliding region (910) in the input plane (900) includes reaction regions (911 to 919) of weights (w ₁ to w ₉ ). The offset of each of the reaction regions (911 to 919) may be determined based on the index of each of the weight elements (w ₁ to w ₉ ). For example, according to the description of FIG. 7, the offset of each of the reaction regions (911 to 919) may be defined as (W+2)*a+b. The offset may be determined based on the sliding regions (e.g., the origin of each sliding region). In this case, the offset of each of the reaction regions (911 to 919) may be 0, 1, 2, (W+2), (W+2)+1, (W+2)+2, (W+2)*2, (W+2)*2+1, (W+2)*2+2.

Input element vectors are extracted from the reaction areas (911 to 919) and stored in registers (r1 to r9). For example, a first input element vector of the reaction area (911) may be stored in register (r1), and a second input element vector of the reaction area (912) may be stored in register (r2). In this way, the input element vectors may be sequentially stored in the registers (r1 to r9).

Each of the input element vectors is multiplied by its corresponding element among the weight elements (w ₁ to w ₉ ) on an element-by-element basis, and thus weighted input element vectors can be generated. For example, a first input element vector of a reaction area (911) can be stored in a register (r1) and multiplied by a weight element (w ₁ ), and thus a first weighted input element vector can be generated. A second input element vector of a reaction area (912) can be stored in a register (r2) and multiplied by a weight element (w ₂ ), and thus a second weighted input element vector can be generated. The sizes of the reaction areas (911 to 919), the input element vectors, and the weighted input element vectors can correspond to a SIMD operation unit.

Through this process, weighted input element vectors generated can be accumulated to generate an accumulated vector corresponding to a sliding area (910). In addition, as this process is repeated for each sliding area, accumulated vectors corresponding to each sliding area can be generated, and the accumulated vectors can be gathered to form an accumulated plane. The accumulated plane and the accumulated vector can refer to different forms of accumulated data, and can be collectively referred to as accumulated data.

Referring to FIG. 10, an existing accumulated vector (hereinafter referred to as a first accumulated vector) is loaded into an output area (1011) within an output plane (1010) and stored in a register (r10). When a new accumulated vector (hereinafter referred to as a second accumulated vector) is generated through registers (r1 to r9), the first accumulated vector and the second accumulated vector are accumulated in the register (r10) and stored in the output area (1011).

Fig. 10 assumes that the process of storing an accumulated vector in the output area (1011) is performed at least once. For example, Fig. 10 can correspond to a situation in which a first accumulated vector is generated through a MAC operation between a first input plane and a first weight plane, each corresponding to a first input channel, and is stored in the output area (1011), and then a second accumulated vector is generated through a MAC operation between a second input plane and a second weight plane, each corresponding to a second input channel, and the first accumulated vector and the second accumulated vector are accumulated and stored in the output area (1011). If an initial value is stored in the accumulation area (1011), that is, if the accumulation vector is generated for the first time, the process of loading the accumulated vector in the accumulation area (1011) can be omitted, and the newly generated accumulated vector can be stored in the accumulation area (1011) without a separate accumulation operation.

When the accumulated vectors are repeatedly stored in the output area (1011) as many times as the number of input channels (the number of accumulations is one less than the number of input channels), the output element vector corresponding to the output area (1011) can be determined. In addition, when the process for the output area (1011) is also performed for the remaining output areas in the output plane (1010), the output plane (1010) can be determined. Therefore, the convolution operation of the accumulated method according to the embodiment can be implemented through SIMD.

Fig. 11 is a diagram illustrating a zero-skip process of a convolution operation in an accumulation manner according to one embodiment. Since the convolution operation according to the embodiments is performed in units of input planes (more specifically, in units of reaction areas within the input plane), zero-skip can be efficiently processed through software.

Referring to FIG. 11, multiplication results (1141 to 1143) can be generated according to the product between input elements in each reaction area (1121 to 1123) and each weight element (w ₁ to w ₉ ). In the embodiment of FIG. 11, it is assumed that the weight elements (w ₃ to w ₅ , w ₈ , w ₉ ) correspond to 0. Hereinafter, a weight element corresponding to 0 may be referred to as a zero weight element, and a weight element not corresponding to 0 may be referred to as a non-zero weight element. In this case, since multiplication results such as the multiplication result (1143) based on the zero weight element do not affect data of an accumulation plane or an output plane, operations on these multiplication results can be omitted.

FIG. 12 is a diagram illustrating a process of performing zero-skip using a predefined operation type according to one embodiment. Referring to FIG. 12, zero encoding is performed in step (1210). The number of non-zero weight elements included in weight elements can be determined through zero encoding. For example, in FIG. 12, the number of non-zero weight elements as a result of zero encoding can be determined as 4.

In step (1220), an operation type corresponding to the number of non-zero weight elements is selected from among the operation types, and data corresponding to the non-zero weight elements is loaded into a register. In FIG. 12, operation type 4 corresponding to the number of non-zero weight elements of 4 is selected. The operation types may be set to perform operations in a predetermined manner according to the number of non-zero weight elements. For example, operation types may be set for each case from a case where no non-zero weight element is included in the weight elements to a case where all of the weight elements correspond to non-zero weight elements. If the number of operation types is defined as N and the number of weight elements is defined as K*K, then N=K*K+1. FIG. 12 shows a case where K=3 and N=10.

Data loaded into a register may correspond to at least a part of an input plane. For example, an input element vector corresponding to a non-zero weight element may be loaded into a register. An offset corresponding to the input element vector may be determined based on an index of the non-zero weight element, and the input element vector may be extracted from the input plane through the determined offset and stored in the register. In Fig. 12, offsets of 0, 1, (W+2)+2, (W+2)*2 are determined based on w ₁ , w ₂ , w ₆ , and w ₇ corresponding to the non-zero weight elements, and input element vectors corresponding to the determined offsets may be loaded into registers of reg1, reg2, reg3, and reg4.

The predetermined method of operation may include performing a MAC operation between non-zero weight elements and data loaded into a register to generate accumulated data. Here, the data may be loaded into the register based on the number and offset of the non-zero weight elements. For example, a MAC operation may be performed between the non-zero weight elements and input element vectors stored in the register. In FIG. 12, weighted input element vectors corresponding to the multiplication results between the non-zero weight elements (w ₁ , w ₂ , w ₆ , w ₇ ) and the input element vectors stored in the registers (reg1 , reg2 , reg3 , reg4 ) are generated, and accumulated data may be generated according to the accumulation of the weighted input element vectors.

In step (1230), source code corresponding to each operation type may be executed. For example, source code corresponding to each of operation types 0 to 9 may be stored in a memory code area, and source code corresponding to the selected operation type may be loaded from the memory code area and executed. In Fig. 12, source code corresponding to operation type 4 may be executed. Since such source code occupies a small memory space, use of the source code does not significantly reduce memory efficiency.

FIG. 13 is a flow chart illustrating a convolution operation process of an accumulative method according to one embodiment. Referring to FIG. 13, in step (1301), a weight kernel (w ^d ) is obtained. d represents an index of an output channel, can be a natural number from 1 to D, and can be initially 1. The weight kernels can each correspond to an output channel. For example, the weight kernel (w ¹ ) can correspond to a first output channel, and the weight kernel (w ² ) can correspond to a second output channel.

In step (1302), the input plane (i _c ) is obtained, and in step (1303), the weight plane ( ) is obtained. c represents the index of the input channel, can be a natural number from 1 to C, and can be initially 1. The input planes and the weight planes can each correspond to the input channels. For example, the input plane (i ₁ ) and the weight plane ( ) can correspond to the first input channel, respectively, and the input plane (i ₂ ) and the weight plane ( ) can each correspond to a second input channel.

In step (1306), a MAC operation is performed. For example, at least some of the input elements in the input plane (i _c ) and the weight plane ( ) may generate accumulated data by accumulating multiplication results between at least some of my weight elements. For example, input element vectors corresponding to at least some of the weight elements in the input plane (i _c ) may be extracted, weighted input element vectors corresponding to multiplication results between the input element vectors and at least some of the weight elements may be generated, and the weighted input element vectors may be accumulated to generate accumulated data. At this time, offsets corresponding to the input element vectors may be determined based on indices of at least some of the weight elements, and the input element vectors may be extracted from the input plane based on the offsets.

In one embodiment, zero-skip can be implemented through steps (1304, 1305). In step (1304), zero encoding is performed, and in step (1305), an operation type is selected. When the number of non-zero weight elements is determined through zero encoding, an operation type corresponding to the number of non-zero weight elements is selected, and input elements corresponding to the non-zero weight elements can be loaded into a register. For example, input element vectors corresponding to the non-zero weight elements can be loaded into a register.

Depending on the selected operation type, operations may be performed according to a predetermined process. For example, the operations may include performing a multiplication between non-zero weight elements and input elements (e.g., input element vectors) in a register, and accumulating the multiplication result to generate accumulated data (e.g., accumulated vector). Accordingly, when the accumulated data is generated, the multiplication operation between zero weight elements and input elements may be omitted.

In step (1307), the output is accumulated. For example, accumulated data corresponding to the output of the MAC operation can be accumulated. For example, when the first iteration corresponding to c = 1 is performed, the input plane (i ₁ ) is obtained, and the weight plane ( ) is obtained, and at least some of the first input elements in the input plane (i ₁ ) and the weight plane ( ) the first accumulated data can be generated by accumulating the results of multiplications between at least some of my first weight elements. When the second iteration corresponding to c=2 is performed, the input plane (i ₂ ) is obtained, and the weight plane ( ) is obtained, and at least some of the second input elements in the input plane (i ₂ ) and the weight plane ( ) the second accumulated data can be generated by accumulating the multiplication results between at least some of my second weight elements. At this time, the first accumulated data and the second accumulated data can be accumulated. When the Cth iteration corresponding to c=C is performed, an output plane can be generated based on the sum of each accumulated data for each input channel.

In step (1308), c and C are compared. If c and C are different, for example, if c is smaller than C, c is increased by 1 in step (1309), and step (1302) is performed. If c and C are equal, d and D are compared in step (1309). If d and D are different, for example, if d is smaller than D, d is increased by 1 in step (1311), and step (1301) is performed. Through steps (1308, 1309), convolution can be performed on all input channels while the output channel is fixed, and through steps (1310, 1311), convolution can be performed on all output channels while changing the output channel.

FIG. 14 is a diagram illustrating a data processing method for a neural network according to one embodiment. Referring to FIG. 14, in step (1410), the processing device obtains a first input plane corresponding to a first input channel among input planes of input feature maps corresponding to input channels, respectively, obtains a first weight plane corresponding to the first input channel among weight planes of weight kernels corresponding to the input channels, respectively, accumulates multiplication results between at least some of the first input elements in the first input plane and at least some of the first weight elements in the first weight plane to generate first accumulated data, and generates a first output plane corresponding to the first output channel among output planes of output feature maps corresponding to output channels, respectively, based on the first accumulated data. In addition, the contents described through FIGS. 1 to 13 may be applied to the data processing method for a neural network.

FIG. 15 is a block diagram illustrating a processing device for processing data for a neural network according to one embodiment. Referring to FIG. 15, a processing device (1500) may receive input data and process an operation of a neural network related to the input data. For example, the operation of the neural network may include an object recognition operation and a user authentication operation. The processing device (1500) may perform one or more operations described or illustrated in this specification in relation to processing of the neural network, and may provide a processing result of the neural network to a user. The processing device (1500) may perform an accumulative convolution in the process of processing the operation of the neural network.

The processing device (1500) may include one or more processors (1510) and memory (1520). The memory (1520) may be coupled to the processor (1510) and may store instructions executable by the processor (1510), data to be computed by the processor (1510), or data processed by the processor (1510). The memory (1520) may include a non-transitory computer-readable medium, such as a high-speed random access memory and/or a non-volatile computer-readable storage medium (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

The processor (1510) may execute instructions for performing one or more operations described with reference to FIGS. 1 to 14. For example, when an instruction stored in the memory (1520) is executed in the processor (1510), the processor (1510) may obtain a first input plane corresponding to a first input channel among input planes of input feature maps corresponding to input channels, respectively, obtain a first weight plane corresponding to the first input channel among weight planes of weight kernels corresponding to the input channels, respectively, accumulate multiplication results between at least some of the first input elements in the first input plane and at least some of the first weight elements in the first weight plane to generate first accumulated data, and generate a first output plane corresponding to the first output channel among output planes of output feature maps corresponding to output channels, respectively, based on the first accumulated data.

FIG. 16 is a diagram illustrating an electronic device according to an embodiment of the present invention. Referring to FIG. 16, an electronic device (1600) may receive input data and process an operation of a neural network related to the input data. For example, the operation of the neural network may include an object recognition operation and a user authentication operation. The electronic device (1600) may perform the above-described cumulative convolution in the process of processing the operation of the neural network. The electronic device (1600) may include a processing device described through FIGS. 1 to 15, or may perform a function of a processing device described through FIGS. 1 to 15.

The electronic device (1600) may include a processor (1610), a memory (1620), a camera (1630), a storage device (1640), an input device (1650), an output device (1660), and a network interface (1670). The processor (1610), the memory (1620), the camera (1630), the storage device (1640), the input device (1650), the output device (1660), and the network interface (1670) may communicate with each other via a communication bus (1680).

The processor (1610) executes functions and instructions for execution within the electronic device (1600). For example, the processor (1610) may process instructions stored in the memory (1620) or the storage device (1640). The processor (1610) may perform one or more operations described with reference to FIGS. 1 to 15.

The memory (1620) stores information for processing the operation of the neural network. The memory (1620) may include a computer-readable storage medium or a computer-readable storage device. The memory (1620) may store instructions for execution by the processor (1610) and may store related information while software or an application is executed by the electronic device (1600).

The camera (1630) can capture still images, video images, or both. The camera (1630) can capture a facial area that a user inputs to attempt facial authentication. The camera (1630) can also provide 3D images that include depth information about objects.

The storage device (1640) comprises a computer-readable storage medium or a computer-readable storage device. According to one embodiment, the storage device (1640) can store a larger amount of information than the memory (1620) and can store the information for a longer period of time. For example, the storage device (1640) can comprise a magnetic hard disk, an optical disk, flash memory, a floppy disk, or any other form of non-volatile memory known in the art.

The input device (1650) can receive input from a user via traditional input methods such as a keyboard and mouse, and new input methods such as touch input, voice input, and image input. For example, the input device (1650) can include a keyboard, a mouse, a touch screen, a microphone, or any other device that can detect input from a user and transmit the detected input to the electronic device (1600).

The output device (1660) can provide output of the electronic device (1600) to the user via visual, auditory, or tactile channels. The output device (1660) can include, for example, a display, a touch screen, a speaker, a vibration generating device, or any other device capable of providing output to the user. The network interface (1670) can communicate with an external device via a wired or wireless network.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone, but those skilled in the art will appreciate that the processing device may include multiple processing elements and/or multiple types of processing elements. For example, the processing device may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

The software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and/or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal waves, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over network-connected computer systems, and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known to and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, flash memories, etc. Examples of the program commands include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on the above. For example, even if the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents, appropriate results can be achieved.

Claims (28)

A method for data processing for a neural network performed by a processor,
A step of obtaining a first input plane corresponding to a first input channel among input planes of input feature maps corresponding to each input channel;
A step of obtaining a first weight plane corresponding to the first input channel among the weight planes of the weight kernels corresponding to the input channels respectively;
A step of generating first accumulated data by accumulating multiplication results between at least some of the first input elements in the first input plane and at least some of the first weight elements in the first weight plane;
A step of storing the first accumulated data;
A step of obtaining a second input plane corresponding to a second input channel among the above input planes;
A step of obtaining a second weight plane corresponding to the second input channel among the above weight planes;
A step of generating second accumulated data by accumulating multiplication results between at least some of the second input elements in the second input plane and at least some of the second weight elements in the second weight plane;
a step of loading the first accumulated data to generate a sum of the first accumulated data and the second accumulated data; and
A step of generating a first output plane corresponding to the first output channel among output planes of output feature maps corresponding to the output channels based on the sum of the first accumulated data and the second accumulated data.
A method of processing data including: In the first paragraph,
The step of generating the above first output plane is
A step of generating the first output plane based on the sum of each accumulated data for each input channel including the first accumulated data.
How to process data. delete In the first paragraph,
The step of generating the above first output plane is
A step of generating the first output plane based on the sum of the first accumulated data and the second accumulated data,
How to process data. In the first paragraph,
The step of generating the above first accumulated data is
A step of extracting first input element vectors corresponding to at least some of the first weight elements in the first input plane;
generating first weighted input element vectors corresponding to a result of a multiplication between said first input element vectors and at least some of said first weight elements; and
A step of generating the first accumulated data by accumulating the first weighted input element vectors.
A method of processing data, comprising: In paragraph 5,
The step of extracting the first input element vectors above is
determining offsets corresponding to the first input element vectors based on at least some of the indices of the first weight elements; and
A step of extracting the first input element vectors from the first input plane based on the offsets,
How to process data. In paragraph 5,
The sizes of the first input element vectors and the first weighted input element vectors correspond to a SIMD (single instruction multiple data) operation unit.
How to process data. In the first paragraph,
When the first accumulated data is generated, a multiplication operation between zero weight elements corresponding to 0 among at least some of the first weight elements and at least some of the first input elements is omitted.
How to process data. In the first paragraph,
A step of determining the number of non-zero weight elements that do not correspond to 0 among the first weight elements; and
A step of selecting an operation type corresponding to the number of non-zero weight elements determined above among operation types set to perform operations in a predetermined manner, respectively.
A method of processing data, further comprising: In Article 9,
The step of generating the above first accumulated data is
A step of accumulating the multiplication results between the non-zero weight elements corresponding to at least some of the first input elements and at least some of the first weight elements based on the selected operation type to generate the first accumulated data,
How to process data. In Article 9,
The step of generating the above first accumulated data is
A step of extracting first input element vectors corresponding to the non-zero weight elements in the first input plane based on the indices of the non-zero weight elements;
generating first weighted input element vectors corresponding to a result of a multiplication between said first input element vectors and said non-zero weight elements corresponding to at least some of said first weight elements; and
A step of accumulating the first weighted input element vectors to generate the first accumulated data.
A method of processing data, comprising: A computer program stored on a medium for executing the method of any one of claims 1, 2, and 4 to 11 in combination with hardware. In a data processing device for a neural network,
processor; and
Memory containing instructions executable by said processor
Including,
When the above instructions are executed in the processor, the processor obtains a first input plane corresponding to a first input channel among input planes of an input feature map corresponding to each of the input channels, obtains a first weight plane corresponding to the first input channel among weight planes of weight kernels corresponding to each of the input channels, accumulates multiplication results between at least some of the first input elements in the first input plane and at least some of the first weight elements in the first weight plane to generate first accumulated data, and stores the first accumulated data.
Obtain a second input plane corresponding to a second input channel among the input planes, obtain a second weight plane corresponding to the second input channel among the weight planes, accumulate multiplication results between at least some of the second input elements in the second input plane and at least some of the second weight elements in the second weight plane to generate second accumulated data, and load the first accumulated data to generate a sum of the first accumulated data and the second accumulated data.
Generating a first output plane corresponding to the first output channel among the output planes of the output feature maps corresponding to the output channels based on the sum of the first accumulated data and the second accumulated data,
Data processing device. In Article 13,
The above processor
Generating the first output plane based on the sum of each accumulated data for each input channel including the first accumulated data,
Data processing device. delete In Article 13,
The above processor
A step of generating the first output plane based on the sum of the first accumulated data and the second accumulated data,
Data processing device. In Article 13,
The above processor
Extracting first input element vectors corresponding to at least some of the first weight elements in the first input plane, generating first weighted input element vectors corresponding to a multiplication result between the first input element vectors and at least some of the first weight elements, and accumulating the first weighted input element vectors to generate the first accumulated data.
Data processing device. In Article 17,
The above processor
Determining offsets corresponding to the first input element vectors based on at least some of the indices of the first weight elements, and extracting the first input element vectors from the first input plane based on the offsets.
Data processing device. In Article 17,
The sizes of the first input element vectors and the first weighted input element vectors correspond to a SIMD (single instruction multiple data) operation unit.
Data processing device. In Article 13,
When the first accumulated data is generated, a multiplication operation between zero weight elements corresponding to 0 among at least some of the first weight elements and at least some of the first input elements is omitted.
Data processing device. In Article 13,
The above processor
Among the first weight elements, the number of non-zero weight elements that do not correspond to 0 is determined, and among the operation types set to perform operations in a predetermined manner, an operation type corresponding to the determined number of non-zero weight elements is selected.
Data processing device. In Article 21,
The above processor
generating the first accumulated data by accumulating the multiplication results between the non-zero weight elements corresponding to at least some of the first input elements and at least some of the first weight elements based on the selected operation type;
Data processing device. In Article 21,
The above processor
Extracting first input element vectors corresponding to the non-zero weight elements in the first input plane based on the indices of the non-zero weight elements, generating first weighted input element vectors corresponding to a multiplication result between the first input element vectors and the non-zero weight elements corresponding to at least some of the first weight elements, and accumulating the first weighted input element vectors to generate the first accumulated data.
Data processing device. delete delete delete delete delete

Images