KR101965850B1 - A stochastic implementation method of an activation function for an artificial neural network and a system including the same - Google Patents

Abstract

The present invention relates to the implementation of an active function that constitutes an artificial neural network. Specifically, the present invention can implement four different types of active functions by adjusting the comparator inputs that make up the artificial neural network.

Description

[0001] The present invention relates to a stochastic implementation method of an active function used in an artificial neural network, and a system including the stochastic implementation method,

The present invention relates to a recognition system including an implementation method and an activation function of an activation function used in an artificial neural network.

Artificial Neural Network (ANN) is a statistical network that processes data in a manner similar to biological neural networks. Artificial neural networks can be used in various fields such as character recognition, image recognition, speech recognition, and face recognition. An artificial neural network can be implemented in a very large scale integrated circuit (VLSI). Similar to biological neural networks, artificial neural networks can include synapses and artificial neurons.

With the development of technology, the complexity of artificial neural networks can increase. Specifically, the number of synapses in the artificial neural network and the number of artificial neurons can be increased. As a result, there is a problem that the area and power consumption of the artificial neural network implemented by hardware increase.

[1] K. Leboeuf, R. Muscedere, and M. Ahmadi, "Performance analysis of table-based approximations of the hyperbolic tangent activation function," in IEEE IEEE International Conference on Computing and Communications, . 1-4. [2] A. H. Namin, K. Leboeuf, R. Muscedere, H. Wu, and M. Ahmadi, "Efficient hardware implementation of the hyperbolic tangent sigmoid function," 2009 IEEE International Symposium on Circuits and Systems, 2009, pp. 2117-2120. [3] Y. Ji, F. Ran, C. Ma, and DJ Lilja, "A Hardware Implementation of a Radial Basis Function Neural Network Using Stochastic Logic, , 2015, pp. 880-883. [4] M. Martincigh and A. Abramo, "A new architecture for digital stochastic pulse-mode neurons based on the voting circuit," IEEE Transactions on Neural Networks, vol. 16, pp. 1685-1693, 2005. [5] H. Li, B. Liu, X. Liu, M. Mao, Y. Chen, Q. Wu, et al., "The applications of memristor devices in next-generation cortical processor designs, Symposium on Circuits and Systems (ISCAS), 2015, pp. 17-20. [6] F. Tenore, RJ Vogelstein, R. Etienne-Cummings, G. Cauwenberghs, and P. Hasler, "A floating-gate programmable array of silicon neurons for central pattern generating networks," 2006 IEEE International Symposium on Circuits and Systems, 2006, pp. 4 pp. 3160. [7] M. S. J. Tomlinson, D. J. Walker, and M. A. Sivilotti, "A Digital Neural Network Architecture for VLSI," in Neural Networks, 1990 IJCNN International Joint Conference on, 1990, pp. 545-550. [8] S. Haykin, Neural Networks and Learning Machines (3rd Edition), Prentice Hall, 2009. [9] P. Nuzzo, F. D. Bernardinis, P. Terreni, and G. V. d. Plas, " Noise Analysis of Regenerative Comparators for Reconfigurable ADC Architectures, " IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 55, pp. 1441-1454, 2008. [10] Y. LeCun and C. Cortes, "The MNIST database of handwritten digits," ed, 1998.

The present invention can provide a method for implementing an active function without an analog-digital converter in the implementation of an active function through a conventional stochastic computing method.

A clock generator for generating a system clock, an analog random noise generator for generating an analog random signal, and a comparator for receiving analog random noise and neuron weights for performing a comparison, said comparator generating an active function, Function, and wherein the activation function is at least one of a step function, a sigmoid function, an identity function, or a ReLU function,

The present invention can provide a method for implementing an active function without an analog-digital converter in the implementation of an active function through a conventional stochastic computing method.

1 illustrates an example of an artificial neural network according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating an exemplary artificial neuron shown in FIG. 1. FIG.
3 illustrates an analog-to-digital converter for stochastic computing in accordance with one embodiment of the present invention.
Figure 4 shows a theoretical curve of average output spikes depending on the distribution.
5 shows a configuration of an analog random signal generator according to an embodiment of the present invention.
6 shows the DAC output distribution for each case (M = 8, reference voltage = 1.5V).
Figure 7 shows a module comprising a probabilistic computing active function array according to an embodiment of the present invention.
Figure 8 shows simulation results for four different types of activation functions based on stochastic computing.
9 shows an input terminal of a latch comparator according to an embodiment of the present invention.
10 and 11 illustrate test results for evaluating the performance of probabilistic computing active function arrays in accordance with an embodiment of the present invention.
Figure 12 shows the effect on DAC mismatch in recognition through the sigmoid activation function.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals are used to designate identical or similar elements, and redundant description thereof will be omitted. The suffix " module " and " part " for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role. In the following description of the embodiments of the present invention, a detailed description of related arts will be omitted when it is determined that the gist of the embodiments disclosed herein may be blurred. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. , ≪ / RTI > equivalents, and alternatives.

Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present application, the terms "comprises", "having", and the like are used to specify that a feature, a number, a step, an operation, an element, a component, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

1 illustrates an example of an artificial neural network according to an embodiment of the present invention.

An artificial neural network (ANN, 10) is an artificially constructed network similar to a biological neural network. Referring to FIG. 1, an artificial neural network 10 may include an input layer 11, a hidden layer 12, and an output layer 13.

The input layer 11, the hidden layer 12, and the output layer 13 may be connected to each other via a synapse. Biologically, synapses represent junctions between nerve cells and other nerve cells. Similarly, artificial neurons and other artificial neurons in the artificial neural network 10 can be coupled to each other via synapses.

The artificial neural network 10 may include a plurality of artificial neurons 100. The plurality of artificial neurons 100 receive input data from outside (X ₁ to X _I ), input data from an input neuron and concealed neurons that process and process the data, and output data (Y _i to Y _J ).

The input layer 11 may comprise a plurality of input neurons, the hidden layer 12 may comprise a plurality of concealed neurons, and the output layer 13 may comprise a plurality of output neurons. Here, the number of artificial neurons included in each of the input layer 11, the hidden layer 12, the long output layer 13, and the like is not limited to the illustrated one. The hidden layer 12 may contain more layers than those shown in Figure 1 and the number of layers described above is related to the accuracy or learning rate of the artificial neural network 10. [

For example, the input data may be image data representing a face or a handwriting of a person. The artificial neural network 10 recognizes the image data and can output what the image is.

FIG. 2 is a block diagram illustrating an exemplary artificial neuron shown in FIG. 1. FIG.

Referring to FIG. 2, the artificial neuron 100 may include a summation circuit 110 and an activation function circuit 120.

The summing circuit 110 may sum up the input signals A ₁ to A _K using the weights W ₁ to W _K. Each of the input signals may represent an output signal generated from any artificial neuron. And the intensity of each synapse (i.e., the degree of coupling between artificial neurons and other artificial neurons). Specifically, the summation circuit 110 may multiply the weights and the input signals, respectively, and then combine the multiplication results to generate a summation (B). Accordingly, when the weight is low, the weight of the input signal is low in the summation result B, and when the weight is high, the weight of the input signal is high in the summation result B. The summed result can be expressed by Equation (1).

The activation function circuit 120 may output the activation result (C) using the summing result and the activation function (AF). The activation result can be expressed by Equation (2).

3 illustrates an analog-to-digital converter for stochastic computing in accordance with one embodiment of the present invention.

Figure 4 shows a theoretical curve of average output spikes depending on the distribution.

One of the key elements of Artificial Neural Networks (ANN) is the Activation Function (AF). The activation function converts the weighted sum of neuron inputs into a probability of a firing rate. A hardware implementation of the active function requires complex circuitry and requires significant power consumption. This makes it difficult to integrate multiple neurons on a single chip.

The following describes a circuit technique for recognizing four different types of activation functions based on stochastic computing (step, identity, rectified-linear (ReLU), and sigmoid). The active function circuit according to an embodiment of the present invention is much simpler than the conventional one and consumes less power.

The probability of postsynaptic potential firing rate in an artificial neural network is modeled by the activation function. A variety of activation functions can be used. A commonly used function is a sigmoid function expressed by Equation (3).

Here, G is a gain factor that determines the slope of the curve.

The activation function can be implemented in two ways: an approximation and stochastic computing (see prior art documents 3 to 4).

In the approximation method, a look-up table (see prior art document 1) and piecewise linear approximations (PLAs) (see prior art document 2) are generally used. For accurate approximation, a sufficient number of linear segments is required for the PLA. Therefore, there is a problem that the hardware becomes complicated for accurate approximation.

In addition, a fixed form of the activation function may result in a decrease in synaptic plasticity. On the other hand, probabilistic computing methods can be implemented with relatively simple digital logic. The probabilistic computing method has advantages over the approximation method in hardware complexity.

Conventional probabilistic computing methods are concentrated on work in the digital domain. While probabilistic computing methods are more spatially efficient than approximation methods, implementations in the digital domain consume more power and area than comparatively simple analog implementations. In particular, multiplication requires multiple floating points to evaluate the activity of each synapse neuron. In this case, as the number of synapses increases, the number of floating points required increases exponentially, thereby worsening the problem of power consumption and area.

Further, additional analog-to-digital converters (ADCs) for reading the analog storage state when constructing an artificial neural network with analog synapse devices such as memristors (see prior art document 5) and floating-gats (see prior art document 6) ) (See prior art document 7).

The present invention proposes a simple architecture for VLSI implementation of one floating point active function for sensing analog voltage. The proposed architecture uses the central limit theorem for Gaussian random value generation and performs stochastic computing to generate an active function using only one regenerative clocked comparator.

Probabilistic computing according to an embodiment of the present invention is a result of applying a regenerative clock comparator, which is a digital bit stream whose output is represented by pulse rate (frequency) modulation, to a probability rule, one of which is an analog random noise. The average pulse rate represents the probability of a firing rate, in this case an active function. The configuration of probabilistic computing according to an embodiment of the present invention includes an analog-to-pulse converter (A2P) and an analog random noise generator (RNG).

In general, a comparator is used to implement a hard threshold for an active function in an artificial neural network. As shown in FIG. 3, random noise may be input to the input terminal of the regenerative clock comparator. The probability that the comparator output spike will be " HIGH (logic 1) " when the noise V _n is described as an equivalent Gaussian random variable with zero mean and standard deviation? _N is given by Equations 4 and 5 Is expressed.

If v _j is the output activity of the jth neuron, w _ij is the weight of the connection from jth neuron to i th, and z _i is the weighted sum of the i th neuron. If there are N attempts for equally given z _i and independent noise, then the digital output spike is determined by the binomial distribution B (N, P). The average rate S of spikes with mean ( _s ) and standard deviation ( _s ) is expressed as in Equations (6) and (7).

As shown in FIG. 4, the average rate of the comparator output depends on z _i . The slope of S monotonically decreases while σ _n increases. On the other hand, if? _N is very small, the function shown in Fig. 4 has a step function form.

On the other hand, when the random signal v _n is an equivalent uniform random variable having a zero average, the probability that the comparator output becomes " HIGH "

When receiving repeatedly a plurality of times compared to a given z _i, the comparator outputs the average rate (L) is the average time of Equation 6 P _G is replaced with P _U, respective average (oi _L) and standard deviation (σ _L of ) Have the same results as in Equations (6) and (7). Finally, L corresponds to the identity activation function as shown in FIG.

As shown in Equation (7), the deviation of the actual activation function depends on the number of trials. Increasing N to ensure proper accuracy increases power consumption. However, this can be solved to some extent because the artificial neural network provides high noise immunity.

An N-bit linear shift register (LFSR) generates a uniform distribution pseudo random bitstream of up to 2 ^N -1 cycles. This process is transformed into the Gaussian distribution by the central limit theorem. According to the central limit theorem, as the sample size increases, the resulting sample approximates a Gaussian distribution with zero mean and a unit variance. A digital-to-analog converter (DAC) is needed to convert the water sample to an analog voltage. The digital-to-analog converter has a linear one-to-one relationship between the digital input and the analog output, and reflects the Gaussian or uniform characteristics as they are.

5 shows a configuration of an analog random signal generator according to an embodiment of the present invention.

An XOR-based LFSR has two N-bit lengths. Each is divided into two groups, MSB and LSB. Therefore, N = 2M. The initial conditions of the LFSR are different to produce a better uncorrelated random signal. Likewise, the same group data is input to the adder to maintain uncorrection in decimation. Addition is performed through a simple full adder. After decimation, the remaining output is supplied to the M-bit DAC except for the LSB truncated at the output (M + 1) bits of the adder module. The MUX controls any bitstream type applied to the DAC.

6 shows the DAC output distribution for each case (M = 8, reference voltage = 1.5V).

The operating principle of a single active function structure has been described above. Hereinafter, a hardware implementation will be described.

Figure 7 shows a module comprising a probabilistic computing active function array according to an embodiment of the present invention.

The module 400 including the probabilistic computing active function array according to one embodiment of the present invention includes an analog random noise generator (RNG) 410, 32 latch comparators 420, A clock generator 430, and a multiplexer 440.

The analog random noise generator is the same as described in FIG. Specifically, the analog random noise generator includes a digital-to-analog converter 411 and a linear feedback shift register (LFSR) 412.

The latch comparator 420 has been described in FIG. The negative input nodes of the 32 latch comparators share the same analog RNG output. Parallel latching can be performed by simultaneously operating a plurality of latch comparators.

The clock generator 430 generates a system clock for use in the system.

The multiplexer 440 outputs the signals input from the plurality of latch comparators.

Figure 8 shows simulation results for four different types of activation functions based on stochastic computing.

In the first embodiment, when generating a step-type active function, the latch comparator performs a single comparison with the common mode voltage (V _CM ) of the analog RNG.

In the second embodiment, when generating a sigmoid-type active function, the latch comparator performs multiple comparisons with Gaussian noise.

In the third embodiment, when generating an identity function of an identity type, Gaussian distribution noise is replaced with uniform distribution noise, followed by a sigmoid type procedure.

In the fourth embodiment, when the ReLU type active function is generated, the reference voltage of the DAC increases from 0 to V _CM .

9 shows an input terminal of a latch comparator according to an embodiment of the present invention.

Latch comparators are a key component for probabilistic computing. The input terminal of the latch comparator is generally comprised of an NMOS pair or a PMOS pair. For single-ended circuits, the input voltage range is limited due to the threshold of the transistor. To solve this problem, as shown in Fig. 5, a complementary transistor is used as an input terminal. As a result, the PMOS or NMOS operates even if the input signal is too low or high. The size of the transistor is the result of minimizing the inherent thermal noise of the comparator. Since thermal noise follows the Gaussian distribution, transformations in the linear activation function (identity, ReLU) can occur. Increasing the transistor width can reduce the input reference noise.

10 and 11 illustrate test results for evaluating the performance of probabilistic computing active function arrays in accordance with an embodiment of the present invention.

10 illustrates a stochastic computing network according to an embodiment of the present invention.

As shown in Figure 10, the MNIST training test (prior art document 10) was used for testing. The sample consists of an 8-bit grayscale and a 28-by-28 pixel number of digits (0-9). The probabilistic computing network shown in FIG. 10 has 800 neurons (25 modules) in the visible region, 320 (10 modules) in the hidden layer, and 32 (1 module) neurons in the output layer. The size of the MNIST image is 784. However, the active function array according to an embodiment of the present invention is composed of 32 units, and the null value that does not affect the training is applied to the remaining 16 neurons of the visible region. Also, only 10 neurons in the output layer are used. The weights are trained by conventional super-bias runs based on back-propagation.

11 (a) shows the recognition rate according to the number of attempts when the sigmoid activation function is used.

11 (b) shows the recognition rate according to the number of attempts when the ReLU activation function is used.

In each case probabilistic computing sigmoid activation function and ReLU activation function are used. As shown in FIG. 11 (a), when the sigmoid activation function is used, the handwriting recognition is successful if at least 32 comparisons are performed. In addition, as shown in FIG. 11 (b), when the ReLU activation function is used, handwriting recognition is successful if at least 64 comparisons are performed.

Figure 12 shows the effect on DAC mismatch in recognition through the sigmoid activation function.

12 shows a case where DAC mismatch occurs by 20% as an example. Mismatches caused by process variations are generally considered to be a major cause of performance degradation of the VLSI system. However, it can be seen that DAC mismatch as shown in FIG. 12 does not affect probabilistic computing according to an embodiment of the present invention.

As described above, the present invention has been described with reference to particular embodiments, such as specific elements, and limited embodiments and drawings. However, it should be understood that the present invention is not limited to the above- And various modifications and changes may be made thereto without departing from the scope of the present invention. Accordingly, the spirit of the present invention should not be construed as being limited to the embodiments described, and all of the equivalents or equivalents of the claims, as well as the following claims, fall within the scope of the spirit of the present invention .

Claims (1)

A recognition system using an artificial neural network,
A clock generator for generating a system clock;
An analog random noise generator for generating an analog random signal; And
A comparator for receiving the analog random noise and the neuron weights and performing a comparison,
Wherein the comparator generates an active function of any one of a step type, a sigmoid type, an identity type, and a ReLU type, performs comparison through a generated activation function,
The comparator
Performing a single comparison with the common mode voltage of the analog random noise generator when generating the step type active function,
Performing multiple comparisons with Gaussian noise when generating the sigmoid type active function,
The Gaussian noise is replaced with the homogeneous noise when the identity function is generated,
A reference voltage of a digital-analog converter for changing the input random sample to an analog voltage when generating the ReLU type active function is increased from 0 to the common mode voltage to perform a comparison
Recognition system.

Images