JP6898778B2 - Machine learning system and machine learning method - Google Patents

Description

The present invention relates to a machine learning system and a machine learning method.

Patent Document 1 states, "A machine learning device includes an interlayer accelerator. The interlayer accelerator is based on an input vector of a first layer included in a neural network having three or more layers and a learning weight matrix of the first layer. It contains a plurality of interlayer units that generate an input vector of the second layer next to the first layer. Each of the plurality of interlayer units includes a coupled oscillator array and an activation function applicator. The array includes a plurality of oscillators that oscillate at a frequency corresponding to the difference between the plurality of elements of the input vector of the first layer and the plurality of elements of the row vector corresponding to any row of the learning weight matrix. The oscillating signal generated by the oscillator of the above is combined to obtain an arithmetic signal. The activation function applicator generates any element of the input vector of the second layer by applying the activation function to the arithmetic signal. To do. "

Non-Patent Document 1 describes a technique for reducing learning time in deep learning by performing parallel distributed processing using a plurality of computing nodes having high-performance hardware.

JP-A-2017-333385

Jeffrey Dean, Greg S. Corrado, RajatMonga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, MarcAurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. "Large scale distributed deep networks" . "In NIPS, 2012.

Recently, the development / research of deep learning is rapidly progressing, and it is expected to be applied to various fields such as image recognition, character recognition, voice recognition, smartphones, autonomous robots, and drones. In deep learning, learning is performed using a model such as a convolutional neural network (CNN (Convolutional Neural Network)) or a feedforward neural network (FFNN (Feedforward Neural Network)).

Here, in learning using a model, since the number of weighting coefficients to be handled is enormous, a large amount of time is required for learning. Therefore, for example, as disclosed in Non-Patent Document 1, a method has been proposed in which training data and weighting coefficients are updated in a distributed manner among a plurality of calculation nodes (hereinafter, referred to as a parallel distributed method).

However, in the parallel distribution method, it is necessary to share the weighting coefficient and the update amount of the weighting coefficient between the computing nodes, and a large amount of traffic is generated due to the data transfer performed between the computing nodes during learning, and the number of computing nodes is particularly large. The state of waiting for transfer frequently occurs, which becomes a bottleneck in shortening the learning time.

The present invention has been made in view of such a background, and an object of the present invention is to provide a machine learning system and a machine learning method capable of efficiently performing machine learning.

One of the present inventions for achieving the above object is a machine learning system that includes a plurality of information processing devices connected so as to be communicable and performs machine learning by a parallel distribution method using the plurality of information processing devices. When the information processing apparatus transmits and receives parameters used for learning a model in the machine learning to and from the other information processing apparatus, and transmits the parameters to the other information processing apparatus. A compression unit that differentially encodes and compresses the parameters, an expansion unit that expands the parameters received from the other information processing device, and a control device that is communicably connected to each of the information processing devices are provided. The control device has a quantization parameter control unit that reduces the value of the quantization parameter used by each of the information processing devices for the difference coding as the machine learning progresses .

In addition, the problems disclosed in the present application and the solutions thereof will be clarified by the column of the form for carrying out the invention and the drawings.

According to the present invention, machine learning can be performed efficiently.

It is a figure which shows the schematic structure of the learning system. It is a figure which shows an example of the information processing apparatus used for the realization of a calculation node. It is a figure explaining the schematic operation of a worker and a parameter server. It is a block diagram explaining the outline of the function of a worker and a parameter server. It is a figure explaining the spatial correlation of a machine learning model. It is a figure explaining the temporal correlation of the machine learning model. It is a figure which shows the relationship between the continuous update time and the weighting coefficient W for each of the adjacent positions of the machine learning model space. This is an example of a variable length code table. It is a flowchart explaining a learning process (parameter server side). It is a flowchart explaining a learning process (worker side). It is a processing block diagram explaining the details of the W compression part of a parameter server. It is a processing block diagram explaining the details of the W extension part of a worker. It is a processing block diagram explaining the details of the ΔW compression part of a worker. It is a processing block diagram explaining the details of the ΔW extension part of a parameter server. (A) is a graph showing an example of a change in the weighting coefficient W with respect to an increase in the number of updates, (b) is a graph showing a change in the number of workers with an increase in the number of updates, and (c) is a graph showing a change in the quantization parameter q with respect to the number of updates. It is a graph which shows. This is another configuration example of the learning system. Another configuration example of the learning system Another configuration example of the learning system

Hereinafter, embodiments will be described with reference to the drawings. In the following description, the same or similar configurations may be designated by the same reference numerals and duplicate description may be omitted.

[First Embodiment]
FIG. 1 shows a schematic configuration of an information processing system (hereinafter, referred to as learning system 1) that performs machine learning, which will be described as the first embodiment. As shown in the figure, the learning system 1 includes one parameter server 200 and a plurality of workers 100. In the following description, each of the worker 100 and the parameter server 200 may be referred to as a calculation node. Further, in the following description, machine learning will be described as being deep learning.

The parameter server 200 and the plurality of workers 100 are connected to each other so as to be able to communicate with each other via the communication line 50. The learning system 1 performs the processing in machine learning by a method (hereinafter, referred to as a parallel distribution method) in which a plurality of workers 100 and a parameter server 200 each share the processing.

FIG. 2 is an example of the information processing device 10 used to realize the calculation node. As shown in the figure, the information processing device 10 includes a processor 11, a main storage device 12, an auxiliary storage device 13, an input device 14, an output device 15, and a communication device 16. These are connected to each other so as to be able to communicate with each other via a communication means such as a bus.

In addition, all or a part of the information processing apparatus 10 may be configured by using a virtual resource such as a cloud server in a cloud system, for example. Of the configurations shown in the figure, the auxiliary storage device 13, the input device 14, and the output device 15 are not essential configurations.

The processor 11 is configured by using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), and the like. The processor 11 may be, for example, one of the cores constituting the multi-core processor. Further, the processor 11 may be one of the processors constituting the multiprocessor system.

The main storage device 12 is, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a non-volatile semiconductor memory (NVRAM (Non Volatile RAM)), or the like, and stores programs and data. The main storage device 12 may be mounted in the same module or package as the processor 11.

The auxiliary storage device 13 is, for example, a hard disk drive or an SSD (Solid State Drive).
), Optical storage devices (CD (Compact Disc), DVD (Digital Versatile Disc), etc.), storage systems, IC cards, SD memory cards, data reading / writing devices for recording media such as optical recording media, etc. is there. Programs and data stored in the auxiliary storage device 13 are loaded into the main storage device 12 at any time. The auxiliary storage device 13 may be independent of the information processing device 10 such as network storage.

The input device 14 is an interface that accepts data input from the outside, and is, for example, a data reading device from a recording medium (nonvolatile memory, optical recording medium, magnetic recording medium, optical magnetic recording medium, etc.), a keyboard, and the like. Mouse, touch panel, etc. Note that, for example, the information processing device 10 may be configured to receive data input from another device via the communication device 16.

The output device 15 is a user interface that provides data and information such as processing progress and processing results to the outside, and is, for example, a screen display device (liquid crystal display (Liquid Crystal Display), projector, graphic card, etc.), printing device, recording. A device for recording data on a medium. In addition, for example, the information processing device 10 may be configured to provide data such as a processing progress and a processing result to another device via the communication device 16.

The communication device 16 is a wired or wireless communication interface that realizes communication with other devices or elements, and is, for example, a NIC (Network Interface Card), a wireless communication module, or the like.

Returning to FIG. 1, the communication line 50 is, for example, a LAN (Local Area Network) or a WAN (Wide).
Area Network), the Internet, public communication networks, leased lines, etc. For example, communication line 5
As 0, it is effective to use InfiniBand, which has low latency and can output throughput in giga bps units. When the learning system 1 is realized by using, for example, a multi-core system, a multi-processor system, or the like, the communication line 50 may be configured by using a bus. All or part of the communication line 50 may be realized by either a wired system or a wireless system.

For example, if M computing nodes having one CPU and two GPUs are prepared, the CPU of one of the computing nodes is operated as the parameter server 200, and the GPU of each computing node is operated as the worker 100, the maximum is large. A learning system 1 including 2 × M workers 100 can be configured. Further, for example, if one computing node having one CPU and eight GPUs is prepared, the CPU is operated as a parameter server 200, and the GPU of each computing node is operated as a worker 100, a maximum of eight workers are used. A learning system 1 including 100 can be configured. In this case, data transfer can be performed using the high-speed internal bus at the computing node. Both the CPU and GPU can be used as both the worker 100 and the parameter server 200.

FIG. 3 is a diagram illustrating a schematic operation of the parameter server 200 and the worker 100. As described above, the learning system 1 performs the processing in machine learning in a parallel distributed system. Specifically, the training data used for machine learning is divided into each worker 100 (for example, the data is divided into the number of the distribution destination workers 100 (hereinafter referred to as N)). Each worker 100 obtains the update amount ΔW of the weighting coefficient of the model by applying the distributed training data to the machine learning model, and transfers the obtained update amount ΔW of the weighting coefficient to the parameter server 200 via the communication line 50. Send. On the other hand, the parameter server 200 updates the weighting coefficient W using the update amount ΔW of the weighting coefficient sent from each worker 100, and transfers the updated latest weighting coefficient W to each worker 100 via the communication line 50. Send. The learning system 1 advances machine learning by repeating the above processing.

The method of calculating the update amount ΔW of the weighting coefficient is not necessarily limited. For example, the worker 100 uses the stochastic gradient descent method (SGD: Stochastic Gradient Descent), the error back propagation method, the quasi-Newton method, the genetic algorithm, and the like described in Non-Patent Document 1 to update the weighting coefficient Δ.
Find W. The training data 104 shared by each worker 100 may be transmitted to each worker 100 in advance, or may be stored in a server device or the like shared between calculation nodes via a communication line 50 or the like.

Here, when machine learning is performed by the parallel distribution method, there is a problem that the amount of data (weight coefficient W and update amount ΔW of the weight coefficient) transferred between the parameter server 200 and the worker 100 is large. For example, if the amount of data transferred exceeds the line speed, a transfer wait will occur, leading to a decrease in throughput. This tendency becomes more remarkable as the number of workers 100 increases, and the large amount of data to be transferred becomes a bottleneck when increasing the number of workers 100 (parallel number) to shorten the processing time.

Therefore, the learning system 1 of the present embodiment differentially encodes and compresses the data (weight coefficient W and update amount ΔW of the weight coefficient) transferred between the parameter server 200 and the worker 100, thereby compressing the traffic between the calculation nodes. Reduce the amount to eliminate the above bottleneck. Further, since the efficiency (compression rate) of the difference coding depends on the machine learning model, the learning system 1 selects an appropriate method of the difference coding according to the machine learning model.

FIG. 4 is a block diagram illustrating the outline of the functions of the worker 100 and the parameter server 200. As shown in the figure, the worker 100 includes the functions of the W receiving unit 111, the W expanding unit 112, the ΔW calculation unit 113, the ΔW compression unit 114, and the ΔW transmitting unit 115. Further, the parameter server 200 includes the functions of the ΔW receiving unit 211, the ΔW expanding unit 212, the W updating unit 213, the W compression unit 214, and the W transmitting unit 215.

Each of the above-mentioned functions included in the worker 100 and the parameter server 200 is realized, for example, by the processor 11 reading and executing a program stored in the main storage device 12 and the auxiliary storage device 13. Further, these functions are realized by, for example, hardware (FPGA (Field-Programmable Gate Array), ASIC (Application Specific Integrated Circuit), etc.) included in the information processing apparatus 10.

Each function of the worker 100 will be described in order. The W receiving unit 111 receives the weighting coefficient W sent from the parameter server 200 via the communication line 50. The W stretching unit 112 stretches the weighting coefficient W received by the W receiving unit 111. The ΔW calculation unit 113 obtains the update amount ΔW of the weight coefficient using the weight coefficient W extended by the W extension unit 112. The ΔW compression unit 114 generates compressed data (hereinafter, also referred to as compression data (ΔW)) in which the update amount ΔW of the weighting coefficient obtained by the ΔW calculation unit 113 is differentially encoded and compressed. The ΔW transmission unit 115 transmits the compressed data (ΔW) generated by the ΔW compression unit 114 to the parameter server 200 via the communication line 50.

Subsequently, each function of the parameter server 200 will be described in order. The ΔW receiving unit 211 receives the compressed data (ΔW) sent from each worker 100 via the communication line 50. The ΔW expansion unit 212 expands the compressed data (ΔW) received by the ΔW reception unit 211, and restores the update amount ΔW of the weighting coefficient. The W update unit 213 updates the weight coefficient W based on the update amount ΔW of the weight coefficient restored by the ΔW extension unit 212. The W compression unit 214 generates data in which the weight coefficient W after updating by the W update unit 213 is compressed (hereinafter, also referred to as compressed data (W)). The W transmission unit 215 transmits the compressed data (W) generated by the W compression unit 214 to each worker 100 via the communication line 50.

The method of updating the weighting coefficient W by the W updating unit 213 is not particularly limited, but to give an example, the W updating unit 213 updates the weighting coefficient W by the following equation.
Wx, n + 1 = Wx, n --ηΔWx, n ・ ・ ・ Equation 1
Here, x is information for specifying the position where the weighting coefficient is given in the deep learning model space, n is the number of times the weighting coefficient is updated (that is, the updating time), and η is the learning coefficient. The learning coefficient η may be a constant, but for example, in order to accelerate the convergence of the weighting coefficient W, the value may be changed so as to decrease as the learning progresses.

Subsequently, a method of compressing and decompressing the weight coefficient W and the update amount ΔW of the weighting coefficient by the W stretching unit 112, the ΔW compression unit 114, the ΔW stretching unit 212, and the W compression unit 214 will be described. The learning system 1 efficiently compresses and expands the weighting coefficient W and the update amount ΔW of the weighting coefficient W by performing differential coding using the spatial correlation and the temporal correlation of the weighting coefficient W.

FIG. 5 is a diagram for explaining the spatial correlation of a convolutional neural network (hereinafter, also referred to as CNN (Convolutional Neural Network)) shown as an example of a deep learning model. FIG. 5A is an image diagram of the model space of CNN. Further, FIG. 5B is a graph showing the relationship between the number of updates n (horizontal axis) and the weighting coefficients Wx-1, n (vertical axis) at the position x-1, and FIG. 5C is an update. It is a graph showing the relationship between the number of times n (horizontal axis) and the weighting coefficients Wx, n (vertical axis) at the position x adjacent to the position x-1. From these graphs, when CNN is used as a deep learning model, the correlation (spatial correlation) of the weighting coefficient W becomes high between adjacent positions (difference between the two at the same time (update count n)). It can be seen that there is a property that is biased toward 0). That is, if the above difference is encoded by utilizing this property, the compression rate can be increased.

On the other hand, FIG. 6 is a diagram for explaining the temporal correlation of a feedforward neural network (hereinafter, referred to as FFNN (Feedforward Neural Network)) shown as an example of a deep learning model. FIG. 6A is an image diagram of the model space of FFNN. Further, FIG. 6B is a graph showing the relationship between the number of updates n (horizontal axis) and the weighting coefficients Wx-1, n (vertical axis) at the position x-1, and FIG. 6C is an update. It is a graph showing the relationship between the number of times n (horizontal axis) and the weighting coefficients Wx, n (vertical axis) at the position x adjacent to the position x-1. FIG. 6 (b)
As shown in (c), when FFNN is used as a deep learning model, the spatial correlation of the weighting coefficient W is low between adjacent positions, but the temporal variation of the weighting coefficient is generally gentle in both cases. , It can be seen that there is a property that the temporal correlation of the weighting coefficient W is high between consecutive times (for example, n and n-1) (the difference between the coefficient values at consecutive times is biased to 0). In other words, the compression rate can be increased by coding the above difference using the property of.

Similar to FFNN, other networks that tend to have a high temporal correlation include recurrent networks (RNNs (Recurrent Neural Networks)), restricted Boltzmann machines (RBMs), and autoencoders (AutoEncoder). ), Fully coupled network, etc. Similar to FFNN, the compression rate can be increased by utilizing the temporal correlation for these as well.

As described above, the weighting coefficient W of the model in deep learning has different spatial correlation and temporal correlation depending on the type of model. Therefore, the learning system 1 switches the difference coding method according to the type of the model (for example, when the model is CNN, the difference coding is performed by a calculation method focusing on spatial correlation, and the model is other than CNN. In this case, differential coding is performed by a calculation method that focuses on temporal correlation) to improve the compression rate.

In implementing such a mechanism, for example, information for specifying the type of model in deep learning (hereinafter, also referred to as model specific information) is shared between each worker 100 and the parameter server 200 (compression side and decompression side). Each worker 100 or the parameter server 200 switches the difference calculation method with reference to the model specific information when compressing or decompressing the weight coefficient W and the update amount ΔW of the weight coefficient.

Subsequently, the compression and decompression of the weighting coefficient W and the update amount ΔW of the weighting coefficient by the difference coding will be described in detail together with the figures shown in FIGS. 7 to 9.

FIG. 7 is a diagram showing the relationship between the continuous update time (n-1, n) and the weighting coefficient W for each of the adjacent positions (x-1 and x) in the deep learning model space. FIG. 7A is a graph showing the relationship between Wx-1, n-1 and Wx-1, n, and FIG. 7B is a graph showing the relationship between Wx, n-1 and Wx, n. ..

For example, when the deep learning model is CNN, the learning system 1 uses spatial correlation to compress and expand Wx-1, n once for the weighting coefficient W as shown in the following equation <Wx. The difference is obtained by subtracting -1, n> (hereinafter, the value once compressed and decompressed in this way is enclosed in <>) from Wx, n.
Difference of weight W = Wx, n− <Wx-1, n> ・ ・ ・ Equation 2

Further, as shown in the following equation, the learning system 1 similarly obtains a difference for the update amount ΔW of the weighting coefficient.
Difference in weight update amount ΔW = ΔWx, n− <ΔWx-1, n> ・ ・ ・ Equation 3

It should be noted that it is not the Wx-1, n or ΔWx-1, n itself, but the compressed and decompressed value (<Wx-1, n> or
<ΔWx-1, n>) is subtracted in order to match the values on the compression side and the decompression side in consideration of the quantization error. That is, since the decompression side receives the value including the quantization error, in order to match the value between the compression side and the decompression side, the value obtained by compressing and decompressing on the compression side (value reflecting the quantization error) is used. Must be generated. Hereinafter, the value that the compression side compresses and decompresses and reflects the quantization error is also referred to as a “reference value”.

When the model is other than CNN, the learning system 1 uses the temporal correlation to compress and expand Wx, n-1 once for the weighting coefficient W as shown in the following equation <Wx, n-1. Find the difference by subtracting> from Wx, n.
Difference of weight W = Wx, n− <Wx, n-1> ・ ・ ・ Equation 4

Further, as shown in the following equation, the learning system 1 obtains the difference for the update amount ΔW of the weighting coefficient in the same manner.
Difference in weight update amount ΔW = ΔWx, n− <ΔWx, n-1> ・ ・ ・ Equation 5

Subsequently, the learning system 1 generates quantization data for the difference obtained as described above based on the following equation using, for example, the quantization parameter q.
Quantization data = quotient of difference divided by quantization parameter q ... Equation 6

The value of the quantization parameter q may be the same for all the weighting coefficients W, but may be different for each weighting coefficient W, for example. For example, the quantization parameter q is set to a large value for the weighting coefficient W that changes drastically per update, while the quantization parameter q is set to a small value for the weighting coefficient W that changes little per update. By setting to, the compression coefficient can be improved.

FIG. 8 is an example of a variable length code table 800 (quantization table) referred to by the learning system 1 when coding the quantization data. As shown in the figure, the variable length code table 800 includes information in which the values 811 of the quantized data and the code 812 are associated with each other. The learning system 1 converts the quantization data into a variable length code with reference to the variable length code table 800. In the variable length code table 800, the smaller the value of the quantization parameter q, the shorter the code can be assigned to improve the compression rate.

In addition, when decompressing, the procedure opposite to that in the case of compression described above is followed. For example, the learning system 1 restores the compressed data (W) and the compressed data (ΔW) by referring to the variable-length code table 800, multiplies the compressed data by the quantization parameter q, and then sets the reference value. Compressed data (W) and compressed data (ΔW) are decompressed by adding them together.

Subsequently, the functions of the parameter server 200 and the worker 100 will be described in detail.

FIG. 9 is a flowchart illustrating a process (hereinafter, referred to as a learning process S900) performed by the parameter server 200 in the case of learning one time of deep learning. Hereinafter, the learning process S900 will be described with reference to the figure.

First, the parameter server 200 receives compressed data (compressed data (ΔW)) of the weight coefficient update amount ΔWx, n from the worker 100 via the communication line 50 (S911).

Subsequently, the parameter server 200 determines whether or not the deep learning model is CNN (whether the compressed data (ΔW) is encoded using spatial correlation or temporal correlation) (S912). .. When the parameter server 200 determines that the model is CNN (S912: YES), the process proceeds to S913. On the other hand, when the parameter server 200 determines that the model is not CNN (S912: NO), the process proceeds to S914.

When proceeding to the process of S913, the parameter server 200 decompresses the compressed data (ΔWx, n− <ΔWx-1, n>) using the reference value <ΔWx-1, n> and outputs <ΔWx, n>. Acquire (S915).

When proceeding to the process of S914, the parameter server 200 acquires the reference value <ΔWx, n-1>, decompresses the compressed data (ΔWx, n− <ΔWx, n-1>), and <ΔWx, n>. Is acquired (S915).

Subsequently, the parameter server 200 uses the weighting coefficient Wx, n according to Equation 1 and the like based on <ΔWx, n>.
Is updated to obtain Wx, n + 1 (S916).

Subsequently, the parameter server 200 compresses the acquired Wx, n + 1. The parameter server 200 determines whether or not the deep learning model is CNN (whether it is encoded using spatial correlation or temporal correlation) (S917), and a reference value < Obtain the Wx-1, n + 1> or reference value <Wx, n>, subtract the obtained reference value from Wx, n + 1, obtain the difference, compress the obtained difference, and compress the data (W). Is generated (S918, S919).

Subsequently, the parameter server 200 transmits the generated compressed data (W) to the worker 100 (S920).

FIG. 10 is a flowchart illustrating a process (hereinafter, referred to as a learning process S1000) performed by the worker 100 when learning a single sentence of deep learning. Hereinafter, the learning process S1000 will be described with reference to the figure.

First, the worker 100 receives compressed data (compressed data (W)) having weighting coefficients Wx and n from the parameter server 200 via the communication line 50 (S1011). The worker 100 then determines whether the deep learning model is CNN (whether the compressed data was encoded using spatial or temporal correlation) (1012). If the worker 100 determines that the model is CNN (S1012: YES), the process proceeds to S1013. If the worker 100 determines that the model is not a CNN (S1012: NO), the process proceeds to S1014.

When proceeding to the process of S1013, the worker 100 acquires the reference value <Wx-1, n>, decompresses the compressed data (Wx, n-<Wx-1, n>), and performs <Wx, n>. Acquire (S1015).

When proceeding to the processing of S1014, the worker 100 acquires the reference value <Wx, n-1>, decompresses the compressed data (Wx, n- <Wx, n-1>), and performs <Wx, n>. Acquire (S1015).

Subsequently, the worker 100 updates the update amount ΔWx, n of the weighting coefficient based on the acquired <Wx, n> by a stochastic gradient descent method or the like to obtain ΔWx, n + 1 (S1016).

Subsequently, the worker 100 compresses the obtained ΔWx, n + 1. The worker 100 determines whether or not the target model is CNN (whether it is encoded using spatial correlation or temporal correlation) (S1017), and a reference value is determined according to the determination result. Obtain <ΔWx-1, n + 1> or reference value <ΔWx, n>, obtain the difference by subtracting the obtained reference value from ΔWx, n + 1, compress the obtained difference, and compress the data (ΔW). ) Is generated (S1018, S1019).

Subsequently, the worker 100 transmits the generated compressed data (ΔW) to the parameter server 200 (S1020).

Next, each function of the W compression unit 214 of the parameter server 200, the W decompression unit 112 of the worker 100, the ΔW compression unit 114 of the worker 100, and the ΔW decompression unit 212 of the parameter server 200 will be described in detail.

FIG. 11 is a processing block diagram illustrating details of the W compression unit 214 of the parameter server 200. As shown in the figure, the W compression unit 214 includes a reference value specifying unit 2141, a difference calculation unit 2142, a quantization unit 2143, a variable length coding unit 2144, an inverse quantization unit 2145, and an expansion unit 2146. Further, the W compression unit 214 stores the model specific information, the quantization parameter q, and the reference value for each worker 100 described above.

The reference value specifying unit 2141 specifies the model type (spatial correlation, temporal correlation) based on the model specifying information.

The difference calculation unit 2142 obtains the difference between the weighting coefficients Wx, n updated in S916 and the reference value.

The quantization unit 2143 generates quantization data obtained by quantizing the above difference using the quantization parameter q.

The variable-length coding unit 2144 compresses the quantized data with reference to the variable-length code table 800, and outputs the compressed data (W).

The dequantization unit 2145 performs dequantization of the quantized data.

The stretching unit 2146 stretches the weighting coefficients Wx and n after dequantization using the reference value, and stores the stretched value as a new reference value.

FIG. 12 is a processing block diagram illustrating the details of the W extension portion 112 of the worker 100.

As shown in the figure, the W extension unit 112 includes a variable length restoration unit 1121, an inverse quantization unit 1122, a reference value specifying unit 1123, and an extension unit 1124. Further, the W expansion unit 112 stores the model specific information, the quantization parameter q, and the reference value of the worker.

The variable length restoration unit 1121 restores the compressed data (W) of the weighting coefficient W input from the W reception unit 111 with reference to the variable length code table 800, and generates the quantization data.

The dequantization unit 1122 dequantizes the quantization data using the quantization parameter q.

The reference value specifying unit 1123 specifies the model type (spatial correlation, temporal correlation) based on the model specifying information.

The expansion unit 1124 expands the weighting coefficients Wx and n by calculating the sum with the stored reference value, outputs the expanded value, and updates the stored reference value.

FIG. 13 is a processing block diagram illustrating the details of the ΔW compression unit 114 of the worker 100.

As shown in the figure, the ΔW compression unit 114 includes a reference value specifying unit 1141, a difference calculation unit 1142, a quantization unit 1143, a variable length coding unit 1144, an inverse quantization unit 1145, and an expansion unit 1146. Further, the ΔW compression unit 114 stores the model identification information, the quantization parameter q, and the reference value of the worker 100.

The reference value specifying unit 1141 specifies the model type (spatial correlation, temporal correlation) based on the model specifying information.

The difference calculation unit 1142 obtains the difference between the update amount ΔW of the weighting coefficient calculated by the ΔW calculation unit 113 and the reference value.

The quantization unit 1143 generates quantization data obtained by quantizing the above difference using the quantization parameter q.

The variable-length coding unit 1144 compresses the quantized data with reference to the variable-length code table 800, and outputs the compressed data (W).

The dequantization unit 1145 performs dequantization on the quantized data.

The extension unit 1146 adds the reference values to obtain an extension value of the weight coefficient update amount ΔWx, n, and updates the reference value.

FIG. 14 is a processing block diagram illustrating details of the ΔW extension unit 212 of the parameter server 200.

As shown in the figure, the ΔW expansion unit 212 includes a variable length restoration unit 2121, an inverse quantization unit 2122, a reference value identification unit 2123, and an expansion unit 2124. Further, the ΔW expansion unit 212 stores the model specific information, the quantization parameter q, and the reference value for each worker 100.

The variable-length restoration unit 2121 restores the compressed data (ΔW) of the update amount ΔWx, n of the weighting coefficient input with reference to the variable-length code table 800, and generates the quantization data.

The dequantization unit 2122 dequantizes the quantization data using the quantization parameter q.

The reference value specifying unit 2123 specifies the type of model (spatial correlation, temporal correlation) based on the model specifying information.

The expansion unit 1124 expands ΔWx, n by calculating the sum with the reference value stored for each worker 100, outputs the expanded value, and updates the stored reference value for each worker 100. To do.

By the way, FIG. 15A is a graph showing an example of the change in the value of the weighting coefficients Wx and n whose values fluctuate sharply during learning. As the learning progresses (as the number of updates n increases), the weighting coefficient W has a property of becoming more gradual, and the compression rate gradually increases as the learning progresses.

Therefore, for example, as shown in FIG. 15B, by increasing the number of 100 workers participating in parallel distribution as learning progresses, improvement in processing performance and effective utilization of computational resources can be expected. In this case, when a plurality of parameter servers 200 are used as described later (FIGS. 17 and 18), if the degree of parallelism is controlled so that the number of unutilized computing nodes and low operating rate computing nodes is reduced. , Processing performance can be improved and computational resources can be effectively used.

Further, for example, as shown in FIG. 15C, if the value of the quantization parameter q is reduced as the learning progresses and the quantization is performed finely, the learning accuracy is gradually increased while suppressing the fluctuation of the compression rate. You can go.

As shown in FIG. 16, for example, an information processing device connected to the communication line 50 (for example, a device provided with the hardware shown in FIG. 2, hereinafter referred to as a control device 300) participates in parallel distribution. By providing a function for controlling the number of workers 100 (hereinafter referred to as a worker number control unit 311) and a function for controlling the value of the quantization parameter q (hereinafter referred to as a quantization parameter control unit 312), a mechanism for controlling the value of the control and the quantization parameter q number of worker 100 to participate in the parallel distributed as described above can be easily realized. In this case, for example, the worker number control unit 311 increases the number of workers 100 to the extent that the data transfer amount does not exceed the line speed of the communication line 50. Further, the quantization parameter control unit 312 determines, for example, the value of the quantization parameter q so that the change in the amount of data transfer is small, and transmits the determined value to the parameter server 200 and the worker 100 at any time via the communication line 50 or the like. Notice.

17 or 18 is another example of the hardware configuration of the learning system 1. As shown in these figures, a plurality of parameter servers 200 may exist. FIG. 17 shows a case where one type of training data is learned by using a plurality of (two in this example) parameter servers 200. In this case, for example, the average value of the weighting coefficients and the weighted average value of all the parameter servers 200 after the learning is completed are set as the final learning result. On the other hand, FIG. 18 shows a case where a plurality of (two in this example) training data A and B are learned by the parameter server 200 associated with each of them. In this case, for example, the weighting coefficient of each parameter server 200 after the learning is completed is used as the learning result of each training data.

Although the embodiments have been described in detail above, it goes without saying that the present invention is not limited to the above embodiments and can be variously modified without departing from the gist thereof. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to add / delete / replace a part of the configuration of the above embodiment with another configuration.

For example, in the above method, the reference value is determined according to the deep learning model (S917 in FIG. 9, S1017 in FIG. 10 and the like), but the worker 100 and the parameter server 200 are automatically provided with a model having a high compression ratio. A function (model selection unit) for selecting and compressing may be provided. In that case, it is necessary to specify which model is selected and compressed on the decompression side. For example, this is information (for example, a bit) that identifies the model selected on the compression side to the worker 100 or the parameter server 200. ) Is added to the compressed data (W) and the compressed data (ΔW), and on the decompression side, which method is selected is specified by referring to the above information, and the compressed data (W) and the compressed data are specified by the specified method. A function (model specific information sharing unit) for extending (ΔW) may be provided. According to this, data transfer via the communication line 50 can be performed more efficiently.

Further, in the above method, the difference between the weighting coefficient W and the weighting coefficient ΔW is obtained between adjacent positions and between adjacent times (continuous updates), but between more distant positions and more distant times. The difference between the weighting coefficient W and the weighting coefficient ΔW may be obtained.

Further, each of the above configurations, functional units, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

Further, in each of the above figures, the control lines and information lines are shown as necessary for explanation, and not all the control lines and information lines in the implementation are necessarily shown. For example, in practice almost all configurations may be considered interconnected.

Further, the arrangement form of various functional units, various processing units, and various databases of various devices described above is only an example. The arrangement form of various function units, various processing units, and various databases can be changed to the optimum arrangement form from the viewpoints of hardware and software performance, processing efficiency, communication efficiency, and the like provided in various devices.

Further, the above-mentioned database configuration (schema, etc.) can be flexibly changed from the viewpoints of efficient use of resources, improvement of processing efficiency, improvement of access efficiency, improvement of search efficiency, and the like.

1 Learning system, 10 information processing equipment, 50 communication lines, 100 workers, 111 W receiver, 112 W extension unit, 1121 variable length restoration unit, 1122 inverse quantization unit, 1123
Reference value identification unit, 1124 extension unit, 1121 variable length restoration unit, 1122 inverse quantization unit, 1123 reference value identification unit, 113 ΔW calculation unit, 114 ΔW compression unit, 1141 reference value identification unit, 1142 difference calculation unit, 1143 quantum Quantization unit, 1144 variable length coding unit, 1145 inverse quantization unit, 1146 expansion unit, 115 ΔW transmission unit, 200 parameter server, 211 ΔW reception unit, 212 ΔW expansion unit, 2121 variable length restoration unit, 2122 inverse quantization unit , 2123 Reference value identification unit, 2124 Decompression unit, 213 W update unit, 214 W compression unit, 2141 Reference value identification unit, 2142 Difference calculation unit, 2143 quantization unit, 2144 variable length coding unit, 2145 inverse quantization unit, 2146 Expansion unit, 215 W transmission unit, 300 controller, 311 worker number control unit, 312 quantization parameter control unit, 800 variable length code table, S900 learning process, S1000 learning process

Claims (15)

A machine learning system that includes a plurality of information processing devices that are communicably connected and performs machine learning by a parallel distribution method using the plurality of information processing devices.
The information processing device
A transmission / reception unit that transmits / receives parameters used for learning a model in the machine learning to / from another information processing device.
When transmitting the parameter to the other information processing apparatus, a compression unit that differentially encodes and compresses the parameter, and a compression unit.
An extension unit that extends the parameter received from the other information processing device, and an extension unit.
A control device that is communicatively connected to each of the information processing devices,
With
The control device has a quantization parameter control unit that reduces the value of the quantization parameter used by each of the information processing devices in the differential coding as the machine learning progresses.
Machine learning system. The machine learning system according to claim 1.
The compression unit performs the difference coding by utilizing the spatial correlation or the temporal correlation of the model.
Machine learning system. The machine learning system according to claim 2.
The compression unit performs the difference coding by calculating the difference of the parameters at the respective adjacent positions in the model space of the model at the same time.
Machine learning system. The machine learning system according to claim 3.
The model is a convolutional neural network,
Machine learning system. The machine learning system according to claim 2.
The compression unit performs the difference coding by calculating the difference of the parameters at each of the consecutive times at the same position in the model space of the model.
Machine learning system. The machine learning system according to claim 5.
The models include feedforward neural networks (FFNNs (Feedforward Neural Networks)), recurrent networks (RNNs (Recurrent Neural Networks)), restricted Boltzmann Machines (RBMs), autoencoders, and all. One of the combined networks,
Machine learning system. The machine learning system according to any one of claims 1 to 6.
The parameter is at least one of the weighting factor of the model and the update amount of the weighting factor.
Machine learning system. The machine learning system according to any one of claims 1 to 6.
The control device includes a control unit that increases the number of information processing devices that participate in machine learning by the parallel distribution method as the machine learning progresses.
Machine learning system. The machine learning system according to any one of claims 1 to 6.
The information processing device
A model selection unit that selects and trains a model with a high compression rate by differential coding, and
A model-specific information sharing unit that shares model-specific information, which is information that identifies the selected model, by transmitting and receiving to and from other information processing devices.
With
The compression unit performs the difference coding of the parameters by a calculation method according to the model specific information.
The stretched portion stretches the parameter by a calculation method according to the model specific information.
Machine learning system. The machine learning system according to claim 7.
At least one of the information processing devices functions as a parameter server that updates the weighting factor.
At least one of the information processing devices functions as a worker for obtaining the update amount of the weighting coefficient.
The information processing device that functions as the parameter server updates the weighting coefficient based on the updating amount of the weighting coefficient sent from the worker.
The information processing device that functions as the worker obtains an update amount of the weighting coefficient based on the weighting coefficient sent from the parameter server.
Machine learning system. It is a machine learning method using a machine learning system that includes a plurality of information processing devices connected so as to be able to communicate and performs machine learning by a parallel distribution method using the plurality of information processing devices.
The information processing device
A step of transmitting and receiving parameters used for learning a model in the machine learning with the other information processing device, and
When the parameter is transmitted to the other information processing apparatus, the step of differentially encoding and compressing the parameter, and
A step of decompressing the parameter received from the other information processing device, and
And
A step in which the control device communicably connected to each of the information processing devices reduces the value of the quantization parameter used by each of the information processing devices in the differential coding as the machine learning progresses.
A machine learning method to perform. The machine learning method according to claim 11.
The information processing device performs the difference coding by utilizing the spatial correlation or the temporal correlation of the model.
Machine learning method. The machine learning method according to claim 12.
The information processing apparatus performs the difference coding by calculating the difference of the parameters at the adjacent positions at the same time in the model space of the model.
Machine learning method. The machine learning method according to claim 12.
The information processing apparatus performs the difference coding by calculating the difference of the parameters at each of the consecutive times at the same position in the model space of the model.
Machine learning method. The machine learning method according to claim 11.
The control device increases the number of information processing devices that participate in machine learning by the parallel distribution method as the machine learning progresses.
Machine learning method.

Images