CN108932671A - A kind of LSTM wind-powered electricity generation load forecasting method joined using depth Q neural network tune - Google Patents

Abstract

The present invention relates to a kind of LSTM wind-powered electricity generation load forecasting methods joined using depth Q neural network tune, and this approach includes the following steps：1) initial data for acquiring power system environment, chooses training set and forecast set；2) using LSTM as prediction model, the hyper parameter in prediction model is adjusted using DQN, specifically includes environmental parameter adjusting, state adjustment, movement selection, the intensified learning reward of regularized learning algorithm rate using the hyper parameter in DQN adjusting prediction model；3) training result is fed back to using experience recovery method and carries out parameter optimization in DQN by the prediction model after training set to be substituted into adjustment parameter, obtains optimal L STM prediction model；4) wind-powered electricity generation load prediction is carried out using optimal L STM prediction model.Compared with prior art, the present invention is not necessarily to need professional's de-regulation when different regions, is greatly improved forecasting efficiency.

Description

A LSTM Wind Power Load Forecasting Method Using Deep Q Neural Network Tuning Parameters

technical field

The invention relates to the technical field of electric power information, in particular to an LSTM wind power load forecasting method using a deep Q neural network for parameter tuning.

Background technique

Wind power load forecasting is an important part of power dispatching work, and the quality of its forecasting directly determines whether wind power can be connected to the grid system. Wind power load belongs to time series and is updated continuously with time changes. RNN (Recurrent Neural Networks, cyclic neural network) with LSTM (Long Short Term Memory networks, long short-term memory network) structure can effectively solve the problem of the disappearance of the time gradient of the RNN network, and due to the special network structure of RNN, it is useful for time series data. unique advantage.

The cyclic neural network has a special network structure, that is, the input of the hidden layer: in addition to the input layer input at the current moment, there is also the input layer input at the previous moment, as shown in Figure 1. In Figure 1, x, x1, and x2 are inputs at different time nodes, o, o1, and o2 are outputs corresponding to time respectively, and U and V are linear relationship matrices, which are shared in the entire RNN. The data related to wind power load, including time, wind speed of the wind field, real-time power, frequency, wind direction, and outdoor temperature, are used as the input of the prediction model, and the output result o is obtained through network calculation, and then o is compared with the corresponding wind load The error can be obtained by comparison. After the error is obtained, the gradient descent (Gradient Descent) and BPTT (Back-Propagation Through Time, time-based backpropagation) methods are used to train the model. BPTT uses backpropagation to solve the gradient and update the network parameter weight. Expanding the loop in RNN, the upper layer of neural network will pass information to the next layer, which is why RNN has advantages in processing time series data. There is no need to train all the parameters of the neural network, only one layer needs to be trained, and the parameters in it are all shared parameters.

Ordinary RNN may have the problem of gradient disappearance or gradient explosion in the face of long-term spans. LSTM can retain errors for reverse transmission along time and layers. LSTMs keep the error at a more constant level, allowing recurrent networks to learn over many time steps (more than 1000 time steps), thus opening the way to establish long-distance causal connections.

LSTMs store information in gated cells outside the normal flow of information in the recurrent network. These cells can store, write or read information, just like data in computer memory. The cells decide what information to store and when to allow it to be read, written, or cleared by the opening and closing of the gate. But unlike digital memory in a computer, these gates are analog, consisting of element-wise multiplications of sigmoid functions whose outputs all range between 0 and 1. Compared to digital storage, analog values have the advantage of being differentiable and thus suitable for backpropagation. These gates open and close based on the received signal, and similar to the nodes of a neural network, they use their own set of weights to filter information, and decide whether to allow information to pass according to its strength and input content. These weights, like the weights that modulate the input and hidden states, are adjusted through the learning process of the recurrent network. That is, the memory unit learns when data is allowed to enter, leave, or be removed through an iterative process of guessing, error backpropagation, and gradient descent to adjust weights. Its structure is shown in Figure 2. The bottom three arrows in Figure 2 represent the flow of information into memory cells from multiple points. The current input and the past cell state are not only sent into the memory cell itself, but also into the three gates of the cell, and these gates will determine how to process the input. Input enters (y ⁱⁿ ), when to clear the current cell state And when to let the unit state affect the network output at the current time step (y ^out ). S _c is the current state of the memory cell, and gy ⁱⁿ is the current input. Each gate can be opened and closed, and the gate recombines the on and off states at each time step. A memory cell can decide at each time step whether to forget its state, allow writing, or allow reading. The accuracy of LSTM predictions is directly related to hyperparameters. Therefore, appropriate hyperparameters enable the prediction model to reach or be very close to the global optimum. The existing technology usually adopts the Q-Learning algorithm, and its algorithm flow is as follows:

Initialize Q(s,a), a∈A(s), any value, and Q(terminal-state)=0;

Repeat (for each episode);

Initialize state S;

Repeat (for each step in the episode):

Use a certain policy, such as (ε-greedy) to select an action to execute according to the state S;

After executing the action, observe the reward and the new state S′;

Q(S _t ,A _t )←Q(S _t ,A _t )+a(R _t+1 +λmax _a Q(S _t+1 ,a)-Q(S _t ,A _t ))

S←S′

Loop until terminated.

α in the algorithm is the learning rate, which controls how much difference between the previous Q-value and the newly proposed Q-value is taken into account. Q refers to the corresponding Q value, and λ is the discount factor. When the discount factor is 0, the prediction model will tend to make decisions in the current table, and when it is 1, it will tend to expand the Q table by trying something that has not been done before. Generally speaking, the discount factor takes a number between 0 and 1 to balance rewards and exploration. R _t+1 +λmax _a Q(S _t+1 ,a) is the target Q value, and the Q-Learning algorithm mainly makes Q(S _t ,a) close to the target Q value. It is helpful to optimize the wind power forecasting model and adapt it to different regions. However, the wind power load is greatly affected by the regional environment, and the model parameters in different regions are quite different. When the prediction model is applied in different regions, professionals are required to adjust it, and the adjustment of the prediction model parameters is labor-intensive and inconvenient.

Contents of the invention

The purpose of the present invention is to provide an LSTM wind power load forecasting method that uses deep Q neural network parameter tuning to automatically adjust parameters, improve forecasting efficiency, and adapt to different regions in order to overcome the above-mentioned defects in the prior art.

The purpose of the present invention can be achieved through the following technical solutions:

An LSTM wind power load forecasting method using deep Q neural network parameter adjustment, the method includes the following steps:

S1: Collect the original data of the power system environment, select the training set and prediction set;

S2: Use LSTM as the prediction model, and use DQN to adjust the hyperparameters in the prediction model;

The specific content of using DQN to adjust the parameters in the prediction model includes environmental parameter adjustment, state adjustment, action selection and reinforcement learning rewards for adjusting the learning rate. The environment parameter adjustment combines the LSTM prediction model and a series of actions to form a Markov decision model; the realization of state adjustment, action selection and the reinforcement learning reward of adjusting the learning rate is based on the formed Markov decision model.

Among them, the specific content of environmental parameter adjustment is as follows:

The learning rate adjustment function f(x) is used to adjust the adaptive learning rate, and the regular parameter adjustment function g(x) is used to adjust the adaptive regular parameters. Assume (p, y) is a training sample, p is the input, including the learning rate x _t and regularization The parameter z _t , y is the expected output, and a is the actual output, then:

In the formula, n is the number of samples.

The specific content of status adjustment is as follows:

A feature vector containing six state features is used to represent the state. The feature vector of the six state features includes the hyperparameters expected to be adjusted, the candidate iteration target value, the maximum target value of the past M steps, the dot product between the descent direction and the gradient, and MI /MAX encoding, function evaluation numbers and alignment metrics, then:

Assume For the list of M lowest target values obtained at time t-1, the state [S _t ] encoding is determined by:

In the formula, when f(x _t ) is less than When the minimum value of , it is coded as 1, it is 0 in the previous M Fs, and -1 is used in other cases;

Given the state adjustment [st] _alignment is:

descending direction The expression is:

In the formula, for about the gradient of is the mean value of the learning rate.

The specific content of the action selection is:

For a given state, the method of resetting the learning rate or regularization parameter to the initial value after accepting iterations is used for action selection. When controlling the learning rate, there are two actions, keeping the learning rate or half the learning rate; for adjusting The regularization coefficient, except for two alternatives, is allowed to increase by a quarter.

The expression of the reinforcement learning reward r _id (f, x _t ) with adjusted learning rate is:

In the formula, f _lb is the target lower bound of the function value, and c is the target lower bound value.

S3: Substituting the training set into the prediction model after adjusting the parameters, feeding back the training results to DQN for parameter optimization, and obtaining the optimal LSTM prediction model;

The technique of experience playback is used for the training part. Every time the parameters of the neural network are updated, part of the previous training results are randomly retrieved from the data to update the DQN, and then obtain the optimal LSTM prediction model.

S4: Use the optimal LSTM forecasting model for wind power load forecasting.

Compared with the prior art, the present invention uses DQN to enable the prediction model to learn and adjust hyperparameters by itself, which can adapt to wind power prediction models in different regions, and requires professionals to adjust when there is no need for different regions, which greatly improves the prediction efficiency.

Description of drawings

Figure 1 is a structure diagram of RNN;

Figure 2 is a structure diagram of LSTM;

Fig. 3 is a schematic flow sheet of the method of the present invention;

Fig. 4 is the convergence effect figure when DQN learning rate is 0.05, e_greedy=0.01 in the embodiment of the present invention;

Fig. 5 is a comparison chart of prediction model accuracy using Q gradient descent and general gradient descent in the embodiment of the present invention;

FIG. 6 is a comparison diagram of the convergence effect of prediction model error reduction using Q gradient descent and general gradient descent in the embodiment of the present invention.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

Example

As shown in Fig. 3, the present invention relates to a kind of LSTM wind power load forecasting method that adopts depth Q neural network tuning parameter, and the main content of this method is:

1) Collect the original data of the power system environment, and select the training set and prediction set.

2) Using LSTM as the prediction model, using DQN to dynamically adapt to the hyperparameters in the prediction model to obtain the output value of the prediction model. The specific content of using DQN to dynamically adapt to the hyperparameters in the prediction model includes environmental parameter adjustment, state adjustment, action selection, and reinforcement learning rewards for adjusting the learning rate.

3) Substituting the training set into the prediction model after adjusting the parameters, feeding back the training results to DQN for parameter optimization, and obtaining the optimal LSTM prediction model;

4) Use the optimal LSTM forecasting model for wind power load forecasting.

Use LSTM as a prediction model, and use a deep Q neural network (DQN) to dynamically adapt to the hyperparameters in the prediction model. Whenever DQN makes an action, it takes a learning rate value, and this value will be simulated into the prediction model. Then there will be an output, and it will be rewarded and valued. At this time, DQN will include the action and the corresponding reward value in a Q table, and the amount of data is huge, so it is necessary to use a deep network to record the results of previous attempts. In this way, DQN can learn the skills of adjusting hyperparameters from the table. The environment, actions and rewards are defined as follows:

1. Environment

In the formula, the learning rate adjustment function f(x _t ) and the regular parameter adjustment function g(z _t ) make the gap between the output of the prediction model and the expectation narrow under different learning rates and regular parameters. In the two formulas, x _t and z _t represent the learning rate and regularization parameters respectively; (p, y) is a training sample, n is the number of samples, p is the input, including learning rate x _t and regularization parameter z _t ; y is the expected output, a for the actual output. Here, the adjustment function f(x _t ) and the regular parameter adjustment function g(z _t ) adopt the cross-entropy cost function. When the error is large, the weight update is fast, and when the error is small, the weight update is slow. Use f(x) to adjust the adaptive learning rate, and use g(x) to adjust the adaptive regularization parameters. The environment combines a predictive model with a series of actions and other necessary elements to form a Markov decision model, that is, the selection of actions depends only on the user's current state and has nothing to do with previous historical behaviors. Reinforcement learning rewards for state adjustment, action selection, and adjustment of the learning rate are done based on this Markov decision model.

2. Status

States are represented by eigenvectors with six state characteristics. The state features are the hyperparameters we expect to adjust, the candidate iteration target value, the maximum target value in the past M steps, the dot product between the descent direction and the gradient, MI/MAX encoding, function evaluation number, and alignment measure. The first four features can be Obtained directly, for the last two features, there are:

Assume is a list of the M lowest target values obtained at time t-1, and the state [S _t ] code is determined by the following formula:

In the formula, when f(x _t ) is smaller than the previous When it is still young, it is coded as 1, and it is 0 in the previous M Fs, and -1 in other cases. Let becomes the descending direction, and its expression is:

In the formula, for about the gradient of is the mean value of the learning rate.

Given the state adjustment [st] _alignment is:

Furthermore, in order to make state features independent of a specific objective function, all features are transformed into features in the interval [-1, 1].

3. Action

For a given state, the action is how to change the combination of learning rate and regularization parameters. In general, the learning rate and regularization parameters are very small. Therefore, a strategy of resetting the learning rate or regularization parameters to initial values after accepting iterations is adopted. Therefore, when controlling the learning rate, there are two actions: maintain the learning rate or half the learning rate. For adjusting the regularization coefficient, it is allowed to increase by a quarter in all but two choices.

4. Rewards

To tune the learning rate, the reward is defined as the inverse distance from the target net training loss to a lower bound. The reinforcement learning reward r _id (f, x _t ) for adjusting the learning rate is as follows:

In the formula, f _lb is the target lower bound of the function value, generally speaking, it can be set to zero as the target of the sum of loss functions. c is the target lower bound.

5. Experience playback

In the training part, the technique of experience playback is applied. Every time the parameters of the neural network are updated, a small batch of previous training results are randomly retrieved from the data to help train the neural network.

An experience contains a(s _i , a _i , r _i+1 , s _i+1 , label) ^j , where i refers to time step i; j refers to e_greed as j. These tuples are stored in the memory of experience E. In addition to updating DQN with most recent experience, a subset S ∈ E is pulled from memory for updating DQN in mini-batches.

6. Training results

In this embodiment, the LSTM prediction model is set to 6 inputs (time, wind speed of the wind field, real-time power, frequency, wind direction, outdoor temperature), one output (load), the recurrent neural network is set to 3 layers, and the hidden units are 128 , the activation function selects the softsign activation function, and there are 8932 load data in the experiment, of which 80% of the data are used as training data and 20% of the data are used as the test set. The discount factor λ is set to 0.99 at the beginning, and the exploration probability is set to 1, which decays uniformly to 0.1 within 100 steps. When the learning rate of DQN is 0.05 and e_greed is 0.01, as shown in the vertical axis of Figure 4, the loss of DQN begins to converge significantly. After 8 hours of training and iterating to 100 steps, DQN reaches an accuracy of about 40%, which is 10% higher than the baseline, and its loss convergence speed is also faster than the baseline (gradient descent), as shown in Figure 5. The horizontal axis is the number of iteration steps, and the vertical axis The axis is the accuracy rate of the prediction model. Compared with the gradient descent method, the Q gradient descent model has obvious advantages in both the rate of increase and the magnitude of the accuracy rate. In Figure 6, the horizontal axis is the number of iteration steps, and the vertical axis is the prediction model error of Q gradient descent and gradient descent, and the Q gradient descent model error decreases more rapidly. Due to the limitation of computing power, we iterated for 100 steps, but we can see that the learning ability of DQN still makes the accuracy of the target network rise rapidly and the convergence of its error is also very good.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any worker familiar with the technical field can easily think of various equivalents within the technical scope disclosed in the present invention. Modifications or replacements shall all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (8)

1. it is a kind of using depth Q neural network tune join LSTM wind-powered electricity generation load forecasting method, which is characterized in that this method include with Lower step：

1) initial data for acquiring power system environment, chooses training set and forecast set；

2) using LSTM as prediction model, the hyper parameter in prediction model is adjusted using DQN；

3) training set is substituted into the prediction model after adjustment parameter, training result is fed back to and carries out parameter optimization in DQN, is obtained Optimal L STM prediction model；

4) wind-powered electricity generation load prediction is carried out using optimal L STM prediction model.

2. a kind of LSTM wind-powered electricity generation load forecasting method joined using depth Q neural network tune according to claim 1, special Sign is, includes environmental parameter adjusting, state tune using the particular content that DQN adjusts the parameter in prediction model in step 2) Whole, movement selection and the intensified learning reward of regularized learning algorithm rate, environmental parameter, which is adjusted, combines LSTM prediction model and a series of Movement forms a Markovian decision model, the reality of the intensified learning reward of state adjustment, movement selection and regularized learning algorithm rate Now based on the Markovian decision model.

3. a kind of LSTM wind-powered electricity generation load forecasting method joined using depth Q neural network tune according to claim 2, special Sign is that the particular content that environmental parameter is adjusted is：

Adaptive learning rate is adjusted using learning rate adjustment function f (x), is adjusted using regular parameter adjustment function g (x) and adapts to canonical Parameter, it is assumed that (p, y) is a training sample, and p is input, including learning rate x_tWith regular parameter z_t, y is desired output, a For reality output, then have：

In formula, n is number of samples.

4. a kind of LSTM wind-powered electricity generation load forecasting method joined using depth Q neural network tune according to claim 3, special Sign is that the particular content of state adjustment is：

Indicate that state, the feature vector of six state features include that expectation is adjusted using the feature vector comprising six state features Dot product, MI/MAX volume between whole hyper parameter, candidate iterative target value, past M step maximum target value, descent direction and gradient Code, function review number and alignment metric, then have：

IfFor the list of M obtained minimum target values of time t-1, state [S_t] coding determined by following formula：

In formula, as f (x_t) be less thanMinimum value when, be encoded to 1, then take 0 in M F before, other situations take -1；

To the adjustment [st] that does well_alignmentFor：

5. a kind of LSTM wind-powered electricity generation load forecasting method joined using depth Q neural network tune according to claim 4, special Sign is, descent directionExpression formula be：

In formula,ForAboutGradient,For the mean value of learning rate.

6. a kind of LSTM wind-powered electricity generation load forecasting method joined using depth Q neural network tune according to claim 5, special Sign is, act the particular content that selects for：

For given state, using the method that learning rate or regularization parameter are reset to initial value after receiving iteration Movement selection is carried out, when Schistosomiasis control rate, there are two movements, keep learning rate or half learning rate；For adjustment Regularization coefficient allows it to increase a quarter other than two kinds of selections.

7. a kind of LSTM wind-powered electricity generation load forecasting method joined using depth Q neural network tune according to claim 6, special Sign is that the intensified learning of regularized learning algorithm rate rewards r_id(f, x_t) expression formula be：

In formula, f_lbFor the target lower bound of functional value, c is target floor value.

8. a kind of LSTM wind-powered electricity generation load forecasting method joined using depth Q neural network tune according to claim 1, special Sign is that the particular content of step 3) is：

The skill that experience replay is used to training part, when being updated each time to the parameter of neural network, in data The training result before part is randomly transferred, for updating DQN, and then obtains optimal L STM prediction model.