CN112149347B - Power distribution network load transfer method based on deep reinforcement learning - Google Patents

Abstract

The invention provides a power distribution network load transfer method based on deep reinforcement learning. The method comprises the following steps: the power distribution network fails and starts load transfer; inputting real-time state information of the power distribution network to an intelligent agent, calculating a motion evaluation vector, and selecting corresponding motion according to a motion strategy based on the motion evaluation vector; the intelligent agent executes the action on the power distribution network, evaluates the action and the state of the power distribution network after the action, calculates rewards Reward according to constraint conditions and objective functions, determines the value of Done according to the rewards Reward and ending rules, and updates parameters of the intelligent agent; judging whether to end the sequence action according to the end flag bit. The method utilizes the deep reinforcement learning to improve the fault emergency recovery capability and reliability of the power distribution network, and the power distribution network load transfer algorithm based on the deep reinforcement learning avoids a large number of calculation and power network simulation iteration during faults, improves the load transfer speed and ensures that the power distribution network has higher reliability.

Description

Distribution network load transfer method based on deep reinforcement learning

Technical field

The present invention relates to the technical field of distribution network fault processing, and in particular to a distribution network load transfer method based on deep reinforcement learning.

Background technique

With the rapid development of my country's national economy, especially the gradual expansion of the scale of electricity consumption in the tertiary industry, the proportion of small and medium-sized users and residents has gradually increased, the structure of power loads has undergone some changes, the number of nodes in the distribution network has increased significantly, and the number of lines has increased significantly. They are also getting longer and longer, the structure becomes more complex, and the probability of failure increases accordingly. Therefore, after a fault occurs in the distribution network, the line fault can be removed by adjusting the opening and closing status of the network switch, isolating the fault and transferring the load in the fault affected area to reduce the scope of the fault impact, thereby overall improving the economy and safety of the power grid operation. .

At present, the methods proposed by many domestic and foreign scholars for load transfer can basically be divided into the following categories: heuristic algorithms, mathematical optimization methods, expert system methods and artificial intelligence algorithms. The above algorithms can all obtain feasible transfer solution output, but they all have certain flaws.

For example, heuristic algorithms based on intuition or experience constructs and simulating thinking logic. It is based on the remaining capacity of the contact switch and the location of the power outage area. It attempts to use simple operations to provide a solution at once. The optimality of the solution is difficult to achieve. , it is easy to fall into the local optimal solution, and the quality of the solution depends very much on the initial state of the network. Although this method does not require too many power flow calculations and has relatively good real-time performance among various current algorithms, it still requires After conducting multiple power flow calculations to select a solution, it still cannot meet the real-time requirements of load transfer in the distribution network.

Mathematical optimization algorithms that describe the distribution network reconstruction problem using simplified mathematical models, such as the optimal flow pattern method, close each loop and then open the knife gate with the smallest current. When the distribution network structure is large, complex and dimensionally When the number is large, it is necessary to repeatedly calculate until it becomes stable, and the problem of "combination explosion" will occur; its optimization of the power grid simulation process causes many uncertainties in the solution process, which has a greater impact on the accuracy of the final result. Because the mathematical optimization method is relatively simple, it cannot take into account the complex large power grid. Moreover, its calculation is from local to whole, and it is easy to fall into the local optimal solution. The calculation process also takes a lot of time, resulting in excessive power outage time. It cannot meet the real-time requirements of distribution network load transfer.

The expert system method can automatically generate the operation plan required to restore the fault and save it in the library. It has good real-time performance and wide applicability, and can be applied to solve the plan when the network is large. However, the establishment and integration of the expert system's library is time-consuming and laborious, and there are many types of faults in practice, making it impossible to record all situations.

Traditional artificial intelligence algorithms mainly include random search algorithms and supervised learning algorithms. Random search algorithms such as tabu search algorithm, particle swarm search algorithm, and genetic algorithm have many calculations, a large amount of calculations, and a long solution time. The optimal solution may appear or not converge, and they cannot balance the solution speed and the global optimal solution. . Supervised learning algorithms such as neural network methods need to learn based on past experience. It is easy to find the global optimal solution when there are sufficient samples, but it is difficult to obtain better training results when there is a lack of labeled data. This type of method is a method of searching for the optimal solution based on obtaining fault information after a fault occurs. A large number of iterative calculations and power flow solutions are required. If the initial solution is far from the optimal solution, a lot of time will be spent searching for the optimal solution. solution, it is impossible to provide a better solution for the system in a short time.

Contents of the invention

Embodiments of the present invention provide a distribution network load transfer method based on deep reinforcement learning to overcome the problems of the existing technology.

In order to achieve the above object, the present invention adopts the following technical solutions.

A distribution network load transfer method based on deep reinforcement learning, including:

Step 1. Initialize the main neural network Q (S, A, ω, α, β) and the target network T (S, A, ω ^* , α ^* , β ^* ) with the same network structure as the main neural network Q, initialize Experience experience pool R, discount factor γ, learning rate L _r , target network update frequency N _replace , sampling number N _batch , set the end state flag Done = 0, the main neural network Q, target network T and experience pool constitute Intelligent agents in distribution networks;

Step 2. If the distribution network fails, load transfer begins;

Step 3. Read the real-time status information of the distribution network, and input the real-time status information of the distribution network to the main agent. The intelligent agent calculates the evaluation value of each action based on the real-time status information of the distribution network;

Step 4: The agent selects the corresponding action according to the action strategy based on the evaluation value of each action;

Step 5. The agent performs the action on the distribution network, obtains the state S′ of the distribution network after the action, evaluates the action on the distribution network and the state after the action, and calculates the reward Reward according to the constraints and objective function. The reward Reward and the end rule determine the value of Done. After completing a distribution network switching action, this distribution network switching action is stored in the experience pool R as an experience sample e=(s, a, r, s′);

Step 6: Randomly sample the sampling number N _batch experience samples from the experience pool R, calculate the target value using the discount factor γ according to the sampled experience samples, and minimize the loss function based on the target value and the learning rate L _r Update the parameters ω, α, β in the main neural network Q (S, A, ω, α, β);

Step 7. After the main neural network has been updated N _replace times, use the parameters ω, α, β of the main neural network Q to update the parameters ω ^* , α ^* , β ^* of the target network T:

Step 8: Determine whether to end the sequence action based on the end flag bit Done. Done=0, return to step 4; Done=1, exit the loop, and the load transfer process of this distribution network is completed.

Preferably, the step 1 also includes:

Define the system state space, action space and reward function in the distribution network load transfer operation. The interaction between the agent and the distribution network environment is represented by the array [S, A, P(s, s′), R(s, a), Done] represents, where S represents the state space composed of possible states of the distribution network, A represents the possible action set, P(s, s′) represents the transition probability from the distribution network state s to s′, R( s, a) is the action a taken in state s, which triggers the relevant reward, which is fed back to the agent. Done is the flag of the end state. The agent actively chooses to terminate this decision or is suspended due to violation of constraints. When the environment terminates and the operation continues, Done is set to 1, and during normal decision-making steps, Done remains 0;

The state space is defined as an array S = [V, I, SW, F]. V is a voltage vector group, which is used to represent the voltage values of all phases at each node in the distribution network. V _in is the voltage value of the i-th node. The voltage value of the n-th phase; I is the current vector group, which is used to represent the current value of each phase in all lines in the distribution network, I _in is the current value of the n-th phase of the i-th line; SW is the distribution The status value vector of all switches in the power grid, SW _i is the status of the i-th switch, 0 means open, 1 means closed; F is a vector representing the fault status of the distribution network line, F _i is the line numbered i The fault status, 0 means normal, 1 means fault has occurred.

Preferably, the step 1 also includes:

The intelligent agent adopts the Dueling-DQN algorithm. The Dueling-DQN algorithm uses a deep neural network to perform calculations. The deep neural network includes a main neural network Q and a target network T. The main neural network Q and the target network T include : Common hidden layer, value function V and advantage function B;

The common hidden layer of value function V and advantage function B uses a 2-layer neural network to extract the characteristics of the input state quantity. The first layer has 30*N _feature neurons, where N _feature is the number of input state quantities. All neurons directly receive fully connected input of state data, and a bias is added, and the activation function is the Relu function; the second layer is fully connected to the first layer, and also has 30*N _feature neurons;

The agent uses the Dueling-DQN algorithm to calculate the output results of the main neural network Q and the target network T, and calculates the evaluation value of each action.

Preferably, the real-time status information of the distribution network is read in step 3, and the real-time status information of the distribution network is input to the main agent. The intelligent agent calculates the real-time status information of the distribution network based on the real-time status information of the distribution network. The evaluation value of each action includes:

The value function V in the main neural network Q and the target network T is related to the state S and has nothing to do with the action A. It is a scalar, denoted as V (S, ω, α). The advantage function B is related to the state S and the state S at the same time. It is related to action A, which is a vector whose length is the number of actions, denoted as B(S, A, w, β). The value function of the agent is expressed as:

Q(S,A,ω,α,β)=V(S,ω,α)+B(S,A,ω,β)

Among them, ω is the network parameter of the public part, and α is the network parameter of the unique part of the value function, and β is the network parameter of the unique part of the advantage function. The final output of the Q network is composed of the output of the price function network and the output of the advantage function network. The output is obtained by linear combination;

The advantage function part has been centralized, and the actual combination formula used is as follows:

in Represents the set of all actions,/> That is, find the number of elements in the set. Q(S, A, ω, α, β) calculated using the above formula is a vector with a length of the number of actions. Each element in it represents each action in the state S. evaluation value.

Preferably, the agent in step 4 selects the corresponding action according to the action strategy based on the evaluation value of each action, including:

The agent selects the corresponding action according to the action strategy based on the action evaluation vector, and selects the optimal action in the non-exploration mode, which is the action with the highest evaluation value Q; in the exploration mode, it adopts the ε-greedy random greedy strategy, that is, taking a random Number x, if x < ε, select the action with the highest evaluation value Q as this action; if x > ε, select a random action from all actions, and ε is the set parameter.

Preferably, the intelligent agent in step 5 performs the action on the distribution network, including:

The action A is a number, and its range is an integer from 0 to 2N _switch . When action A is 2N _switch , it means no action is taken and the decision is exited. This decision is over; when action A is 0 to 2N _switch -1 , calculate the following for action A:

x=A%2

Where x is the remainder obtained by dividing A by 2. The meaning of this formula is as follows:

Each action is to operate a switch or exit directly. If you exit, this decision-making ends.

Preferably, in step 5, the state S′ of the distribution network after the action is obtained, the action of the distribution network and the state after the action are evaluated, the reward Reward is calculated according to the constraints and the objective function, and the reward Reward and the end are calculated according to the constraint conditions and the objective function. Rules determine the value of Done, including:

Constraints for setting up a distribution network include:

The voltage is maintained within the allowable range of ±7% deviation. For voltages that exceed this range, set the voltage penalty value P _Volt = -10, and set the end flag Done to 1; for voltages that do not exceed this range, set the voltage penalty value. P _Volt = 0;

When the current passing through the line and transformer is greater than its limit value, set the current penalty value P _Lim = -10, and set the end flag Done to 1; for the current that does not exceed its limit value, set the current penalty value P _Lim = 0;

Set the agent's ring network penalty P _Loop as:

Set the agent’s invalid action penalty P _Act as:

The objective functions for setting up the distribution network include:

Set the load loss evaluation value E _Loadloss according to the proportion of the lost load:

Among them, L _loss is the power loss load value, L _total is the total load of the entire power system, and the calculated E _Loadloss value is between -2 and 2;

Evaluation value E _Num for the number of switch operations:

Among them, A _Num is the total number of switches that have changed in this decision, L _Num is the total number of switches, and the calculated E _Num value is between -1 and 1.

The evaluation value E _Loss of the line loss of the distribution network:

Among them, Line is the total number of non-outage lines, I _i is the actual current of the i-th line, R _i is the resistance of the i-th line and the transformer, and S is the total power of the entire network;

For nodes whose voltage does not exceed the range of ±7%, the voltage offset evaluation value E _Vot of the line:

Among them, N is the total number of nodes without power outage, and pu _i is the voltage per unit value of node i;

The reward function given by the environment is composed of the sum of the above evaluation values, namely Reward:

Reward＝P _Volt +P _Lim +P _Loop +P _Act +E _Loadloss +E _Num +E _Loss +E _Vot .

Preferably, the sampling number N _batch experience samples are randomly sampled from the experience pool R in step 6, and the target value is calculated using the discount factor γ according to the sampled experience samples, based on the target value and the learning rate. L _r updates the parameters ω, α, β in the main neural network Q (S, A, ω, α, β) by minimizing the loss function, including:

Randomly sample N _batches of experience samples e _i = (s _i , a _i , r _i , s′ _i ) from the experience pool R, N _batch =20, and calculate the target value By minimizing the loss function Update the parameters ω, α, β in the main neural network Q (S, A, ω, α, β), and use the RMSProp algorithm to find the update degree of the parameters. The learning rate L _r is 0.1. For the main neural network Q, once Update represents a learning process of the agent.

It can be seen from the technical solutions provided by the above embodiments of the present invention that the method of the present application uses deep reinforcement learning to improve the fault emergency recovery capability and reliability of the distribution network. The distribution network load transfer algorithm based on deep reinforcement learning avoids It eliminates a large number of calculations and grid simulation iterations during faults, improves the speed of load transfer, and makes the distribution network more reliable. Using reinforcement learning algorithms, through training and experience learning, when a fault occurs, there is no need to spend a lot of time on simulation calculations and analysis, and load transfer decisions can be made directly by analyzing real-time running big data, which can provide faster and better transfer decisions. Provide strategies.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

In order to explain the technical solutions of the embodiments of the present invention more clearly, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. Those of ordinary skill in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of the mapping relationship between load transfer decision-making and reinforcement learning provided by the embodiment of the present application;

Figure 2 is a structural diagram of a neural network provided by an embodiment of the present application;

Figure 3 is a processing flow chart of a distribution network load transfer method based on deep reinforcement learning provided by an embodiment of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and cannot be construed as limitations of the present invention.

Those skilled in the art will understand that, unless expressly stated otherwise, the singular forms "a", "an", "the" and "the" used herein may also include the plural form. It should be further understood that the word "comprising" used in the description of the present invention refers to the presence of stated features, integers, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, Integers, steps, operations, elements, components and/or groups thereof. It will be understood that when we refer to an element being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Additionally, "connected" or "coupled" as used herein may include wireless connections or couplings. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

It will be understood by one of ordinary skill in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It should also be understood that terms such as those defined in general dictionaries are to be understood to have meanings consistent with their meaning in the context of the prior art, and are not to be taken in an idealized or overly formal sense unless defined as herein. explain.

In order to facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation to the embodiments of the present invention.

Due to a certain degree of lag in the construction of the distribution network, the capacity margin of power equipment is relatively small, which increases the difficulty of load transfer in the distribution network. Various sudden power outages require timely proposal of transfer plans. Therefore, for There are higher requirements for the algorithm's computing speed and applicability, and existing algorithms have certain limitations. Most of the existing algorithms perform temporary simulation calculation analysis after a fault occurs, and rarely use the distribution network to run real-time information big data, which consumes a long time; or adopt methods to simplify the simulation process to speed up the calculation, but it is difficult to take into account both good and bad results. Distribution network operation safety and economy.

The embodiment of the present invention adopts a reinforcement learning algorithm. Through training and experience learning, when a fault occurs, there is no need to spend a lot of time on simulation calculation and analysis. The load transfer decision can be made directly by analyzing real-time running big data, which can provide faster results. Better offloading strategies.

Taking the real-time status information of the distribution network as input data, the agent uses the deep reinforcement learning Dueling-DQN algorithm to make decisions and select actions. After the action, it transfers to a new state, uses constraints and objective functions to evaluate the action, and evaluates the action. The agent rewards or punishes, and when the transfer is completed through a series of operations, it stops the operation to obtain the final operation strategy.

Figure 1 is a schematic diagram of the mapping relationship between load transfer decision-making and reinforcement learning provided by the embodiment of this application. The interactive relationship between the distribution network environment and the intelligent agent will be described in detail below in conjunction with Figure 1.

First, it is necessary to define the environment in reinforcement learning, that is, the system state space, action space and reward function in the load transfer operation of the distribution network. The interaction between the agent and the distribution network environment is represented by the array [S, A, P(s, s′), R(s, a), Done], where S represents the state space composed of the possible states of the distribution network, A Represents a set of possible actions, P(s, s′) indicates the transition probability from distribution network state s to s′, R(s, a) takes action a in state s, triggering related rewards, It is fed back to the agent. Done is the flag bit of the end state. When the agent actively chooses to terminate this decision-making or is terminated by the environment due to violation of constraints to continue the operation, Done is set to 1. During normal decision-making steps, Done remains 0.

A. State space

The state space is defined as an array S = [V, I, SW, F]. V is a voltage vector group, which is used to represent the voltage values of all phases at each node in the distribution network. V _in is the voltage value of the i-th node. The voltage value of the n-th phase; I is the current vector group, which is used to represent the current value of each phase in all lines in the distribution network, I _in is the current value of the n-th phase of the i-th line; SW is the distribution The status value vector of all switches in the power grid, SW _i is the status of the i-th switch, 0 means open, 1 means closed; F is a vector representing the fault status of the distribution network line, F _i is the line numbered i The fault status, 0 means normal, 1 means fault has occurred.

B. Action space

Facing the real-time changing distribution network, the reinforcement learning agent needs to perform corresponding operations on the switches in the distribution network to control the status of the distribution network. The agent can decide how to perform the next action based on the current distribution network status and reward function. Action space A is a number, and its range is an integer from 0 to 2N _switch . When action A is 2N _switch , it means no action is taken and exits. This decision-making is over; when action A is 0 to 2N _switch -1, Make the following calculation for A:

x=A%2

Where x is the remainder obtained by dividing A by 2. The meaning of this formula is as follows:

Each action is to operate a switch or exit directly. If you exit, this decision-making ends.

C. Reward function

After the intelligent agent takes the selected action on the environmental distribution network, it will obtain the environment's evaluation of this action. The present invention uses this evaluation as a reward for the intelligent agent. The reward is mainly divided into the constraint part and the objective function part, so that the operation can achieve the most economical operating cost while ensuring the normal operation of the distribution network.

(1) Constraints:

The operation and control of the distribution network must first consider the safe operation of the distribution network and the safety of users' electricity consumption. After the transfer, the voltage and current quality of each node of the distribution line meet the requirements, and the voltage should be maintained within the allowable range of ±7%. Within, for voltages outside this range, a high penalty P _Volt is imposed, and the end flag Done is set to 1.

When the transmission capacity exceeds the limit value of the line and transformer, the power equipment will not be able to guarantee normal operation and may easily cause secondary faults. Therefore, the present invention compares the passing current of the line and transformer with its limit value. If the passing current exceeds the limit, it is deemed that If the transmission capacity of the device exceeds the limit, a high penalty P _Lim will be imposed, and the end flag Done will be set to 1.

When a loop appears in the distribution network after the agent operates, it can appear for a short time as an intermediate transition state, but is not allowed to appear as a long-term operating state. Therefore, the ring penalty P _Loop should consider the action state.

When the agent takes an invalid operation, such as performing a closing action on a closed switch, or performing an opening action on an already open switch, or performing an action on a faulty open line, the action is considered invalid and an invalid action penalty will be given. P _act .

(2) Objective function:

Under the condition that the action can meet the constraints, the normal power supply in the downstream power loss area should be restored as much as possible. Therefore, the load loss evaluation value E _Loadloss is set according to the proportion of the lost load.

Among them, L _loss is the power loss load value, L _total is the total load of the entire power system, and the calculated E _Loadloss value is between -2 and 2.

The action of the switch will have an impact on the life of the switch. There may be some switches that require manual operation. When the number of actions is too many, it will not only increase the probability of operational errors, but also the recovery time of the user's power supply may not meet the requirements. Moreover, This will cause the structure of the medium-voltage distribution network to change too much, making it more difficult for the distribution network to return to its original operating mode after the fault is eliminated or the maintenance is completed. Therefore, we should try to reduce the frequent operation of the switch and reduce the operating costs caused by the switch action. E _Num is the evaluation value of the number of actions.

Among them, A _Num is the total number of switches that have changed in this decision, and L _Num is the total number of switches. The calculated E _Num value is between -1 and 1.

Considering the economic operation of the distribution network, after completing the action, it is necessary to evaluate the line loss of the distribution network. The impedance model of the live line is used for evaluation. E _Loss is the line loss evaluation value.

Among them, Line is the total number of non-outage lines, I _i is the actual current of the i-th line, R _i is the resistance of the i-th line and the transformer, and S is the total power of the entire network. The right end of the formula is the calculated approximate line loss rate. Since the line loss rate of the distribution network and the base layer is often between 5% and 12%, in order to keep the value of E _Loss at approximately -1 to 0, the line loss is The rate is magnified -10 times as the line loss evaluation value.

For nodes whose voltage does not exceed the range of ±7%, the E _Vot evaluation value is used to measure the degree of voltage deviation to ensure that the distribution network after transfer has good voltage quality.

Among them, N is the total number of nodes that are not out of power, and pu _i is the voltage per unit value of node i. Since the result calculated by the formula on the right is less than 0.07, and most voltage values deviate not more than 0.05, in order to make the value of E _Vot Keep it at approximately -1 to 0 and magnify it 20 times.

The reward function given by the environment is composed of the sum of the above evaluation values, namely Reward.

Reward＝P _Volt +P _Lim +P _Loop +P _Act +E _Loadloss +E _Num +E _Loss +E _Vot

D.End condition

If the action causes the voltage to exceed the limit or the equipment transmission capacity to exceed the limit, the action round will be forcibly ended and the action will be deemed to have failed. The end flag is Done = 1; if the distribution network after the action restores the load of all fault-free areas, and there is no If the voltage exceeds the limit or the equipment transmission capacity exceeds the limit, the action round will be judged by the environment as having completed the supply transfer, and the current round will automatically end with the end flag Done = 1; but in special circumstances, such as when some tie lines have insufficient capacity , it is necessary to remove the fault-free load to ensure the quality of power supply, or there are multiple faults that prevent power transfer. At this time, the environment cannot judge whether the power transfer is completed by restoring all non-faulty loads. When the agent believes that the current state is not better When taking action, the agent can choose to end the current round and exit, and the end flag is Done=1. In other cases, Done=0, so that the agent can continue to perform actions.

The processing flow chart of a distribution network load transfer method based on deep reinforcement learning provided by the embodiment of the present invention is shown in Figure 3, and includes the following processing steps:

Step 1. Initialize the parameters ω, α, β of the main neural network Q and the parameters ω ^* , α ^* , β ^* of the target network T, initialize the experience pool R, discount factor γ, learning rate L _r , and target network update frequency N _replace , sampling number N _batch , Done=0.

In the initialization phase, in addition to initializing the main neural network Q (S, A, ω, α, β), another target network T (S, A, ω ^* , α ^* , β ^* ) with the same structure as the Q network is also needed. , the function of this network is mainly to find the error for the main neural network to learn.

Step 2: If a fault occurs in the distribution network, load transfer begins.

Step 3. Read real-time status information such as distribution network node voltage per unit, line current, switch opening and closing status, switch failure status, etc. After processing, the status vector S is obtained and input into the main neural network Q. The agent passes the DuelingDQN algorithm Calculate the action evaluation vector.

The intelligent agent in the distribution network load transfer method based on deep reinforcement learning in the embodiment of the present invention can use Deep Q Network and its evolutionary algorithms DoubleDQN and DuelingDQN. After comparison and testing, the DuelingDQN algorithm is used in the decision-making process of load transfer. has the best performance, so this article will introduce the reinforcement learning agent model using the DuelingDQN algorithm.

The DuelingDQN algorithm uses a deep neural network to obtain the Q values of all actions in Q-learning. Its deep neural network part has the ability to evaluate actions and train and learn. Its neural network structure is shown in Figure 2.

In the deep neural network part of the DuelingDQN algorithm of the present invention, the common hidden layer of the value function V and the advantage function A adopts a 2-layer neural network to extract the characteristics of the input state quantity. The first layer has 30*N _feature neurons. element, where N _feature is the number of input state quantities. All neurons directly accept fully connected input of state data, and a bias is added. The activation function is the Relu function; the second layer is fully connected to the first layer. , similar to the first layer structure, there are 30*N _feature neurons.

The value function neural network and the advantage function neural network each have 2 layers. The first layer is fully connected to the output of the common hidden layer. There are 30*N _feature neurons, and a bias is added. The activation function is the Relu function; value The second layer of function V has 1 neuron, which is fully connected to the first layer, has a bias but no activation function, and directly outputs the result. The second layer of advantage function A is fully connected to the first layer, with N _action neurons, and the results are also directly output. Finally, the output results of the two neural networks are calculated using the above formula to obtain the final Q value.

DuelingDQN's optimization of the DQN algorithm is reflected in the fact that DuelingDQN considers dividing the Q network into two parts. The first part is only related to the state S and has nothing to do with the specific action A to be taken. This part is called the value function (ValueFunction) part, which is A scalar, denoted as V (S, ω, α). The second part is related to both the state S and the action A. This part is called the Advantage Function part, which is a vector whose length is the number of actions, denoted as B(S, A, w, β), then the final calculation formula for the evaluation value of each action is:

Q(S,A,ω,α,β)=V(S,ω,α)+B(S,A,ω,β)

Among them, ω is the network parameter of the common part, α is the network parameter of the unique part of the value function, and β is the network parameter of the unique part of the advantage function. The final output of the Q network is obtained by a linear combination of the output of the price function network and the output of the advantage function network. The value of this action can be directly evaluated. However, this formula cannot identify V(S, ω, α) and B( in the final output. S, A, ω, β). In order to reflect this identifiability, the advantage function part has been centralized. The actual combination formula used is as follows:

in Represents the set of all actions,/> That is, to find the number of elements in the set, the right side of the formula subtracts the average value of the vector elements from the original vector A to obtain the new dominance function A. Q(S, A, ω, α, β) calculated using the above formula is a vector with a length of the number of actions, in which each element represents the evaluation value of each action in the state S.

The purpose of the target network T with the same structure as the main neural network Q is to overcome the oscillation problem during the training process caused by the random fluctuation of samples. Two deep neural networks T and Q with the same structure but different parameters are used. The Q network has the latest Parameters must be updated every time they learn, and the T network only updates once after N _replace actions.

Step 4. The agent selects the corresponding action according to the action strategy based on the action evaluation vector. The non-exploration mode selects the optimal action, that is, the action with the highest evaluation value Q; the exploration mode selects the optimal action or random action a based on ε-greedy.

Among the agents under training, in order to enable the agent to have the ability to jump out of the local optimal solution and conduct global exploration, an ε-greedy random greedy strategy is adopted, that is, a random number x is taken. If x < ε, the highest evaluation value Q is selected. The action is regarded as this action; if x>ε, a random action is selected from all actions. And ε continues to increase with the number of training rounds. When the number of training rounds is sufficient, the parameters in the deep neural network almost no longer change. At this time, ε is 1, and the best action is selected every time.

Step 5. The environment executes the action and obtains the post-action state S′. It evaluates the action and the post-action state, calculates the reward Reward based on the constraints and objective function, determines the value of Done based on the end rule, and completes a distribution network switching action. Finally, this distribution network switching action is stored in the experience pool R as an experience sample e=(s, a, r, s′).

Step 6. Randomly sample N _batches of experience samples e _i = (s _i , a _i , r _i , s′ _i ) from the experience pool R, usually N _batch =20, and calculate the target value By minimizing the loss function Update the parameters ω, α, β in the main neural network Q (S, A, ω, α, β), and use the RMSProp algorithm to find the degree of parameter update. The parameter learning rate L _r of the algorithm determines the degree of parameter update. , that is, the learning speed of the neural network, its value is usually 0.001. An update to the main neural network Q represents a learning process of the agent.

Step 7. Whenever the main neural network is updated N _replace times, usually N _replace =200, the parameters ω, α, β of the main neural network Q are used to update the parameters ω ^* , α ^* , β ^* of the target network T:

ω ^* , α ^* , β ^* ←ω, α, β

Step 8: Determine whether to end the sequence action based on the end flag bit Done. Done=0, return to step 4; Done=1, exit the loop, this load transfer decision is completed, and enter the next step.

The above is a single-step action, and a complete load transfer is likely to consist of multiple sequential switching actions. Therefore, it is judged whether to end the sequence action based on the end flag bit Done. If Done=0, it means that the distribution network still needs to continue to act to complete the supply transfer, then re-read the real-time operation information of the distribution network, input the new state quantity into the Q network for recalculation, and enter the next action decision-making process; if Done=1, this action decision is stopped,

Step 9: Wait for the next distribution network failure to enter a new load transfer decision-making process and go to step 2.

In summary, this application provides a distribution network load transfer method based on deep reinforcement learning, which uses real-time operating data of the distribution network to make load transfer decisions, and uses deep reinforcement learning to improve distribution network faults. Emergency recovery capabilities and reliability, while ensuring the safe and stable operation of the distribution network and the safety of users' electricity consumption, maximize the multi-faceted optimization of voltage quality, distribution network operation and economic efficiency. At the same time, the distribution network load transfer algorithm based on deep reinforcement learning avoids a large number of calculations and grid simulation iterations during faults, improves the speed of load transfer, shortens the power outage time in non-fault areas, and makes the distribution network more efficient. reliability.

The present invention uses a reinforcement learning algorithm called Dueling-DQN algorithm. Compared with commonly used reinforcement learning algorithms such as Q-learning algorithm and DQN algorithm, it can more accurately identify the status characteristics of the distribution network and achieve a more accurate load transfer decision-making scheme.

In the embodiment of the present invention, the present invention uses the reinforcement learning artificial intelligence algorithm to obtain real-time information analysis of the operating distribution network for load transfer decision-making, and can provide the best control strategy in a short time. Using reinforcement learning algorithms, through training and experience learning, when a fault occurs, there is no need to spend a lot of time on simulation calculations and analysis, and load transfer decisions can be made directly by analyzing real-time running big data, which can provide faster and better transfer decisions. Provide strategies.

Those of ordinary skill in the art can understand that the accompanying drawing is only a schematic diagram of an embodiment, and the modules or processes in the accompanying drawing are not necessarily necessary for implementing the present invention.

From the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence or that contributes to the existing technology. The computer software product can be stored in a storage medium, such as ROM/RAM, disk , optical disk, etc., including a number of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

Each embodiment in this specification is described in a progressive manner. The same and similar parts between the various embodiments can be referred to each other. Each embodiment focuses on its differences from other embodiments. In particular, the device or system embodiments are described simply because they are basically similar to the method embodiments. For relevant details, please refer to the partial description of the method embodiments. The device and system embodiments described above are only illustrative, in which the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, It can be located in one place, or it can be distributed over multiple network elements. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

The above are only preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of changes or modifications within the technical scope disclosed in the present invention. All substitutions are within the scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (6)

1. The power distribution network load transfer method based on deep reinforcement learning is characterized by comprising the following steps of:

step 1, initializing a main neural network Q (S, a, ω, α, β) and a target network T (S, a, ω, α, β) identical to the network structure of the main neural network Q, initializing an experience pool R, a discount factor γ, a learning rate Lr, and a target network update frequency N _replace Number of samples N _batch Setting a zone bit Done=0 of an ending state, and forming an intelligent body of the power distribution network by the main neural network Q, the target network T and the experience pool;

the cost function V in the main neural network Q and the target network T is related to the state S, and is independent of the action a, which is a scalar, denoted as V (S, ω, α), the dominance function B is related to both the state S and the action a, which is a vector of length, denoted as the number of actions, denoted as B (S, a, w, β), and the cost function of the agent is expressed as:

Q(s，A，ω，α，β)＝V(S，ω，α)+B(S，A，ω，β)，

wherein ω is a network parameter of the public part, α is a network parameter of the unique part of the cost function, β is a network parameter of the unique part of the dominance function, and the output of the final Q network is obtained by linearly combining the output of the cost function network and the output of the dominance function network;

the dominant function part is subjected to centering treatment, and a combination formula which is practically used is as follows:

Wherein the method comprises the steps ofRepresenting a set of all actions, +.>Namely, the number of elements in the set is calculated, Q (S, A, omega, alpha, beta) obtained by using the above formula is a vector with the length of the action number, wherein each element represents the evaluation value of each action in the state S;

step 2, the distribution network fails and load transfer is started;

step 3, reading real-time state information of the power distribution network, inputting the real-time state information of the power distribution network to the intelligent body, and calculating an evaluation value of each action by the intelligent body according to the real-time state information of the power distribution network;

step 4, the agent selects corresponding actions according to action strategies based on the evaluation value of each action;

step 5, the intelligent agent executes the action on the power distribution network to obtain a state S 'of the power distribution network after the action, evaluates the action of the power distribution network and the state after the action, calculates a Reward report according to constraint conditions and an objective function, determines a Done value according to the Reward report and an ending rule, and stores the switching action of the power distribution network as an experience sample e= (S, a, R, S') in an experience pool R after one switching action of the power distribution network is completed;

step 6, randomly sampling the sampling number N from the experience pool R _batch Calculating a target value by using the discount factor gamma according to the sampled empirical samples, and updating parameters omega, alpha, beta in a main neural network Q (S, A, omega, alpha, beta) by minimizing a loss function based on the target value and a learning rate Lr;

step 7, when the main neural network passes through N _replace After the secondary update, the parameters ω, α, β of the primary neural network Q are used to update the parameters ω, α, β of the target network T:

step 8, judging whether to end the sequence action according to the end flag bit Done, wherein done=0, and returning to the step 4; done=1, exiting the cycle, and ending the load transfer process of the power distribution network;

the step 1 further comprises:

defining a system state space, an action space and a reward function in the load transfer operation of the power distribution network, wherein the interaction of an agent and the power distribution network environment is represented by an array [ S, A, P (S, S '), R (S, a), done ] wherein S represents a state space formed by possible states of the power distribution network, A represents a possible action set, P (S, S ') represents a transition probability of a transition from the state S to the state S ' of the power distribution network, R (S, a) takes an action a in the state S, triggers related rewards, is fed back to the agent, done is a flag bit of an ending state, and is set to 1 when the agent actively selects to terminate the decision or is terminated by the environment for continuous operation due to violation of constraint conditions, and Done is kept to 0 in the normal decision step;

The state space is defined as an array s= [ V, I, SW, F]V is a voltage vector set, which is used to represent the voltage values of all phases at each node in the power distribution network, vin is the voltage value of the nth phase of the ith node; i is a current vector group which is used for representing current values of each phase in all lines in the power distribution network, and Iin is a current value of an nth phase of an ith line; SW is a state value vector of all switches in the power distribution network, SW _i The state of the ith switch is 0, which indicates open, and 1, which indicates close; f is a vector representing the fault state of the power distribution network line, F _i A failure state of the line numbered i, 0 indicates normal, and 1 indicates failure.

2. The method of claim 1, wherein said step 1 further comprises:

the intelligent agent adopts a Dueling-DQN algorithm, the Dueling-DQN algorithm utilizes a deep neural network to calculate, the deep neural network comprises a main neural network Q and a target network T, and the main neural network Q and the target network T comprise: a public hidden layer, a cost function V and a dominance function B;

the common hidden layer of the cost function V and the dominance function B adopts a 2-layer neural network for extracting the characteristics of the input state quantity, and the first layer is 30 x N _feature Neurons of which N _feature For inputting the number of state quantities, all neurons directly receive full-connection input of state data, bias is added, and an activation function is a Relu function; the second layer is fully connected with the first layer, and also has a total connection ratio of 30 x N _feature Spirit of individualMenstruation;

and the intelligent agent calculates the output results of the main neural network Q and the target network T by adopting a lasting-DQN algorithm, and calculates the evaluation value of each action.

3. The method according to claim 2, wherein the agent in step 4 selects a corresponding action according to an action policy based on the evaluation value of each action, including:

the intelligent agent selects corresponding actions according to action strategies based on the action evaluation vectors, and selects optimal actions in a non-exploration mode, wherein the optimal actions are actions with highest evaluation values Q; adopting an epsilon-greedy random greedy strategy in the exploration mode, namely taking a random number x, and selecting the action with the highest evaluation value Q as the current action if x is less than epsilon; if x > epsilon, a random action is selected from all actions, and epsilon is a set parameter.

4. A method according to claim 3, wherein said agent in step 5 performs said actions on the distribution network, comprising:

The action A is a number, the range of the action A is an integer of 0-2 NSwitch, when the action A is 2NSwitch, no operation is taken and the action A exits, and the decision is ended; when the action A is 0-2 NSwitch-1, the following calculation is performed on the action A:

x＝A％2

wherein x is the remainder of dividing A by 2, the formula having the following meaning:

，

each action is to operate one switch or directly exit, if the switch exits, the decision is ended.

5. The method according to claim 4, wherein the step 5 of obtaining the state S' of the distribution network after the action, evaluating the action and the state after the action of the distribution network, calculating the Reward according to the constraint condition and the objective function, and determining the value of Done according to the Reward and the ending rule includes:

the constraint condition of the power distribution network is set, which comprises the following steps:

the voltage is kept within the tolerance range with the deviation of + -7%, and a voltage penalty value P is set for the voltage exceeding the tolerance range _Volt = -10 and set the end flag Done to 1; for voltages not outside this range, a voltage penalty value P is set _Volt ＝0；

When the passing current of the line and the transformer is larger than the limit value, a current penalty value P is set _Lim = -10 and set the end flag Done to 1; for currents not exceeding a limit value, a current penalty value P is set _Lim ＝0；

The ring network punishment P Loop of the intelligent agent is set as follows:

setting invalid action penalty P for an agent _Act The method comprises the following steps:

setting an objective function of the power distribution network comprises:

setting a load loss evaluation value E according to the proportion of the lost load _Loadloss ：

Wherein L is _loss For loss of power load value, L _total E calculated for the total load of the whole power system _Loadloss The value is between-2 and 2;

evaluation value E of the number of times of switch operation _Num ：

Wherein A is _Num The total number of switches, L, of which the decision is changed _Num For the total number of switches, calculate E _Num The value is between-1 and 1;

evaluation value E of line loss condition of power distribution network _Loss ：

Wherein Line is the total number of uninterrupted lines, I _i R is the actual current of the ith line _i The resistance of the ith line and the transformer is shown, and S is the total power of the whole network;

for the nodes whose voltages do not exceed the range of + -7%, the voltage deviation degree evaluation value E of the line _Vot ：

Wherein N is the total number of uninterrupted nodes, pu _i The per-unit value of the voltage of the node i;

the Reward function given by the environment is composed of the sum of the above evaluation values, namely, report:

Reward＝P _Volt +P _Lim +P _Loop +P _Act +E _Loadloss +E _Num +E _Loss +E _Vot 。

6. the method according to any one of claims 2 to 5, wherein said step 6 randomly samples said number of samples N from a pool of experience R _batch A plurality of empirical samples, calculating a target value based on the target value and the learning rate by using the discount factor gamma based on the sampled empirical samples Lr updates parameters ω, α, β in the main neural network Q (S, a, ω, α, β) by minimizing the loss function, including:

random sampling N from experience pool R _batch Sample e of experience _i ＝(si，ai，ri，s′i)，N _batch =20, calculate target value

By minimizing the loss function

Updating parameters omega, alpha and beta in a main neural network Q (S, A, omega, alpha and beta), calculating the updating degree of the parameters by using an RMSProp algorithm, and obtaining a learning rate L _r At 0.1, one update to the primary neural network Q represents a learning process of an agent.