KR20240146224A - Physics-informed reinforcement learning model for compensating unbalanced voltage in the distribution system - Google Patents

Abstract

A voltage imbalance resolution device and method in a distribution system based on a reinforcement learning model considering physical constraints are disclosed. The voltage imbalance resolution method is a voltage imbalance resolution method performed by a voltage imbalance resolution device including at least a processor, and includes a step of receiving observation data of a distribution system, a step of predicting an action from the observation data using a learned artificial neural network model, and a step of generating a constrained action corresponding to the action.

Description

{PHYSICS-INFORMED REINFORCEMENT LEARNING MODEL FOR COMPENSATING UNBALANCED VOLTAGE IN THE DISTRIBUTION SYSTEM}

The present invention relates to an inverter-based distributed power output control within a distribution system, and more particularly, to a technique for controlling active power and reactive power output by taking into account the characteristics of a distribution system.

As the amount of renewable energy connected to the distribution system increases, the number of distributed power sources based on single-phase inverters increases. This causes three-phase voltage imbalances, damages power equipment, and shortens its lifespan, affecting stable and economical grid operation.

The existing inverter output control techniques for resolving voltage imbalance utilize feedback control structures or optimization techniques for compensation. In the case of feedback control logic, the parameters are fixed, making it difficult to control in a wide range. In the case of optimization, it takes a lot of time to perform calculations every time for a given environment.

Accordingly, the present invention proposes a technique for resolving voltage imbalance in a distribution system based on a reinforcement learning model that takes into account the physical constraints of the distribution system.

Republic of Korea Publication Patent No. 2020-0117170 (Published on October 14, 2020) Republic of Korea Publication Patent No. 2012-0065533 (Published on June 21, 2012)

The technical problem to be achieved by the present invention is to provide a device and method for resolving voltage imbalance in a distribution system based on a reinforcement learning model that takes into account the physical constraints of the distribution system.

A voltage imbalance resolution method according to one embodiment of the present invention is a voltage imbalance resolution method performed by a voltage imbalance resolution device including at least a processor, comprising the steps of: receiving observation data of a distribution system; predicting an action from the observation data using a learned artificial neural network model; and generating a constrained action corresponding to the action.

According to the device and method for resolving voltage imbalance in a distribution system according to an embodiment of the present invention, voltage imbalance phenomenon occurring due to single-phase loads and distributed power sources connected to a distribution system can be resolved, thereby contributing to stable system operation.

Additionally, it can enhance grid stability in future distribution systems where distributed power sources such as solar power generation, wind power generation, and energy storage systems are increasing.

In addition, the present invention can derive a fast control signal by utilizing an offline learned model suitable for application to real-time control.

In order to more fully understand the drawings cited in the detailed description of the present invention, a detailed description of each drawing is provided.
FIG. 1 is a conceptual diagram illustrating a voltage imbalance resolution device and method according to one embodiment of the present invention.
Figure 2 shows the DNN structure of the proposed technique, showing (a) an actor network and (b) a critic network.
Figure 3 illustrates a modified IEEE test feeder for applying the proposed technique.
Figure 4 shows randomly generated scenarios, namely (a) load profile, (b) PV profile, and (c) SOC profile.
Figure 5 shows the results of VUF comparison of scenarios, showing the results for (a) no control and (b) control using the proposed technique.
Figure 6 shows the results of comparing the average VUF in each control model with the case of no control.

Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely exemplified for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention can be implemented in various forms and are not limited to the embodiments described in this specification.

Since embodiments according to the concept of the present invention can have various changes and can have various forms, the embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, but includes all modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention.

The terms first or second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are only intended to distinguish one component from another, for example, without departing from the scope of the invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When it is said that a component is "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. On the other hand, when it is said that a component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

The terminology used herein is only used to describe particular embodiments and is not intended to be limiting of the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. It should be understood that, as used herein, the terms "comprises" or "has" and the like are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the present specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms defined in commonly used dictionaries, such as those defined in common usage dictionaries, should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings attached to this specification. However, the scope of the patent application is not limited or restricted by these embodiments. The same reference numerals presented in each drawing represent the same components.

The reinforcement learning (Safe Deep Reinforcement Learning (SDRL)) used in the proposed method can be said to be an advanced Deep Reinforcement learning (DRL) model, and the proposed technique satisfies physical constraints and includes two modules, namely, a learning module (LM) and a constraint module (CM). The main algorithms used in these modules are Deep Deterministic Policy Gradient (DDPG) and Quadratic Programming (QP), respectively. The schematic diagram of the proposed technique is illustrated in Fig. 1.

More specifically, a voltage imbalance resolution device implemented as a computing device including at least a processor and/or a memory includes a learning module (which may be referred to as a learning unit) and a constraint module (which may be referred to as a constraint unit). In addition, the voltage imbalance resolution device may further include a receiving unit (which may be referred to as a receiving module) that receives observed data of a power distribution system. According to an embodiment, the voltage imbalance resolution device may be understood to include a sensing unit (which may be referred to as a sensing module) for measuring data of the power distribution system. The sensing unit can observe the power distribution system and transmit the observed data to the receiving unit. The computing device includes a personal computer (PC), a server, and the like.

MDP(Markov Decision Process)

MDP is S, A, P, R, and is a tuple consisting of S, A, P, R, and A. Here, S is states, A is actions, P is transition probability, R is rewards, and A is actions. represents the discount factor. The definition of MDP is as shown in mathematical expression 1.

[Mathematical formula 1]

Each component of the MDP contains a closed loop, as illustrated in Figure 1. Here, the state, action, and reward are indicated by dotted arrowed lines. P and is included in the reward calculation step. In addition, P is replaced with DNN (Deep Neural Network) in DDPG, The elements are not considered because each time step in the simulation is equally valuable. A detailed description of the remaining MDP elements (states, actions, and rewards) that mitigate voltage imbalances in the distribution system is given later.

1) Observations and States

The observed data from the distribution system include states and/or compensation. The observations are measured from the distribution system. The three-phase voltages (which may mean magnitude and phase) of the ESS (Energy Storage System) PCC (Point of the Common Coupling) node are observed (or measured). The states may mean observations or measurements from the distribution system, which are used to compute compensation and as inputs to critic and actor networks. The states are as shown in Equation 2.

[Mathematical formula 2]

In mathematical expression 2, |V _n | represents the magnitude of each phase voltage, represents the phase of each phase voltage (a, b, c ∈ n). In MDP, the three-phase real power output of the ESS is considered as an action.

2) Actions

Actions are control variables of a reinforcement learning (RL) agent. Actions can be expressed as in mathematical expression 3.

[Mathematical formula 3]

In mathematical expression 3, is the phase-specific active power output of ESS, represents the power factor by phase.

3) Rewards

The reward is the revenue from each time-step, and the sum of the rewards is the episode return. An episode is the total experience of the agent from the beginning to the end. At the end of the episode, the agent updates the actor and critic networks from the return. The reward function for minimizing the voltage imbalance can be expressed as the voltage unbalanced factor (VUF), as in Equation 4.

[Mathematical Formula 4]

R = - VUF = -V ₂ /V ₁ ×100

In mathematical expression 4, V ₁ and V ₂ represent the positive sequence component (positive sequence voltage size) and the negative sequence component (negative sequence voltage size) of the ESS PCC voltage, respectively.

Learning Module (LM)

The learning module can train a given DNN model to generate a behavior prediction model that predicts behavior, and/or can predict behavior corresponding to a state measured at an arbitrary point in the distribution system and received in real time using the generated behavior prediction model. In this case, when predicting behavior in real time, only one model among the two artificial neural network models that predicts behavior may be used.

In order to learn the DNN layer of the learning module (LM), the state is normalized. In addition, in order to efficiently learn the network parameters, z-score normalization can be applied to the DNN input layer. The mean and standard deviation of the state can be calculated from the base case corresponding to the no-ESS control case. The normalized state is calculated as in Equation 5.

[Mathematical Formula 5]

In Equation 5, s is a state, S is a set of states, is the normalized state, μ is the mean value of the state, and σ is the standard deviation of the state (which may also mean the variance depending on the embodiment).

The learning module (DM) learns actions from states and rewards using the DDPG algorithm, which includes two DNNs, called actor and critic networks. These two networks are designed to predict actions and expectations of returns (which may mean estimated rewards) from initial states, as illustrated in Fig. 2. That is, one of the two DNNs may be an artificial neural network that outputs actions based on states, and the other may be an artificial neural network that estimates rewards for actions performed in states. The agent of the DDPG algorithm selects actions based on the actor network. The DNN enables the formulation of MDPs with continuous states and actions.

The actor network can contain three fully connected (FC) layers and three activation function layers. The activation function layers can contain two Rectified Linear Unit (ReLU) layers and one hyperbolic tangent (tanh) layer. At the end, the scaling layer outputs an action to control the ESS. The output of the hyperbolic tangent layer is fixed between -1 and +1, although it is limited by the capacity of the ESS. The scaling layer allows the bounds of the output of the actor network to be designed with appropriate scale and bias parameters.

More specifically, exemplary computational processes of a fully connected layer, a ReLU layer, a hyperbolic tangent layer, and a scaling layer are as shown in Equations 6 to 9, respectively.

[Mathematical Formula 6]

y=w ^T x+b

In mathematical expression 6, y represents the output value, x represents the input value, w represents the weight parameter, and b represents the deviation parameter.

[Mathematical formula 7]

In mathematical expression 7, y represents the output value, x represents the input value, and a represents the scaling factor.

[Mathematical formula 8]

y=tanh(x)

In mathematical expression 8, y represents the output value and x represents the input value.

[Mathematical formula 9]

y=ax+b

In mathematical expression 9, y represents the output value, x represents the input value, a represents the scaling coefficient, and b represents the deviation coefficient.

The DDPG algorithm can be implemented in MATLAB deep learning toolbox. DNN parameters in the critic network DNN parameters in actor networks are initialized with random parameters (random values). In both networks, the parameters can be updated as in previous work (“Deep Deterministic Policy Gradient (DDPG) Agents,” mathworks.com, https://kr.mathworks.com/help/reinforcement-learning/ug/ddpg-agents.html (accessed Oct. 26, 2022).).

Constraint Module (CM)

The constraint module performs an optimization operation based on quadratic programming (QP). That is, the constraint module performs an optimization operation to find a constrained action that is closest to the action output by the learning module according to the objective function and satisfies the constraint conditions. The output of the constraint module may mean a signal for controlling the energy storage device. Therefore, the output of the constraint module may be transmitted to the energy storage device.

1) Constraints

In ESS, the effective power of each phase is constrained to be equal to the scheduled ESS profile, as in Equation 10.

[Mathematical formula 10]

In mathematical expression 10, P _ESS is an ESS profile scheduled one day in advance. That is, assuming that the three-phase ESS output is determined through one-day-ahead scheduling, the present invention can be applied to real-time phase-by-phase control. That is, mathematical expression 10 is a three-phase ESS output P _{ESS, where P ESS} is a determined constant and the phase-by-phase output of the ESS, which is a control variable. The constraint is that the sum of the three-phase ESS outputs must match the determined output.

In addition, the output of each phase of the ESS must have the same sign, as in Equation 11, in order to satisfy the actual ESS operation. The output of each phase of the ESS must be the same as the predetermined output of the three-phase ESS. That is, if it was charged (positive) according to the schedule the day before, the output of each phase must also be positive, and if it was discharged (negative), the output of each phase must also be positive.

[Mathematical formula 11]

The power factor of each phase of the ESS is limited to consider the voltage, current, and apparent power limitations of the inverter, as in Equation 12. That is, the range of power factors for determining the range of reactive power that the inverter can output is limited.

[Mathematical formula 12]

That is, the power factor of each phase can have a value within a predetermined range, for example, a (where a is a real number less than 1, an exemplary value being 0.5) and a value less than or equal to b (where a is a real number greater than or equal to a and less than or equal to 1, an exemplary value being 1).

2) QP(Quadratic Programming)

QP can be introduced to restrict the output behavior of the learning module (LM). The objective function is as shown in Equation 13.

[Mathematical formula 13]

At this time, u and u ₀ are as shown in Equations 14 and 15, respectively. That is, Equation 13 is an objective function that finds the action u that is closest to the action u ₀ determined by the learning module.

[Mathematical formula 14]

[Mathematical formula 15]

Here, u is a constrained predicted action from u ₀ , and u ₀ means the original action from the learning module (LM) (i.e., the action output from the learning module). In addition, P _ESS has three-phase active power output, ESS effective power output by phase (A phase, B phase, C phase), represents the power factor by phase.

The constraint module (CM) selects the closest action with the minimum squared error that satisfies the constraints. The action satisfies Equations (16) and (17).

[Mathematical formula 16]

[Mathematical formula 17]

Here, f is a constant value matrix from observations, and g is a matrix multiplied by the constrained actions. In other words, f is a zero-order term, and g is a coefficient of a first-order term. and are the lower bound and upper bound of the action, respectively.

The constraints terms of QP are formulated according to mathematical expressions 18 to 21.

[Mathematical expression 18]

[Mathematical formula 19]

[Mathematical expression 20]

[Mathematical expression 21]

The simulation environment was formed based on the IEEE 13 test feeder, as illustrated in Fig. 3. PV was installed at 680 nodes, and ESS was installed at 675 nodes where the highest average UVF was observed. That is, Fig. 3 illustrates a distribution system, which is a monitoring target of the present invention. The distribution system used in the present invention may have various forms, and the number of bus bars, the number of lines, the locations or numbers of connected solar power plants (PVs), wind power plants (WTs), and energy storage devices (ESSs), the presence or absence of a low-voltage distribution system, the location of transformers, the size of the operating voltage, etc. are not important.

Load, PV (photovoltaic power generation facility), and ESS SOC profiles were generated to simulate various scenarios, as shown in Fig. 4. It is assumed that the randomly generated scenarios are the results of the one-day-ahead optimization. A thousand scenarios were generated to learn and test the proposed technique. Among them, 700 scenarios were used for offline learning to resolve voltage imbalance, and 300 scenarios were used for testing in online simulation for the learned agent.

For each phase, load and PV profiles were generated for operation from 0 to 1 pu. Each phase of the load was also operated randomly to simulate various situations of unbalanced voltage. SOC profiles were generated for operation from 10% to 90%.

Learning Module Hyperparameters (LM Hyperparameters)

The learning module (LM) contains parameters for DNN and agent learning options, as shown in Table 1.

[Table 1]

Voltage Unbalanced Mitigation Results

The proposed technique can control the active power and reactive power of ESS. In order to compare the performance of the proposed technique, a technique that controls only the active power was performed for the same scenario. In addition, one of the heuristic optimization algorithms (Particle Swarm Optimization (PSO)) was simulated to compare the performance and consumption time. In order to satisfy the operation constraints, a constraint module (CM) was connected to PSO.

The performance of the best scenario (223) in the test set is shown in Fig. 5. The VUF is computed only for 6 nodes instead of all 13 nodes in the simulated distribution system, since the other nodes are single-phase or dual-phase nodes. The highest VUF (4.35%) is observed for scenario 223, 675 buses, which is reduced to 3.08% using the proposed technique. Notably, the VUFs of other nodes and scenarios are also reduced.

Fig. 6 shows the results of comparing the average VUF in the case of no control and each control model. Nodes 671 and 675 are connected to heavy loads that cause high imbalance. And the neighboring nodes have higher imbalance voltage than nodes 632 and 633. In the case of no control, the same real and reactive power were output for each phase. The model based on the proposed technique provided the same appropriate control for the test and learning scenarios. PSO was simulated for the test scenario in order to compare with other models. Here, PSO showed the lowest VUF. This is because it takes a lot of time to optimize each scenario. The results of performance and time consumption are shown in Table 2.

[Table 2]

In Table 2, the computation time for learning was measured as the total learning time using 700 scenarios, and the computation time for testing was calculated as the average value of 300 test scenarios. The VUF improvement was calculated as the difference between the mode without control and each model.

The devices described above may be implemented as hardware components, software components, and/or a collection of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone, but those skilled in the art will appreciate that the processing device may include multiple processing elements and/or multiple types of processing elements. For example, the processing unit may include multiple processors, or one processor and one controller. Other processing configurations, such as a parallel processor, are also possible.

The software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and/or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed on network-connected computer systems and stored or executed in a distributed manner. The software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known to and usable by those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

Although the present invention has been described with reference to the embodiments illustrated in the drawings, this is merely exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. For example, appropriate results can be achieved even if the described techniques are performed in a different order from the described method, and/or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different form from the described method, or are replaced or substituted by other components or equivalents. Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

Claims (7)

In a voltage imbalance resolution method performed by a voltage imbalance resolution device including at least a processor,
A step of receiving observation data of a distribution system;
A step of predicting an action from the observation data using a learned artificial neural network model; and
comprising a step of producing a constrained action corresponding to said action;
How to resolve voltage imbalance. In the first paragraph,
The above observation data includes a state, and the state includes the phase-by-phase voltage magnitude and phase of the three-phase voltage at any point in the distribution system.
How to resolve voltage imbalance. In the second paragraph,
The step of predicting the above action is to normalize the above state and then input the normalized state into the learned artificial neural network model.
The above normalized state is generated through mathematical expression 1,
The above mathematical expression 1 is And,
The above s is a state, the above S is a set of states, and the above is a normalized state, μ represents the mean value of the state, and σ represents the standard deviation of the state.
How to resolve voltage imbalance. In the third paragraph,
The step of producing the above-mentioned constrained behavior produces the above-mentioned constrained behavior through second-order programming optimization,
The above second-order programming optimization has mathematical expression 2 as the objective function,
The above mathematical formula 2 is And,
The above u is the constrained action, and the above u ₀ means the action that is the output of the learned artificial neural network model.
How to resolve voltage imbalance. In paragraph 4,
The step of producing the above-mentioned constrained action produces the above-mentioned constrained action using the first constraint,
The above first constraint is expressed by mathematical expression 3,
The above mathematical formula 3 is And,
The above P _ESS is a predetermined 3-phase ESS (Energy Storage System) output,
Above is the output of ESS,
How to resolve voltage imbalance. In paragraph 5,
The step of producing the above-mentioned constrained action produces the above-mentioned constrained action by further utilizing the second constraint,
The above second constraint is expressed by mathematical expression 4,
The above mathematical expression 4 is person,
How to resolve voltage imbalance. In Article 6,
The step of producing the above-mentioned constrained action produces the above-mentioned constrained action by further utilizing the third constraint,
The above third constraint is expressed by mathematical expression 5,
The above mathematical expression 5 is And,
is the power factor,
How to resolve voltage imbalance.