CN117041132B - A distributed load balancing satellite routing method based on deep reinforcement learning - Google Patents

Abstract

This application relates to a distributed load balancing satellite routing method based on deep reinforcement learning. On the one hand, it obtains the queue occupancy rate and queue status identification of neighbor satellite nodes by sensing the data cache queue load of neighbor satellite nodes; on the other hand, the neighbor nodes are Combined with the geographical distance factor and topological distance factor of the destination node, it is used to measure the proximity between the two. The resulting topological-geographical distance factor can better reflect the proximity between satellite nodes in the same hemisphere or different hemispheres. As a good driving factor for satellite network routing, it accelerates the convergence of routing algorithms, so that communication data flows can not only propagate along the shortest propagation distance path under low load conditions during the routing and forwarding process, but also bypass high-load nodes in time, thus Reduce node congestion as much as possible, achieve load balancing of data traffic transmission, reduce network packet loss rate and improve throughput.

Description

A distributed load balancing satellite routing method based on deep reinforcement learning

Technical field

This application relates to the field of wireless communication technology, and in particular to a distributed load balancing satellite routing method based on deep reinforcement learning.

Background technique

In recent years, low-orbit satellite communication systems represented by "Iridium", "GlobalStar", "OneWeb" and "Starlink" have become a new generation of satellite communication systems The mainstream development trend has led to the construction boom of global low-orbit satellite constellations. The significant advantages of the low-orbit satellite constellation in the number of satellites, orbital altitude, space environment, signal strength, batch development, rapid deployment, etc. will likely bring about the advancement of satellite communications, navigation, remote sensing, reconnaissance, surveillance, early warning detection and other technologies. Major changes and impact on the overall pattern of space electronic information field. Satellite networks with low-orbit satellite constellations as the core get rid of the physical limitations of traditional terrestrial networks that require station construction, can achieve global continuous coverage, and provide real-time communication services with low latency, high speed, high power and high throughput.

The low-orbit satellite communication system forms a space communication network with satellites as switching nodes by establishing inter-satellite links (ISL) between satellites to achieve point-to-point communication between ground stations, satellites, and ground users. The satellite network using inter-satellite links can reduce the number of ground gateway stations deployed, reduce the complexity of the ground segment system, and form a relatively independent data relay system on the ground, thus truly becoming a relatively independent and independent information network in the space, space and ground integrated information network. important part. For low-orbit satellite communication networks that contain a large number of satellite nodes and inter-satellite links, the selection of routing strategies has become the key to ensuring the timely and accurate transmission and delivery of data flows. Because the communication data between users located in different geographical locations on the ground, between users and ground stations, and between ground stations, is relayed and transmitted between satellite nodes through the satellite network based on inter-satellite link networking, and routing The policy determines the path selection for data flow relay transmission in the satellite network. Different from the terrestrial network, although each node in the satellite network serves as a routing and forwarding node, it is in continuous motion, and the earth and the satellite constellation maintain relative motion. The terrestrial network service requests are also unevenly distributed geographically. Therefore, the satellite Internet Traffic transmission is highly dynamic, and it is easy for network traffic distribution to be uneven. Due to the dynamics of low-orbit satellite networks, imbalance of network bandwidth and storage resources, imbalance of network traffic load and other factors, for the same source node-destination node pair, the communication quality of different transmission paths is very different, so how to Planning an efficient and reliable path for each user's communication request has become a key issue in satellite network routing technology research.

Contents of the invention

Based on this, it is necessary to address the above technical issues and provide a distributed load balancing satellite routing method based on deep reinforcement learning, aiming to solve the problem of efficient and reliable inter-satellite links in low-orbit satellite communication systems considering the imbalance of network traffic load. The technical issue of road network routing design is to design a system that can sense the network status and traffic load changes of low-orbit communication satellites in real time, and optimize and adjust the signal propagation path to achieve load balancing and efficient transmission of satellite network data traffic.

A distributed load balancing satellite routing method, the method includes:

The current inter-satellite link is obtained based on the low-orbit satellite network topology snapshot of the current time slot; the inter-satellite link contains multiple satellite nodes; each satellite node corresponds to a routing decision-making agent;

In the inter-satellite link under the current time slot, the geographical distance factor of the neighbor satellite node and the target satellite node is calculated based on the spatial coordinates of the current satellite node's neighbor satellite node and the target satellite node. According to the current satellite node's neighbor satellite node and The virtual topological address of the target satellite node is calculated to obtain the topological distance factor of the neighboring satellite node and the target satellite node, and the topological-geographical distance factor is obtained according to the geographical distance factor and the corresponding topological distance factor; among which, the virtual topological address includes the orbit number of the satellite node. and its sequence number within the track;

The queue status identifier, queue occupancy rate and topology-geographic distance factor of each neighbor satellite node and the target satellite node are used as the observation status information under the current time slot and input into the deep neural network of the routing decision agent corresponding to the current satellite node. In the network, output the status-action value of the current satellite node forwarding the data stream to each neighbor satellite node;

Select the neighbor satellite node corresponding to the maximum state-action value as the next-hop forwarding node of the current satellite node to complete the data forwarding task of the current time slot.

The above-mentioned distributed load balancing satellite routing method based on deep reinforcement learning, on the one hand, obtains the queue occupancy rate and queue status identification of neighbor satellite nodes by sensing the data cache queue load of neighbor satellite nodes; on the other hand, it combines the relationship between neighbor nodes and destination nodes. The geographical distance factor and the topological distance factor are combined to measure the proximity between the two. The topological distance factor takes into account the orbital surface proximity and intra-orbital sequential proximity of the two satellite nodes, so the topological-geographical distance factor can be more accurate. It can well reflect the proximity between satellite nodes in the same hemisphere or different hemispheres, and can be used as a good driving factor for satellite network routing to speed up the convergence of the routing algorithm, so that the communication data flow can be processed according to the requirements under low load conditions during the routing and forwarding process. Propagate through the path with the shortest propagation distance, and can avoid high-load nodes in time, thereby minimizing node congestion, achieving load balancing of data traffic transmission, reducing network packet loss rate and improving throughput.

Description of the drawings

Figure 1 is a flow chart of a distributed load balancing satellite routing method based on deep reinforcement learning;

Figure 2 is a schematic diagram of a "greedy" routing strategy based on geographical location;

Figure 3 is a schematic diagram of the reinforcement learning technology framework;

Figure 4 shows the deep neural network structure of a distributed load balancing satellite routing method based on deep reinforcement learning;

Figure 5 shows the system implementation architecture of a distributed load balancing satellite routing method based on deep reinforcement learning;

Figure 6 is an internal structure diagram of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

In one embodiment, as shown in Figure 1, a distributed load balancing satellite routing method based on deep reinforcement learning is provided, including the following steps:

Step 102: Obtain the current inter-satellite link based on the LEO satellite network topology snapshot of the current time slot.

Among them, the inter-satellite link contains multiple satellite nodes, and each satellite node corresponds to a routing decision-making agent.

The topology control method using virtual topology strategy divides the low-orbit satellite network topology into multiple time slots. The satellite network in each time slot can be regarded as a static topology, that is, the connection relationship between satellite nodes does not change. The static topology in is the inter-satellite link under the corresponding time slot.

Step 104: In the inter-satellite link under the current time slot, calculate the geographical distance factor of the neighbor satellite node and the target satellite node according to the spatial coordinates of the neighbor satellite node and the target satellite node of the current satellite node. According to the neighbors of the current satellite node The virtual topological addresses of the satellite node and the target satellite node are calculated to obtain the topological distance factor of the neighbor satellite node and the target satellite node, and the topological-geographical distance factor is obtained according to the geographical distance factor and the corresponding topological distance factor.

Among them, the virtual topology address includes the orbit number of the satellite node and its sequence number within the orbit.

As shown in Figure 2, a schematic diagram of a "greedy" routing strategy based on geographical location is provided. In the low-orbit satellite network, for the data flow generated at the source node S and the destination node is D , if each hop forwarding selects a satellite node that is geographically closer to the destination node, then/> Will experience the minimum number of forwarding hops. In most cases, the shortest propagation distance path is included in multiple minimum hop count paths, and there is not much difference in propagation delay between the shortest propagation distance path and the minimum hop count path. Therefore, through the "greedy" routing strategy generated by the continuous geographical distance to the destination node, the minimum hop count path to the destination node D can be quickly found, thereby achieving optimal delay performance.

As can be seen from Figure 2, when the data flow Located at node/> And the destination node is/> When , select node/> As the next hop node, it will be geographically closer to the destination node/> . Therefore, in scenarios with light network load, this "greedy" selection strategy based on geographical location will help quickly find the transmission path with the smallest number of forwarding hops and realize data flow/> The end-to-end delay experienced is minimized. In addition to using spatial coordinates to calculate the proximity between satellite nodes, due to the grid characteristics of low-orbit satellite network topology, the transmission distance between satellite nodes can also be measured by grid coordinates. The gridding of satellite networks can model the spherical space network topology into two-dimensional plane grid coordinates, and can measure the proximity between two satellite nodes through topological coordinates. Therefore, the present invention proposes a measurement method that combines geographical distance and topological distance to characterize the proximity between two nodes and is used to drive routing calculations, namely Topologic and Geographic Distance Factor (TGDF). Topological and geographical distance factors TGDP consists of two parts: topological distance factor (Topologic Distance Factor, TDF) and geographical distance factor (Geographic Distance Factor, GDF). In topological slot/> Within, satellite node/> The three-dimensional geographical coordinates and virtual topological address of are respectively and/> , of which/> Number the track,/> Number the sequence within the track.

The advantage of using TGDF values instead of simply using TDF or GDF is that TGDF can well reflect the proximity between satellite nodes in the same hemisphere or different hemispheres. Therefore, TGDF can be used as a driving factor for good satellite network routing to speed up routing. The algorithm converges.

Step 106: Use the queue status identifier, queue occupancy rate and topology-geographic distance factor of each neighbor satellite node and the target satellite node as the observation status information under the current time slot and input it into the routing decision agent corresponding to the current satellite node. In the deep neural network, the state-action value of the current satellite node forwarding the data stream to each neighbor satellite node is output.

In the traditional satellite network communication scenario based on TCP/IP, most of the congestion control in the transport layer protocol is "passive and passive", that is, only when a node experiences congestion and packet loss, other nodes in the network are notified to adopt a congestion control strategy. , reducing the data sending rate to congested nodes, so congestion control has serious lag, and it is impossible to predict and avoid paths in advance when node congestion and packet loss are about to occur.

In order to timely sense the traffic load of each node in the network and predict the queue delay within the node, the data flow Propagation delay and queue delay can be comprehensively measured when forwarding each hop. This method enables satellite nodes to timely sense the queue load of surrounding nodes and the distance to the destination node through the exchange of status information between nodes. The queue load condition includes the queue status identifier and queue occupancy rate, and the distance to the destination node is the topological-geographical distance factor.

Step 108: Select the neighbor satellite node corresponding to the maximum state-action value as the next-hop forwarding node of the current satellite node to complete the data forwarding task of the current time slot.

In the above-mentioned distributed load balancing satellite routing method based on deep reinforcement learning, on the one hand, the queue occupancy rate and queue status identification of the neighbor satellite nodes are obtained by sensing the data cache queue load of the neighbor satellite nodes; on the other hand, the neighbor nodes and the destination node are The geographical distance factor and the topological distance factor are combined to measure the proximity of the two. Among them, the topological distance factor takes into account the orbital surface proximity and the sequential proximity within the orbit of the two satellite nodes. Therefore, the topological-geographical distance factor can Better reflecting the proximity between satellite nodes in the same hemisphere or different hemispheres, it can be used as a driving factor for good satellite network routing to speed up the convergence of the routing algorithm, so that the communication data flow can be processed under low load conditions during the routing and forwarding process. Propagate according to the path with the shortest propagation distance, and can avoid high-load nodes in time, thereby minimizing node congestion, achieving load balancing of data traffic transmission, reducing network packet loss rate and improving throughput.

In one embodiment, the HELLO/ACK detection response message mechanism is used to exchange status information between satellite nodes through inter-satellite links; the status information includes: the IP address of the satellite node, virtual topology address, spatial coordinates, queue Occupancy rate, timestamp.

satellite node Receive neighbor satellite nodes via inter-satellite links/> The incoming HELLO notification message is parsed and contains the node/> status information of satellite nodes/> Also to neighbor nodes/> Send an ACK response message containing its own status information. The HELLO/ACK message contains the IP address, virtual topology address, spatial coordinates, queue occupancy, timestamp and other information of the sending node.

In the low-orbit satellite network, the transmission priority of HELLO/ACK messages in the satellite network is higher than that of business data packets, ensuring that each satellite node can receive status information from surrounding nodes in a timely manner, thereby ensuring that distributed routing decisions have better It has real-time performance and can quickly respond to highly dynamic satellite network environments. In order to ensure the real-time update of link status, the satellite node will update the link status in time intervals. Periodically send HELLO notification messages to neighbor nodes to ensure that neighbor nodes can grasp queue and link status information in real time to assist in routing decisions. When the satellite node sends a HELLO notification message to the neighbor node, the monitoring period/> , if during the monitoring period/> If the ACK response message fed back by the neighbor node is received within 10 seconds, it means that communication with the neighbor node can be established, and the outdated HELLO confirmation packet will be discarded; otherwise, if the monitoring cycle/> If the HELLO_ACK confirmation message from the neighbor node is still not received, it is considered that the communication link with the neighbor node is interrupted. At this time, the link status is set to "disconnected" in the neighbor node operating status table maintained by this node until The communication connection is re-established only after receiving the HELLO_ACK confirmation message fed back by the node. The neighbor node queue monitoring strategy based on HELLO notification messages can timely sense the operating status of neighboring nodes to estimate the queue delay of neighbor nodes, and provide sufficient routing decision information for the next hop forwarding of data packets.

In one embodiment, the geographical distance factor of the neighbor satellite node and the target satellite node is calculated based on the spatial coordinates of the current satellite node's neighbor satellite node and the target satellite node:

;

in, Indicates time slot/> The next page/> neighbor satellite nodes of satellite nodes,/> Indicates the target satellite node,/> Indicates time slot/> The next page/> The spatial coordinates of neighbor satellite nodes of satellite nodes, Indicates the spatial coordinates of the target satellite node,/> Indicates time slot/> The next page/> The geographical distance factor between the neighbor satellite node of a satellite node and the target satellite node,/> represents the speed of light.

It can be seen from the above equation that the GDF value is equal to the propagation delay of the inter-satellite link between nodes, so it intuitively reflects the geographical distance between the two nodes.

In one embodiment, the topological distance factor of the neighbor satellite node and the target satellite node is calculated based on the virtual topological addresses of the current satellite node's neighbor satellite node and the target satellite node:

;

in, Indicates time slot/> The next page/> The orbit number of the neighbor satellite node of a satellite node, Indicates time slot/> Next page/> The orbital sequence number of the neighbor satellite node of a satellite node,/> Indicates the orbit number of the target satellite node,/> Represents the sequence number within the orbit of the target satellite node,/> Indicates time slot/> Next page/> The topological distance factor between the neighbor satellite node of a satellite node and the target satellite node,/> Indicates the influence factor of orbital surface proximity on topological distance factor,/> Represents the influence factor of the sequential proximity within the orbit on the topological distance factor,/> Indicates the number of satellites in each orbit.

It can be seen from the above formula that, The smaller the node/> and/> The closer the orbital surfaces are, the smaller the TDF value reflects/> with/> The closer;/> Represents node/> and/> The minimum number of hops in the north-south direction, when/> and/> The closer the order within the orbit, the smaller the TDF value, which also shows that/> and/> The smaller the distance between them.

In one embodiment, the topological-geographical distance factor obtained based on the geographical distance factor and the corresponding topological distance factor is:

;

in, Indicates time slot/> The next page/> The topology-geographic distance factor of the neighbor satellite node and the target satellite node of the satellite node,/> Represents the influence factor of topological distance factor on topological-geographical distance factor,/> Indicates the influence factor of geographical distance factor on topological-geographic distance factor.

In one embodiment, the queue status identifier of each neighbor satellite node, the queue occupancy rate, and the topological-geographical distance factor from the target satellite node are used as the observation status information under the current time slot and the route corresponding to the current satellite node is input. In deep neural networks for decision-making agents.

Get a pre-built reward function; the expression for the reward function is:

;

in, Indicates time slot/> The next page/> Reward function of satellite nodes,/> Indicates time slot/> The next page/> The queue status identifier of the neighbor satellite node of the satellite node,/> Indicates time slot/> The next page/> Queue occupancy rate of neighbor satellite nodes of a satellite node.

The reward function serves as the "baton" that drives the agent's corrective action strategy in reinforcement learning. For the routing decision-making agent, the reward obtained by selecting an adjacent link-building node as the next-hop forwarding node directly reflects the next step. The potential end-to-end delay performance of a hop node to the destination node. The TGDF between the selected next-hop node and the destination node and the queue occupancy rate QOR of the next-hop node are included in the reward function to comprehensively evaluate the end-to-end delay performance of the node. . Formula in/> The next hop node selected by the routing decision-making agent can be seen from the reward function expression. When the next hop node is the destination node, the agent gets a large positive reward, indicating that the transmission task has been completed; when the next hop node is not the destination node When the node is reached, the agent will receive a negative reward. The negative reward during the route transmission process can encourage the agent's routing decision to approach the destination node as quickly as possible; when the queue status indicator of the selected next hop node /> When , it indicates that the node is in the "idle" state, and the reward function is only related to the TGDF between the next hop node and the destination node. The smaller the TGDF, the closer it is to the destination node, so the smaller the negative reward obtained; when the next hop node Queue status identifier/> When , it indicates that the node is in the "medium load" state. At this time, the reward function begins to consider the queue occupancy rate QOR of the node; the queue status identifier of the current next-hop node/> When , it indicates that the node is in a "busy" state. At this time, the impact weight of the next-hop node queue occupancy rate on the reward function will be further increased, indicating that the queue delay of the next-hop node plays a leading role in the end-to-end delay performance at this time. .

According to the observed state information and reward function, the state-action value of the current satellite node forwarding the data flow to each neighbor satellite node is obtained.

In one embodiment, the step of constructing the deep neural network of the routing decision agent includes:

Constructing a first deep neural network and a second deep neural network; the network structures of the first deep neural network and the second deep neural network are the same; among them, the first deep neural network is used as the main network for real-time estimation of state-action value; The second deep neural network serves as the target network and is used to calculate the target state-action value and provide it to the first deep neural network for updating network parameters.

State-action value refers to the optimal state-action value objective function Q value based on the Bellman equation. The calculation formula is:

;

in is the target Q value,/> is the reward function,/> is an estimate/> Network,/> is the next observation state, for target/> Network,/> is an estimate/> Network parameters,/> for target/> Network parameters.

One DNN is the target Q-value network, and the other DNN is the estimated Q-value network. Every training period, all parameters of the estimated Q-value network will be assigned to the target Q-value network. The network used to determine the next hop forwarding node is Estimated Q network, the target Q network is used to make the update of the estimated Q network parameters more reasonable and prevent the estimated Q network from overestimating the Q value during the training process.

This method uses the classic deep reinforcement learning Double-DQN (Double Deep Q-Network) algorithm to implement a satellite network routing algorithm based on TGDF driver. The Double-DQN algorithm combines Deep Neural Networks (DNN) with the Q-Learning algorithm of reinforcement learning, and uses the powerful nonlinear fitting ability of DNN to fit the Q value of the state-action value function, thereby solving the problem of Q- In the Learning algorithm, as the state space increases, the storage space required for the Q-value table that records the state-action value function increases sharply, resulting in a significant reduction in the convergence rate of the algorithm.

Reinforcement learning is one of the branches of machine learning and is mainly used to solve continuous decision-making problems. Reinforcement learning can learn how to achieve set goals in complex and uncertain environments. As a "reward and punishment" learning algorithm, reinforcement learning does not rely on prior knowledge of the system, nor does it need to establish a precise mathematical model, avoiding the problem of inaccurate model design. It only uses reward and punishment signals given by the environment for training. Algorithm parameters. The basic principle of reinforcement learning is: if the decision-making subject receives a positive reward signal (reinforcement signal) from the environment after performing an action, the tendency of the decision-making subject to perform the current action will be strengthened; conversely, the tendency of the decision-making subject to perform this action will be weakened. The reinforcement learning technology framework is shown in Figure 3. The framework of reinforcement learning mainly consists of five parts: agent, environment, state, action and reward. The agent is the core of reinforcement learning and has the ability to perceive and learn; the state is the parameter form of the agent in the current environment; the action is the execution operation controlled by the agent; the environment is the object explored by the agent; the reward is Evaluation of the agent's decision-making behavior in the current state based on the information provided by the environment. After the agent performs an action, the environment will transition to a new state. For this new state, the environment will give a reward signal (positive or negative reward), update the agent's strategy according to a certain learning rate, and execute the new state. action, and repeat this process until the agent learns the decision-making strategy that can obtain the maximum reward benefit. The above process is the way in which the agent and the environment interact through states, actions, and rewards.

First, identify the five elements of the reinforcement learning model framework in the TGDF-DRL method:

(1) Intelligent agent. Adopting distributed thinking, the routing decision-making agent is deployed inside each satellite node. The satellite node continuously obtains environmental status information according to the HELLO notification message mechanism. When the satellite node needs to transmit data flow, the Double-DQN network inside the node transfers the status The information is taken as input and then the Q value of each action taken in this state is output. The node transmits the data stream to the next hop according to the action selected by the greedy method.

(2) Routing environment. The routing environment is the topological state geometry of the entire low-orbit satellite network. When the data flow Located at satellite node/> When, data flow/> The environmental information that can be perceived for next-hop routing selection only comes from/> Neighbor node geometry /> .

(3) State space. exist time, data relay node/> Status information from neighbor nodes will be observed from the routing environment. Since the overall goal of the routing algorithm is to transmit data with the smallest number of forwarding hops to minimize the propagation delay and queuing delay, neighbor nodes are gathered/> To destination node/> TGDF incorporates status information to measure/> The distance from the neighbor node to the destination node. If the TGDF between the neighbor node and the destination node is smaller, it means that the node is located in/> To destination node/> The greater the probability of the path with minimum end-to-end delay. However, TGDF can only measure the potential propagation delay of the data flow. In order to measure the queue delay of neighbor nodes and more comprehensively evaluate the end-to-end delay performance of the next hop node, neighbor nodes are aggregated/> Queue occupancy/> and queue status identifier/> Include status information.

Therefore, for Time satellite node collection/> And the destination node set/> , the state space is defined as:

;

Since the routing decision agent is deployed on each low-orbit satellite, for a single satellite node The status information that can be observed is:

;

satellite node Observed state/> is a concise and intuitive limited-dimensional vector, and because the state information is related to the current node/> , neighbor node/> and destination node/> It has nothing to do with its own information. It is only used to describe the potential end-to-end delay performance of the task goal of "the data packet located at the current node reaches the destination node". Therefore, network parameter convergence has nothing to do with the specific combination of "current node-destination node". The routing strategy has good convergence performance and generalization for nodes in the entire network.

(4) Action space. exist Time, satellite node/> Data flow/> needs to be is sent to the next node, so the action space is set to node/> The set of adjacent link-building nodes/> , that is, action space/> .

(5) Reward function.

After modeling the satellite routing scenario and reinforcement learning elements, the satellite node needs to store the environmental information observed by the deep neural network DNN learning, and establish the correlation between the environmental information and the next-hop forwarding decision based on the reward function rules. The satellite node routing decision-making agent based on Double-DQN needs to store two DNNs with the same network structure. One DNN is used as the main network MainNet for real-time estimation of the Q value, and the other DNN is used as the target network TargetNet for calculating the target Q value and giving MainNet provides the target for parameter updates. For both the main network and the target network, the network inputs are observation states , including the TGDF of adjacent nodes, queue occupancy QOR and queue status identifier QSF, and the output is the current status The status of each action - action value/> , The deep neural network structure of this method is shown in Figure 4.

After the routing agent completes training, it can output the next-hop forwarding node through the input observation status, and route and forward the data packet. exist Time, satellite node/> First, obtain neighbor nodes based on the parsed HELLO message/> geographical coordinates, topological coordinates, queue occupancy and queue status identification, for the data flow to be sent/> , parse the corresponding destination node/> geographical coordinates and topological coordinates, then node/> The internal processor calculates/> , and will ,/> and/> As the input of the routing decision-making agent, the routing decision-making agent outputs the state-action value function Q value corresponding to the selected next-hop node under the current observation, node/> According to/> The greedy method selects the data stream/> Sent to the next hop forwarding node, thus completing the data forwarding task at the current moment.

As shown in Figure 5, the system implementation architecture of this method is provided. TTL is the abbreviation of Time To Live. This field specifies the maximum number of network segments that an IP packet is allowed to pass before being discarded by the router. This method uses a deep reinforcement learning agent as the core of routing decision-making. The routing decision-making agent senses the data cache queue load of neighbor nodes and gives the next-hop forwarding node based on the proximity between the neighbor node and the destination node, so that During the routing and forwarding process, communication data flows can not only propagate along the shortest propagation distance path under low load conditions, but also bypass high-load nodes in time, thereby minimizing node congestion and achieving load balancing of data traffic transmission. , reduce network packet loss rate and improve throughput.

It should be understood that although various steps in the flowchart of FIG. 1 are shown in sequence as indicated by arrows, these steps are not necessarily executed in the order indicated by arrows. Unless explicitly stated in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in Figure 1 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution of these sub-steps or stages The sequence is not necessarily sequential, but may be performed in turn or alternately with other steps or sub-steps of other steps or at least part of the stages.

In one embodiment, a distributed load balancing satellite routing device based on deep reinforcement learning is provided, including: a topology snapshot acquisition module, a distance factor calculation module, a state-action value calculation module and a data forwarding module, wherein:

The topology snapshot acquisition module is used to obtain the current inter-satellite link based on the low-orbit satellite network topology snapshot of the current time slot; the inter-satellite link contains multiple satellite nodes; each satellite node corresponds to a routing decision-making agent;

The distance factor calculation module is used to calculate the geographical distance factor of the neighbor satellite node and the target satellite node according to the spatial coordinates of the neighbor satellite node and the target satellite node of the current satellite node in the inter-satellite link under the current time slot. The virtual topological address of the satellite node's neighbor satellite node and the target satellite node is calculated to obtain the topological distance factor of the neighbor satellite node and the target satellite node. The topological-geographical distance factor is obtained according to the geographical distance factor and the corresponding topological distance factor; where, the virtual topological address Including the orbit number of the satellite node and its sequence number within the orbit;

The state-action value calculation module is used to use the queue status identifier of each neighbor satellite node, the queue occupancy rate, and the topology-geographic distance factor with the target satellite node as the observation state information under the current time slot and input it to the current satellite node In the corresponding deep neural network of the routing decision-making agent, the status-action value of the current satellite node forwarding the data stream to each neighbor satellite node is output;

The data forwarding module is used to select the neighbor satellite node corresponding to the maximum state-action value as the next-hop forwarding node of the current satellite node to complete the data forwarding task of the current time slot.

For specific limitations on the distributed load balancing satellite routing device based on deep reinforcement learning, please refer to the limitations on the distributed load balancing satellite routing method based on deep reinforcement learning mentioned above, which will not be described again here. Each module in the above-mentioned distributed load balancing satellite routing device can be realized in whole or in part by software, hardware and combinations thereof. Each of the above modules may be embedded in or independent of the processor of the computer device in the form of hardware, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. 6 . The computer device includes a processor, memory, network interface, and database connected through a system bus. Wherein, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage medium stores operating systems, computer programs and databases. This internal memory provides an environment for the execution of operating systems and computer programs in non-volatile storage media. The computer device's database is used to store data such as satellite node locations and queue load conditions. The network interface of the computer device is used to communicate with external terminals through a network connection. The computer program when executed by the processor implements a distributed load balancing satellite routing method.

Those skilled in the art can understand that the structure shown in Figure 6 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Specific computer equipment can May include more or fewer parts than shown, or combine certain parts, or have a different arrangement of parts.

In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of the method in the above embodiment are implemented.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage. In the media, when executed, the computer program may include the processes of the above method embodiments. Any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined in any way. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, all possible combinations should be used. It is considered to be within the scope of this manual.

The above-described embodiments only express several implementation modes of the present application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all fall within the protection scope of the present application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims (7)

1. A distributed load balancing satellite routing method based on deep reinforcement learning, characterized in that the method includes: The current inter-satellite link is obtained based on the low-orbit satellite network topology snapshot of the current time slot; the inter-satellite link contains multiple satellite nodes; each satellite node corresponds to a routing decision-making agent; In the inter-satellite link under the current time slot, the geographical distance factor of the neighbor satellite node and the target satellite node is calculated based on the spatial coordinates of the current satellite node's neighbor satellite node and the target satellite node. According to the current satellite node's neighbor satellite node and The virtual topological address of the target satellite node is calculated to obtain the topological distance factor of the neighboring satellite node and the target satellite node, and the topological-geographical distance factor is obtained according to the geographical distance factor and the corresponding topological distance factor; among which, the virtual topological address includes the orbit number of the satellite node. and its sequence number within the track; The queue status identifier, queue occupancy rate and topology-geographic distance factor of each neighbor satellite node and the target satellite node are used as the observation status information under the current time slot and input into the deep neural network of the routing decision agent corresponding to the current satellite node. In the network, output the status-action value of the current satellite node forwarding the data stream to each neighbor satellite node; Select the neighbor satellite node corresponding to the maximum state-action value as the next-hop forwarding node of the current satellite node to complete the data forwarding task of the current time slot. 2. The method according to claim 1, characterized in that the geographical distance factor of the neighbor satellite node and the target satellite node is calculated according to the spatial coordinates of the neighbor satellite node and the target satellite node of the current satellite node, including: According to the spatial coordinates of the neighbor satellite node and the target satellite node of the current satellite node, the geographical distance factor of the neighbor satellite node and the target satellite node is calculated as: ; in, Indicates time slot/> The next page/> neighbor satellite nodes of satellite nodes,/> Represents the target satellite node, Indicates time slot/> The next page/> The spatial coordinates of neighbor satellite nodes of satellite nodes, Indicates the spatial coordinates of the target satellite node,/> Indicates time slot/> The next page/> The geographical distance factor between the neighbor satellite node of a satellite node and the target satellite node,/> represents the speed of light. 3. The method according to claim 2, characterized in that the topological distance factor of the neighbor satellite node and the target satellite node is calculated according to the virtual topological addresses of the neighbor satellite node and the target satellite node of the current satellite node, including: According to the virtual topological addresses of the neighbor satellite nodes and the target satellite node of the current satellite node, the topological distance factor of the neighbor satellite node and the target satellite node is calculated as: ; in, Indicates time slot/> The next page/> The orbit number of the satellite node's neighbor satellite node,/> Indicates time slot/> The next page/> The orbital sequence number of the neighbor satellite node of a satellite node,/> Indicates the orbit number of the target satellite node,/> Represents the sequence number within the orbit of the target satellite node,/> Indicates time slot/> The next page/> The topological distance factor between the neighbor satellite node of a satellite node and the target satellite node,/> Indicates the influence factor of orbital surface proximity on topological distance factor,/> Represents the influence factor of the sequential proximity within the orbit on the topological distance factor,/> Indicates the number of satellites in each orbit. 4. The method according to claim 3, characterized in that, obtaining the topological-geographical distance factor according to the geographical distance factor and the corresponding topological distance factor includes: According to the geographical distance factor and the corresponding topological distance factor, the topological-geographical distance factor is: ; in, Indicates time slot/> The next page/> The topology-geographic distance factor of the neighbor satellite node and the target satellite node of the satellite node,/> Represents the influence factor of topological distance factor on topological-geographical distance factor,/> Indicates the influence factor of geographical distance factor on topological-geographic distance factor. 5. The method according to claim 4, wherein the step of constructing the deep neural network of the routing decision agent includes: Construct a first deep neural network and a second deep neural network; the first deep neural network and the second deep neural network have the same network structure; wherein the first deep neural network serves as the main network and is used to estimate the state in real time - Action value; the second deep neural network serves as the target network and is used to calculate the target state-action value and provide it to the first deep neural network for updating network parameters. 6. The method according to claim 5, characterized in that the queue status identifier, queue occupancy rate and topology-geographic distance factor of each neighbor satellite node and the target satellite node are used as the observation state under the current time slot. The information is input into the deep neural network of the routing decision agent corresponding to the current satellite node, and the status-action value of the current satellite node forwarding the data stream to each neighbor satellite node is output, including: The queue status identifier, queue occupancy rate and topology-geographic distance factor of each neighbor satellite node and the target satellite node are used as the observation status information under the current time slot and input into the deep neural network of the routing decision agent corresponding to the current satellite node. in the network; Get a pre-built reward function; the expression of the reward function is: ; in, Indicates time slot/> The next page/> Reward function of satellite nodes,/> Indicates time slot/> The next page/> The queue status identifier of the neighbor satellite node of the satellite node,/> Indicates time slot/> The next page/> Queue occupancy rate of neighbor satellite nodes of satellite nodes; According to the observation status information and the reward function, the status-action value of the current satellite node forwarding the data flow to each neighbor satellite node is obtained. 7. The method according to claim 1, characterized in that, the method further comprises: The HELLO/ACK detection response message mechanism is used to exchange status information between satellite nodes through inter-satellite links; the status information includes: the IP address of the satellite node, virtual topology address, spatial coordinates, queue occupancy rate, and timestamp.