CN116600344A - Multi-layer MEC resource unloading method with power cost difference - Google Patents

Abstract

The application discloses a multi-layer MEC resource unloading method with electric power cost difference, which comprises the steps of firstly establishing a network model of the multi-layer MEC resource with the electric power cost difference; then establishing a communication model and a calculation model under different resource levels; allocating channel resources by using a sub-channel user recombination algorithm based on NOMA; and finally, converting the optimization problem into an equivalent reinforcement Learning problem by using a Q-Learning-based calculation unloading and resource allocation algorithm, and converging a Q table through training of the intelligent agent so as to guide the unloading decision of the base station intelligent agent. The problem is expressed as a mixed integer programming problem by combining unloading decision and resource allocation with the weighted sum of the time cost and unloading cost of all users as an optimization target, and a solution scheme for optimizing transmission and unloading based on NOMA and Q-Learning is provided. Simulation results show that the multi-layer MEC architecture is superior to the traditional single-layer MEC architecture, and meanwhile, the algorithm is superior to other basic algorithms in solving.

Description

A multi-layer MEC resource offloading method with power cost difference

Technical Field

The present invention relates to the field of Internet of Things and resource allocation, and in particular to a multi-layer MEC resource unloading method with power cost difference.

Background Art

In recent years, with the explosive growth of IoT services, many emerging business types have emerged, such as 3D games, telemedicine, and AI training. Due to the limitations of computing power and battery capacity of user equipment (UE), it is difficult to perform computing tasks that require a lot of computing power. In order to solve this problem, Multi-Access Edge Computing (MEC) was proposed to expand the computing power of user equipment. By offloading computing tasks to MEC servers with stronger computing power, the computing latency and energy consumption of user equipment can be significantly reduced. However, with the increasing number of user access, the existing local MEC facilities can no longer bear such a large-scale computing demand. Therefore, actively expanding more MEC resources to provide services to users has become an issue that we urgently need to solve.

Low earth orbit (LEO) satellites maintain their own power needs through solar and chemical power generation, and the power cost is relatively low.

With the access of LEO-MEC and W-CS, migrating part of the computing power requirements of users in the east can effectively alleviate the computing power burden of local MEC. Since computing resources at different levels vary greatly in response latency and power costs, this poses a great challenge to problem modeling.

The paper "Xie R, Tang Q, Wang Q, et al. Satellite-terrestrial integrated edge computing networks: Architecture, challenges, and open issues [J]. Ieee Network, 2020, 34 (3): 224-231" studied the architecture of MEC under satellite-ground integration, included satellite computing resources in multi-layer heterogeneous edge computing clusters, and proposed promising technical challenges, including collaborative computing offloading and multi-node task scheduling. The papers "Tang Z, Zhou H, Ma T, et al. Leveraging LEO Assisted Cloud-Edge Collaboration for Energy Efficient Computation Offloading[C]//2021IEEE Global Communications Conference. IEEE, 2021: 1-6." and "Cheng N, Lyu F, Quan W, et al. Space/aerial-assisted computing offloading for IoT applications: A learning-based approach[J]. IEEE Journal on Selected Areas in Communications, 2019, 37(5): 1117-1129" respectively present a MEC integrated network including LEO satellites, which offloads computing tasks to cloud servers via drones or LEO satellite relays. However, the processing capacity on the satellite is not considered, resulting in a waste of space resources. At the same time, offloading to cloud servers via relays will bring additional propagation delays, affecting the user experience quality. The paper "Song Z, Hao Y, Liu Y, et al. Energy-efficient multiaccess edge computing for terrestrial-satellite Internet of Things [J]. IEEE Internet of Things Journal, 2021, 8(18): 14202-14218" offloads some local computing tasks to the satellite MEC server through the ground satellite terminal (TST), and proposes an energy-saving computing offloading and resource allocation algorithm to minimize the weighted energy sum of local devices without violating the task tolerance delay. The paper "Tang Q, Fei Z, Li B, et al. Computation offloading in leo satellite networks with hybrid cloud and edge computing [J]. IEEE Internet of Things Journal, 2021, 8(11): 9164-9176" presents a hybrid cloud and LEO satellite MEC network with a three-layer computing architecture, and proposes a distributed algorithm that uses the alternating direction method of multipliers to approximate the optimal solution to minimize the total energy consumption of ground user computing offloading. It can be seen that these scholars use some optimization strategies to offload some tasks to satellites or ground clouds to reduce the energy consumption of local devices, but do not consider the energy consumption of satellite and ground cloud server computing and processing.

The paper "Wu J, Jia M, Zhang L, et al. DNNs Based Computation Offloading for LEO Satellite Edge Computing [J]. Electronics, 2022, 11 (24): 4108" considers the computational energy consumption of LEO-MEC in the system, proposes a deep learning-based LEO satellite edge computing network offloading algorithm, and optimizes the weighted sum of system energy consumption and latency. The paper "Cao X, Yang B, Shen Y, et al. Edge-Assisted Multi-Layer Offloading Optimization of LEO Satellite-Terrestrial Integrated Networks [J]. IEEE Journal on Selected Areas in Communications, 2022" studies a multi-layer multi-access MEC system, extends the MEC concept to the edge of LEO satellites, and formulates a joint optimization problem of computing and communication resources. By using the classic alternating optimization method to decompose and solve the original problem, the concept of low computational latency and low energy consumption of the scheme is verified.

The paper "Hossain M D, Sultana T, Hossain M A, et al. Fuzzy decision-based efficient task offloading management scheme in multi-tier MEC-enabled networks[J]. Sensors, 2021, 21(4): 1484" proposed a cloud-MEC collaborative task offloading scheme based on fuzzy decision-making. Users select the optimal task offloading target node based on server capacity, delay sensitivity and network conditions, which significantly improves the successful execution rate of task offloading and reduces the task completion time. The paper "Tong M, Wang X, Li S, et al. Joint Offloading Decision and Resource Allocation in Mobile Edge Computing-Enabled Satellite-Terrestrial Network[J]. Symmetry, 2022, 14(3): 564" proposed a joint offloading decision and resource allocation scheme for satellite-terrestrial MEC networks, which minimizes the completion delay of all terminal tasks and ensures the latency requirements of most users.

Summary of the invention

In order to make full use of the advantages of LEO satellites and cheap electricity in the western region, provide more computing offloading opportunities for user devices, and solve the offloading selection problem caused by the differences in multi-level computing resource offloading, the present invention provides a multi-layer MEC resource offloading method with electricity cost differences.

The technical solution for achieving the purpose of the present invention is:

A multi-layer MEC resource offloading method with power cost difference comprises the following steps:

(1) Establish a network model with multiple layers of MEC resources with different power costs;

(2) Establish communication models and computational models at different resource levels;

(3) Allocate channel resources using the NOMA-based sub-channel user reorganization algorithm;

(4) Using the Q-Learning-based computation offloading and resource allocation algorithm, the optimization problem is transformed into an equivalent reinforcement learning problem. By training the intelligent agent, the Q table is converged to guide the offloading decision of the base station agent.

Further:

The step (1) of establishing a network model of multi-layer MEC resources with different power costs includes:

1) The user equipment offloads the computing tasks to the target server for processing through the base station. The base station (BS) is the access point of the user equipment and integrates two types of equipment: traditional cellular base stations and ground satellite terminals. It uses C-band and Ka-band to communicate with ground equipment and low-orbit satellites respectively;

2) Assume that there are M users within the coverage of the base station. Each user has a computing task d _i (i∈M) that needs to be offloaded and the computing task cannot be divided. The user's task data set is recorded as There are three layers of computing resources that provide computing services to users;

3) User equipment uses NOMA technology to upload computing tasks to the base station. The base station controller uniformly schedules user tasks according to the user's task attributes and the real-time status of resources at all levels; when the computing task is unloaded on the local MEC server, a = 1 is recorded, otherwise a = 0; when the computing task is unloaded on the satellite MEC server, b = 1 is recorded, otherwise b = 0; when the computing task is unloaded on the western cloud server, c = 1 is recorded, otherwise c = 0. The unloading decision is recorded as

The three layers of computing resources that provide computing services to users are:

① Local MEC server on the base station side;

② Satellite MEC server carried on low-orbit satellite;

③Western cloud server built in the western region.

The step (2) of establishing communication models at different resource levels includes:

1) Using the NOMA transmission scheme, it is assumed that each subchannel can be occupied by two users at the same time, and the users in the subchannel are orthogonal to each other. Therefore, M users are divided into N pairs, that is, there are N subchannels, denoted as

2) At the receiving end, decoding is performed according to the principle of NOMA uplink channel gain descending decoding; when decoding the first user, the second user is regarded as interference, so the transmission rate of the first decoded user is:

When decoding the second user, there is no user signal to interfere, and the transmission rate is:

Among them, B is the subcarrier bandwidth; P represents the user's transmission power; n ₀ represents the noise power.

3) For user equipment within the coverage of the base station, the base station controller is uniformly used to batch process user tasks. Since the distance between the base station and the satellite is a fixed value, the transmission rate of the ground-satellite communication is set to a constant R _GS , and the transmission rate of the satellite-ground communication is set to a constant R _SG .

The step (2) of establishing a computing model at different resource levels includes:

1) The total cost of user task offloading is divided into time cost and offloading cost, and the weight factors ω and υ are used to represent their proportion in the total cost.

The time cost is composed of transmission, propagation and processing delays, which is related to the distance from the user to the target server; the offloading cost is composed of transmission overhead and computing overhead, which is related to energy consumption and local unit electricity price;

2) The computing power allocated by the local MEC server to UEi is f _i ^L , and the number of CPU cycles required for each bit of data is β (Cycles/bit). Therefore, the time cost of local offloading is:

The first term is the transmission delay from UE to BS; the second term is the processing delay of the computing task;

The transmission power of UEi is _Pi , and the transmission energy consumption from UE to BS is:

Using dynamic voltage and frequency scaling technology, the dynamic power consumption P is proportional to V ² f. Under the low voltage limit, the computing frequency f of the CPU chip is approximately linearly related to the power supply voltage V, that is, V = af. The power consumption of the CPU is modeled as P = εf ³ , where f is the computing frequency of the CPU and ε is the coefficient of the chip architecture. The computing energy consumption of user i on the local MEC server l is:

E _i,l ^C =ε(f _i ^L ) ² d _i β (6)

The local unit electricity price is p ^L , and the cost function of local unloading is:

U _i ^L (d,f)＝ωT _i ^L +υ(E _i ^T p ^L +E _i,l ^C p ^L ) (7)

3) The distance from BS to the access satellite is H, and the computing power allocated by the satellite MEC server to UEi is _fi ^S , so the time cost of satellite unloading is:

The first term is the transmission delay from UE to BS; the second and third terms are the uplink transmission and propagation delays of earth-satellite communication; the fourth term is the processing delay of the computing task;

The transmission power of BS is P _GS , then the transmission energy consumption from BS to satellite s is:

The computing energy consumption of UEi on satellite MEC server s is:

E _i,s ^C =ε(f _i ^S ) ² d _i β (10)

The unit electricity price of the satellite server is p ^S , and the cost function of satellite unloading is:

U _i ^S (d,f)＝ωT _i ^S +υ(E _i ^T p ^L +E _gs ^T p ^L +E _i,s ^C p ^S ) (11)

4) The computing power allocated by the western cloud server to UEi ^is _fiW , so the time cost of unloading in the west is:

The first item in the formula is the transmission delay from UE to BS; the second and third items are the transmission delays of ground-satellite and satellite-ground communications; the fourth item is the propagation delay from BS to the western cloud server via satellite relay; and the fifth item is the processing delay of the computing task in the western cloud server.

The transmission power of the satellite is P _SG , and the transmission energy consumption from satellite s to western cloud server w is:

The computing energy consumption of UEi on the western cloud server w is:

E _i,w ^C =ε(f _i ^W ) ² d _i β (13)

The unit electricity price of the western cloud server is p ^W , and the cost function of unloading in the west is:

U _i ^W (d,f)=ωT _i ^W +υ(E _i ^T p ^L +E _gs ^T p ^L +E _sg ^T p ^S +E _i,w ^C p ^W ) (14).

The step (3) allocates channel resources by using a NOMA-based sub-channel user reorganization algorithm, including:

1) Use Represents a collection of users; represents a set of subchannels, where each subchannel contains 2 users; represents the sum of the rates of users in each subchannel, where

2) Select one user from each of the two sub-channels to swap positions, generate new sub-channel user combinations n′ _i and n′ _j (i≠j), and calculate q′ _i and q′ _j . If the following inequality is satisfied, the user reorganization is successful:

The Q-Learning-based computation offloading and resource allocation algorithm in step (4) converts the optimization problem into an equivalent reinforcement learning problem for solution, including:

1) Convert the optimization problem into an equivalent reinforcement learning problem, namely:

State space: The state space is the set of available resources of the target server, expressed as S = { _FL , _FS , _FW };

Action space: The action space consists of the offloading decision vector and resource allocation vector It consists of two parts, expressed as A = {af ₁ …af _M ,bf ₁ …bf _M ,cf ₁ …cf _M };

Reward: The reward function for executing action ^ak in state ^sk is defined as R( ^sk , ^ak ) = -U( ^sk , ^ak ).

2) The base station controller selects the action with the largest known action value with probability ε (0 < ε < 1); and randomly selects an action with probability 1-ε to execute:

3) During the process of utilization or exploration, each action needs to be updated through formula (17); after repeated training, until the Q table converges, it can guide the agent to choose the optimal strategy:

Where R represents the immediate reward obtained by executing action a ^k in state s ^k ; It represents the maximum Q value among all actions that can be taken after state transfer; α is the learning rate, which refers to the proportion of the new Q value in the whole; γ is the discount rate.

The advantages of the present invention are: for multi-layer MEC resources with different power costs, the weighted sum of minimizing the time cost and unloading cost of all users is taken as the optimization goal, the problem is expressed as a mixed integer programming problem by combining unloading decision and resource allocation, and a solution for optimizing transmission and unloading based on NOMA and Q-Learning is proposed. The simulation results show that the multi-layer MEC architecture of the present application is better than the traditional single-layer MEC architecture, and it is verified that the algorithm is better than other basic algorithms in solving the problem.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the structure of a network model according to an embodiment of the present invention;

FIG2 is a graph showing an iterative process of optimizing the average transmission rate in an embodiment of the present invention;

FIG3 is a schematic diagram of the total cost change of the algorithm training process according to an embodiment of the present invention;

FIG4 is a schematic diagram showing cost comparison under different transmission schemes according to an embodiment of the present invention;

FIG5 is a schematic diagram showing cost comparison under different unloading strategies in an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

Example:

As shown in Figure 1, it is a network model with three layers of computing resources (edge, satellite, and cloud). In this model, user devices can offload computing tasks to the target server for processing through the base station. Among them, the base station serves as the access point of the user device. It integrates two types of equipment, the traditional cellular base station and the ground satellite terminal, and uses the C band and Ka band to communicate with the ground equipment and low-orbit satellites respectively.

Assume that there are M users within the coverage of the base station. Each user has a computing task d _i (i∈M) that needs to be offloaded and the computing task cannot be divided. The user's task data set is recorded as The three layers of computing resources that provide computing services to users are:

(1) Local MEC server on the base station side;

(2) Satellite MEC server on low-orbit satellite;

(3) Western cloud server built in the western region.

The user device uploads the computing task to the base station using NOMA technology. The base station controller schedules the user tasks in a unified manner according to the user's task attributes and the real-time status of resources at each level. When the computing task is unloaded on the local MEC server, a = 1 is recorded, otherwise a = 0; when the computing task is unloaded on the satellite MEC server, b = 1 is recorded, otherwise b = 0; when the computing task is unloaded on the western cloud server, c = 1 is recorded, otherwise c = 0. We record the unloading decision as

The NOMA transmission scheme is adopted in the transmission stage. In this scheme, it is assumed that each subchannel can be occupied by two users at the same time, and the users in the subchannel are orthogonal to each other, so M users are divided into N pairs, that is, there are N subchannels, recorded as At the receiving end, decoding is performed according to the principle of NOMA uplink channel gain descending decoding. When decoding the first user, the second user is regarded as interference, so the transmission rate of the first decoded user is:

When decoding the second user, there is no user signal to interfere, and the transmission rate is:

Among them, B is the subcarrier bandwidth; P represents the user's transmission power; n ₀ represents the noise power.

Since this paper batches user tasks through the base station, the distance between the base station and the satellite is a constant, so the physical distance cannot be used as the only condition to measure the channel gain and the user grouping of the satellite-to-ground channel. In order to simplify the model, we set the transmission rate of the ground-to-satellite communication to a constant R _GS and the transmission rate of the satellite-to-ground communication to a constant R _SG .

In the process of task offloading, time delay and offloading overhead are often the two issues that users are most concerned about. We divide the total cost of user task offloading into time cost and offloading cost, and use weight factors ω and υ to represent their proportion of the total cost.

The time cost is composed of transmission, propagation and processing delays, which is related to the distance from the user to the target server; the offloading cost is composed of transmission overhead and computing overhead, which is related to energy consumption and the local unit electricity price.

(1) Local MEC offloading

In the local offloading solution, since the computing task is executed on the local MEC server on the base station side, the propagation delay can be ignored. The computing power allocated by the local MEC server to UEi is f _i ^L , and the number of CPU cycles required for each bit of data is β (Cycles/bit), so the time cost of local offloading is:

The first term in the formula is the transmission delay from UE to BS; the second term is the processing delay of the computing task.

The transmission power of UEi is _Pi , and the transmission energy consumption from UE to BS is:

The most advanced CPU architecture usually adopts dynamic voltage and frequency scaling technology. The dynamic power consumption P is proportional to V ² f. Under the low voltage limit, the computing frequency f of the CPU chip is approximately linearly related to the power supply voltage V, that is, V = af. We model the power consumption of the CPU as P = εf ³ , where f is the computing frequency of the CPU and ε is the coefficient of the chip architecture. Then the computing energy consumption of user i on the local MEC server l is:

E _i,l ^C =ε(f _i ^L ) ² d _i β (23)

The local unit electricity price is p ^L , and the cost function of local unloading is:

U _i ^L (d,f)＝ωT _i ^L +υ(E _i ^T p ^L +E _i,l ^C p ^L ) (24)

(2) Satellite MEC offloading

In the satellite offloading scheme, the distance between the BS and the access satellite is H, and the computing power allocated by the satellite MEC server to UEi is _fi ^S , so the time cost of satellite offloading is:

The first term in the formula is the transmission delay from UE to BS; the second and third terms are the uplink transmission and propagation delays of earth-satellite communication; and the fourth term is the processing delay of the computing task.

The transmission power of BS is P _GS , then the transmission energy consumption from BS to satellite s is:

The computing energy consumption of UEi on satellite MEC server s is:

E _i,s ^C =ε(f _i ^S ) ² d _i β (27)

The unit electricity price of the satellite server is p ^S , and the cost function of satellite unloading is:

U _i ^S (d,f)＝ωT _i ^S +υ(E _i ^T p ^L +E _gs ^T p ^L +E _i,s ^C p ^S ) (28)

(3) Uninstallation of Western Cloud

In the western offloading scheme, the computing task needs to be uploaded to the western cloud server via satellite relay. When the BS and the cloud server are not within the coverage of the same LEO satellite, the intersatellite link is required to assist the computing task in data transmission. Due to the unpredictability of the intersatellite link state, this paper ignores the transmission and propagation delay of the computing task from one LEO satellite to another. The computing power allocated by the western cloud server to UEi is _fi ^W , so the time cost of western offloading is:

The first item in the formula is the transmission delay from UE to BS; the second and third items are the transmission delays of ground-satellite and satellite-ground communications; the fourth item is the propagation delay from BS to the western cloud server via satellite relay; and the fifth item is the processing delay of the computing task in the western cloud server.

The transmission power of the satellite is P _SG , and the transmission energy consumption from satellite s to western cloud server w is:

The computing energy consumption of UEi on the western cloud server w is:

E _i,w ^C =ε(f _i ^W ) ² d _i β (31)

The unit electricity price of the western cloud server is p ^W , and the cost function of unloading in the west is:

U _i ^W (d,f)＝ωT _i ^W +υ(E _i ^T p ^L +E _gs ^T p ^L +E _sg ^T p ^S +E _i,w ^C p ^W ) (32)

Since the calculated result is much smaller than the size of the input data, this paper does not consider the data return link.

For multi-layer MEC resources with different latency and power costs, our goal is to minimize the weighted sum of all user time costs and offloading costs under limited computing power. To this end, we formulate the optimization problem as:

In problem (33), C1 means that the task is inseparable and the computing task can only be completely executed by the local MEC server, satellite MEC server or western cloud server; C2-C4 means that the computing resources allocated to the UE cannot exceed the computing power limit of the target server.

Due to uninstall decision is a binary variable, and the resource allocation vector is dynamically changing, so this problem is a mixed integer programming problem, which is NP-hard and very complicated to solve using traditional methods. We propose a solution strategy based on Q-Learning.

Sub-channel user reorganization algorithm based on NOMA

In the transmission phase, we use NOMA technology to allocate channel resources. Since the distances from different UEs to the BS are different, that is, the user channel gains are different, different user combinations in the same subchannel will affect the user's transmission rate. Represents a collection of users; represents a set of subchannels, where each subchannel contains 2 users; represents the sum of the rates of users in each subchannel, where

Definition 1: Select one user from each of any two sub-channels to swap their positions, generate new sub-channel user combinations n _i ′ and n′ _j (i≠j), and calculate q _i ′ and q′ _j . If the following inequality is satisfied, the user reorganization is successful.

In order to find the best user combination for each sub-channel, according to Definition 1, we adopt a NOMA-based sub-channel user reorganization algorithm to improve the average transmission rate of users.

Computation offloading and resource allocation algorithm based on Q-Learning

Q-Learning is a model-free reinforcement learning process that aims to enable an agent to learn a strategy in an unfamiliar environment to maximize the cumulative reward obtained by the agent. For multi-layer MEC resources, a Q-Learning-based computation offloading and resource allocation algorithm (Q-CORAA) is adopted.

The system model is defined by three key elements: state, action, and reward. In order to transform the optimization problem into an equivalent reinforcement learning problem, we express these key elements as:

(1) State space: The state space is the set of available resources of the target server, expressed as

S = { _FL , _FS , _FW }.

(2) Action space: The action space is composed of the unloading decision vector and resource allocation vector Therefore, the action space can be expressed as

A＝{af ₁ ...af _M ,bf ₁ ...bf _M ,cf ₁ ...cf _M }.

(3) Reward: The reward is related to the objective function. Since our optimization problem is to minimize the cost and the goal of Q-Learning is to maximize the reward, we define the reward function for performing action ^ak in state ^sk as R( ^sk , ^ak )=-U( ^sk , ^ak ).

In Q-Learning, the optimal strategy can be obtained through the Q function. Therefore, our first task is how to train a Q function and make it converge. In this paper, both the state space S and the action space A are finite. This is a finite Markov decision process. We can regard the Q function as a table with |S| rows and |A| columns storing Q values, called a Q table, and select actions according to the guidance of the Q table.

Initially, the Q table is empty and cannot guide the agent to choose the best action. Therefore, it needs to interact with the environment to collect data, fill in and update the Q table based on experience. In this process, we hope that the agent can use the existing experience to perform the best action, and also hope that it can explore some unknown action options. In the trade-off between utilization and exploration, ε-greedy is a commonly used strategy.

Definition: ε-greedy strategy

When the agent makes a decision, it selects the action with the largest known action value with a probability of ε (0 < ε < 1); and randomly selects an action to execute with a probability of 1-ε.

In the process of exploitation or exploration, each action needs to be updated by formula (17). After repeated training, the Q table can guide the agent to choose the optimal strategy until it converges.

Where R represents the immediate reward obtained by executing action a ^k in state s ^k ; represents the maximum Q value among all actions that can be taken after state transition; α is the learning rate, which refers to the proportion of the new Q value in the whole; γ is the discount rate, which defines the importance of future rewards. The larger the value, the more emphasis is placed on long-term rewards. The optimization problem (33) is solved by using Algorithm 2.

Simulation Settings

The simulation was performed on MATLAB R2018b. A circular simulation area with a radius of 500m was considered. The BS was located at the center, covering the entire area, and the low-orbit satellite was located 500km above the simulation area. The UEs were randomly distributed in the area. Each UE input the number of task bits d _i = (1~3)Mbits, and the number of CPU cycles required to calculate one task bit β = 120cycles/bit. The computing power of the local MEC, satellite MEC, and western cloud server was set to 1Gcycles/s, 2Gcycles/s, and 3Gcycles/s, respectively. The unit electricity price was dynamically adjusted according to the difficulty of obtaining local power resources. In this paper, p ^L = 0.8, p ^S = 0.6, and p ^W = 0.4. The transmit power of each UE was 20dBm, the subcarrier bandwidth was 1MHz, the noise power was -100dBm, and the path loss from UE to BS was L(d) = 15.3+37.6lgd. Assume that there is always a LEO satellite providing full coverage of the area, the transmission power of the BS and the LEO satellite is 23dBm, and the satellite-to-ground communication rate is 10Mbps.

Table 1 Main simulation parameters

Performance Evaluation

In Figure 2, we show the iteration process of N-SURA. The average transmission rate of 10 users reaches a stable level after a limited number of iterations. It can be seen that compared with NOMA, N-SURA can achieve a higher average transmission rate.

In Figure 3, we show the training process of the agent. It can be seen that as the number of training times increases, the agent gradually approaches the optimal unloading strategy, keeping the total cost at a low level.

In Figure 4, we used different transmission schemes to solve the optimization problem using the Q-CORA algorithm. It can be seen that under different numbers of users, N-SURA has the best effect, followed by NOMA, and OFDMA is the worst.

From Figure 5, we can see that under different numbers of users, the total cost of traditional single-layer MEC computing offloading is the highest. The main reason is that the local unit electricity price is slightly higher than other levels, which greatly increases the cost of user offloading, verifying the feasibility of the multi-layer MEC architecture proposed in this paper. At the same time, under the multi-layer MEC architecture, the total cost of using the Q-CORA algorithm is the lowest, indicating that the algorithm proposed in this paper is obviously better than other offloading algorithms.

Claims (7)

1. A multi-layer MEC resource offloading method with power cost difference, characterized by comprising the following steps: (1) Establish a network model with multiple layers of MEC resources with different power costs; (2) Establish communication models and computational models at different resource levels; (3) Allocate channel resources using the NOMA-based sub-channel user reorganization algorithm; (4) Using the Q-Learning-based computation offloading and resource allocation algorithm, the optimization problem is transformed into an equivalent reinforcement learning problem. By training the intelligent agent, the Q table is converged to guide the offloading decision of the base station agent. 2. The multi-layer MEC resource offloading method according to claim 1 is characterized in that: step (1) of establishing a network model of multi-layer MEC resources with different power costs comprises: 1) The user equipment offloads the computing tasks to the target server for processing through the base station. The base station (BS) is the access point of the user equipment and integrates two types of equipment: traditional cellular base stations and ground satellite terminals. It uses C-band and Ka-band to communicate with ground equipment and low-orbit satellites respectively; 2) Assume that there are M users within the coverage of the base station. Each user has a computing task d _i (i∈M) that needs to be offloaded and the computing task cannot be divided. The user's task data set is recorded as There are three layers of computing resources that provide computing services to users; 3) User equipment uses NOMA technology to upload computing tasks to the base station. The base station controller uniformly schedules user tasks according to the user's task attributes and the real-time status of resources at all levels; when the computing task is unloaded on the local MEC server, a = 1 is recorded, otherwise a = 0; when the computing task is unloaded on the satellite MEC server, b = 1 is recorded, otherwise b = 0; when the computing task is unloaded on the western cloud server, c = 1 is recorded, otherwise c = 0. The unloading decision is recorded as 3. The multi-layer MEC resource offloading method according to claim 2 is characterized in that: the three-layer computing resources providing computing services for users are respectively: ① Local MEC server on the base station side; ② Satellite MEC server carried on low-orbit satellite; ③Western cloud server built in the western region. 4. The multi-layer MEC resource offloading method according to claim 1 is characterized in that: step (2) of establishing communication models under different resource levels includes: 1) Using the NOMA transmission scheme, it is assumed that each subchannel can be occupied by two users at the same time, and the users in the subchannel are orthogonal to each other. Therefore, M users are divided into N pairs, that is, there are N subchannels, denoted as 2) At the receiving end, decoding is performed according to the principle of NOMA uplink channel gain descending decoding; when decoding the first user, the second user is regarded as interference, so the transmission rate of the first decoded user is: When decoding the second user, there is no user signal to interfere, and the transmission rate is: Among them, B is the subcarrier bandwidth; P represents the user's transmission power; n ₀ represents the noise power. 3) For user equipment within the coverage of the base station, the base station controller is uniformly used to batch process user tasks. Since the distance between the base station and the satellite is a fixed value, the transmission rate of the ground-satellite communication is set to a constant R _GS , and the transmission rate of the satellite-ground communication is set to a constant R _SG . 5. The multi-layer MEC resource offloading method according to claim 1 is characterized in that: step (2) of establishing a computing model at different resource levels includes: 1) The total cost of user task offloading is divided into time cost and offloading cost, and the weight factors ω and υ are used to represent their proportion in the total cost. The time cost is composed of transmission, propagation and processing delays, which is related to the distance from the user to the target server; the offloading cost is composed of transmission overhead and computing overhead, which is related to energy consumption and local unit electricity price; 2) The computing power allocated by the local MEC server to UEi is f _i ^L , and the number of CPU cycles required for each bit of data is β (Cycles/bit). Therefore, the time cost of local offloading is: The first term is the transmission delay from UE to BS; the second term is the processing delay of the computing task; The transmission power of UEi is _Pi , and the transmission energy consumption from UE to BS is: Using dynamic voltage and frequency scaling technology, the dynamic power consumption P is proportional to V ² f. Under the low voltage limit, the computing frequency f of the CPU chip is approximately linearly related to the power supply voltage V, that is, V = af. The power consumption of the CPU is modeled as P = εf ³ , where f is the computing frequency of the CPU and ε is the coefficient of the chip architecture. The computing energy consumption of user i on the local MEC server l is: E _i,l ^C =ε(f _i ^L ) ² d _i β (6) The local unit electricity price is p ^L , and the cost function of local unloading is: U _i ^L (d,f)＝ωT _i ^L +υ(E _i ^T p ^L +E _i,l ^C p ^L ) (7) 3) The distance from BS to the access satellite is H, and the computing power allocated by the satellite MEC server to UEi is _fi ^S , so the time cost of satellite unloading is: The first term is the transmission delay from UE to BS; the second and third terms are the uplink transmission and propagation delays of earth-satellite communication; the fourth term is the processing delay of the computing task; The transmission power of BS is P _GS , then the transmission energy consumption from BS to satellite s is: The computing energy consumption of UEi on satellite MEC server s is: E _i,s ^C =ε(f _i ^S ) ² d _i β (10) The unit electricity price of the satellite server is p ^S , and the cost function of satellite unloading is: U _i ^S (d,f)＝ωT _i ^S +υ(E _i ^T p ^L +E _gs ^T p ^L +E _i,s ^C p ^S ) (11) 4) The computing power allocated by the western cloud server to UEi ^is _fiW , so the time cost of unloading in the west is: The first term is the transmission delay from UE to BS; the second and third terms are the transmission delays of ground-satellite and satellite-ground communications; the fourth term is the propagation delay from BS to the western cloud server via satellite relay; and the fifth term is the processing delay of the computing task in the western cloud server. The transmission power of the satellite is P _SG , and the transmission energy consumption from satellite s to western cloud server w is: The computing energy consumption of UEi on the western cloud server w is: E _i,w ^C =ε(f _i ^W ) ² d _i β (13) The unit electricity price of the western cloud server is p ^W , and the cost function of unloading in the west is: U _i ^W (d,f)=ωT _i ^W +υ(E _i ^T p ^L +E _gs ^T p ^L +E _sg ^T p ^S +E _i,w ^C p ^W ) (14). 6. The multi-layer MEC resource offloading method according to claim 1 is characterized in that: the step (3) uses the NOMA-based sub-channel user reorganization algorithm to allocate channel resources, including: 1) Use Represents a collection of users; represents a set of subchannels, where each subchannel contains 2 users; represents the sum of the rates of users in each subchannel, where 2) Select one user from each of the two sub-channels to swap positions, generate new sub-channel user combinations n′ _i and n′ _j (i≠j), and calculate q′ _i and q′ _j . If the following inequality is satisfied, the user reorganization is successful: 7. The multi-layer MEC resource offloading method according to claim 1 is characterized in that: the Q-Learning-based computation offloading and resource allocation algorithm in step (4) converts the optimization problem into an equivalent reinforcement learning problem for solution, including: 1) Convert the optimization problem into an equivalent reinforcement learning problem, namely: State space: The state space is the set of available resources of the target server, expressed as S = { _FL , _FS , _FW }; Action space: The action space consists of the offloading decision vector and resource allocation vector It consists of two parts, expressed as A = {af ₁ …af _M ,bf ₁ …bf _M ,cf ₁ …cf _M }; Reward: The reward function for executing action ^ak in state ^sk is defined as R( ^sk , ^ak ) = -U( ^sk , ^ak ). 2) The base station controller selects the action with the largest known action value with probability ε (0 < ε < 1); and randomly selects an action with probability 1-ε to execute: 3) During the process of utilization or exploration, each action needs to be updated through formula (17); after repeated training, until the Q table converges, it can guide the agent to choose the optimal strategy: Where R represents the immediate reward obtained by executing action a ^k in state s ^k ; It represents the maximum Q value among all actions that can be taken after state transfer; α is the learning rate, which refers to the proportion of the new Q value in the whole; γ is the discount rate.