CN116612636B - Signal lamp cooperative control method based on multi-agent reinforcement learning - Google Patents

Abstract

The present invention proposes a collaborative control method for traffic lights based on multi-agent reinforcement learning and multi-modal signal perception, which includes: collecting data from various sensors and defining multi-modality, obtaining information in real time through data fusion technology; using collaborative vehicles The road multi-agent reinforcement learning algorithm performs collaborative control of traffic lights and vehicles; preprocesses data collected from various sensors, uses feature fusion methods to fuse data from different modalities, and builds a local state space for each agent; for traffic lights Design action spaces for agents and vehicle agents; design reward functions for multi-agent reinforcement learning based on the goal of traffic flow control; design communication protocols suitable for vehicle-road collaborative control scenarios; use historical data or simulation environments to strengthen multi-agents The learning model is trained to find the optimal strategy. By introducing vehicles as intelligent entities, the present invention achieves more effective vehicle-road collaboration and further improves the traffic control effect.

Description

Traffic light collaborative control method based on multi-agent reinforcement learning

Technical field

The invention belongs to the field of vehicle-road collaboration, and in particular relates to a signal lamp collaborative control method based on multi-agent reinforcement learning.

Background technique

As urban traffic becomes increasingly busy, traditional signal light control methods are no longer able to meet the efficient traffic needs of modern cities. In order to solve this problem, researchers have begun to use intelligent transportation systems (ITS) to improve road traffic efficiency. Among them, the traffic light control system based on multi-agent reinforcement learning and multi-modal signal perception has attracted much attention.

Traditional signal light control methods are usually based on fixed signal periods or predetermined traffic flow patterns, which lack adaptability to real-time traffic conditions. Therefore, there is an urgent need for a collaborative method that breaks through the limitations of traditional traffic light control methods and improves the adaptability to real-time traffic conditions.

Contents of the invention

The purpose of the present invention is to propose a signal light collaborative control method based on multi-agent reinforcement learning, by introducing vehicles as intelligent agents, to achieve more effective vehicle-road collaboration and further improve the traffic control effect.

In order to achieve the above objectives, the present invention provides a signal light collaborative control method based on multi-agent reinforcement learning, which method includes:

S1. Collect data from various sensors and perform multi-modal definition, and obtain information in real time through data fusion technology;

S2. Use the collaborative vehicle-road multi-agent reinforcement learning algorithm to collaboratively control traffic lights and vehicles, and find the optimal strategy through learning to achieve efficient traffic flow control;

S3. Preprocess data collected from various sensors, use feature fusion methods to fuse data from different modalities, and build a local state space for each agent;

S4. Design an action space for the agent;

S5. Design a reward function for multi-agent reinforcement learning based on the goal of traffic flow control;

S6. Design communication protocol;

S7. Use historical data or simulation environment to train the multi-agent reinforcement learning model and find the optimal strategy.

Further, the multi-modal definition in S1 includes visual mode and radar mode; the image data collected by the visual mode through the camera; the distance and speed information collected by the radar module through the radar; the information includes road conditions. information, vehicle position and speed.

Further, the data fusion technology is specifically:

S1.1. Model the scene as a graph structure;

S1.2. Extract features from data of each modality;

S1.3. Feature fusion based on graph convolutional neural network;

S1.4. Output the reinforcement learning status.

Further, the objective function of the collaborative vehicle-road multi-agent reinforcement learning algorithm is defined as:

The reward function is defined as R(s,a), where s represents the state and a represents the action of the agent. By adjusting the action of the agent to maximize the cumulative reward, it is expressed as:

J(θ)＝∑_t R(s_t,a_t)

Among them, J(θ) represents the objective function; s_t represents the state of the agent at time t; a_t represents the action taken by the agent at time t;

Then the loss function L(θ) is expressed as:

L(θ)＝0.5*E[(R(s,a)+γ*max_a'Q(s',a';θ')-Q(s,a;θ))^2]

Among them, E[·] represents the expected value, θ represents the parameters of the current agent, θ' represents the parameters of the target agent, γ is the discount factor, a' represents the action taken by the agent in state s', Q(s, a; θ) is the action value function, used to estimate the cumulative reward of taking action a in state s; Q(s', a'; θ') represents the network's action value evaluation of action a' in state s' , used to measure the quality of action a' in state s'.

Further, the implementation steps of the objective function of the collaborative vehicle-road multi-agent reinforcement learning algorithm include:

S2.1, using centralized training and distributed execution strategies, through centralized training in the training phase, collaboration between agents is achieved. In the execution phase, each agent uses a distributed strategy to make decisions based on local states. ;

S2.2. Obtain a comprehensive state space containing multi-modal information through the data of various sensors in step S1, so that the agent can more accurately perceive traffic conditions;

S2.3. Treat vehicles and traffic lights as different intelligent entities to achieve collaborative control between vehicles and traffic lights to improve traffic smoothness.

Further, the data in step S3 includes vehicle data, road condition data and traffic light data;

Further, the state space is expressed as follows:

s={vehicle data, road condition data, traffic light data}.

Further, the step S4 specifically includes:

S4.1. Discretize the control strategy of the traffic light into a series of optional actions;

S4.2. Dynamically adjust the phase setting of the signal light based on real-time traffic data;

S4.3. Design an adaptive traffic light control strategy.

Further, the communication protocol includes communication between vehicles and roadside facilities, communication between signal lights, communication between the central controller and signal lights, and data fusion and processing.

Further, the step S7 specifically includes: using historical data or simulation environment to train the multi-agent reinforcement learning model, finding the optimal strategy, and deploying the optimal strategy to the signal light control system.

The beneficial technical effects of the present invention lie in at least the following points:

(1) Multi-modal signal sensing technology provides rich real-time traffic information for the signal light control system. By integrating data from multiple sensors such as cameras, radars, and vehicle-mounted sensors, the system can more accurately perceive traffic conditions and provide more targeted decision-making basis for signal light control.

(2) Through the multi-modal feature fusion method based on graph convolutional neural network, the topological relationship between vehicles and traffic lights can be better captured. At the same time, through the above methods, multi-modal signal sensing data can be fused into a unified state space, providing richer and more accurate information for multi-agent reinforcement learning algorithms.

(3) Through the optimal strategy of the present invention, our signal light control system will have strong dynamic adjustment capabilities, be able to better adapt to changing traffic conditions in practical applications, and improve the overall signal light control effect.

Description of the drawings

The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. For those of ordinary skill in the art, without exerting creative efforts, other embodiments can be obtained based on the following drawings. Picture attached.

Figure 1 is a flow chart of the signal light collaborative control method based on multi-agent reinforcement learning of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and cannot be understood as limiting the present invention.

Embodiment 1

In one or more embodiments, as shown in Figure 1, a signal light collaborative control method based on multi-agent reinforcement learning is disclosed, including the following steps:

S1. Collect data from various sensors and perform multi-modal definition, and obtain information in real time through data fusion technology;.

Specifically, step S1 is responsible for collecting data from various sensors (such as cameras, radars, vehicle-mounted sensors, etc.), and obtaining information such as road condition information, vehicle position and speed in real time through data fusion technology. This information will be used to construct the state space of multi-agent reinforcement learning.

The definition of multimodality specifically includes:

a. Visual mode: Image data collected through cameras can provide information such as vehicle position, shape, speed, etc. The cameras can be installed on the roadside or in vehicles to transmit image data in real time.

b. Radar mode: The distance and speed information collected through radar helps to accurately detect the position, speed and distance of the vehicle. The radar can be mounted on the roadside or on a vehicle and transmit distance data in real time. These data can be transmitted to the signal light control system in real time through the vehicle communication system (such as V2X communication).

The steps of data fusion in step S1 specifically include:

S1.1. Model the scene as a graph structure. In the traffic light control scenario, there is a certain topological relationship between vehicles and traffic lights. We can model the scene as a graph structure, with vehicles and traffic lights as nodes and their interrelationships as edges;

S1.2. Extract features from the data of each modality. For the data of each modality, feature extraction is first performed. For the visual modality, you can use a convolutional neural network (CNN) to extract features; for the radar modality, you can use a one-dimensional convolutional neural network or a recurrent neural network (RNN) to extract features; for the vehicle sensor modality, you can use fully connected Neural network (FCNN) extracts features. Then, the extracted features are represented as node features.

S1.3. Feature fusion based on graph convolutional neural network. The extracted multi-modal features are input into the graph convolutional neural network (GCN) as node features. GCN can capture the topological relationship between nodes while maintaining node characteristics. Through the propagation and convergence operations of GCN, we can obtain comprehensive features that contain multi-modal information and topological relationships between nodes.

S1.4. Output the reinforcement learning status. The features after GCN fusion are used as the state space of reinforcement learning for subsequent multi-agent reinforcement learning algorithms.

Through this multi-modal feature fusion method based on graph convolutional neural network, we can better capture the topological relationship between vehicles and traffic lights. At the same time, through the above methods, multi-modal signal sensing data can be fused into a unified state space, providing richer and more accurate information for multi-agent reinforcement learning algorithms.

S2. The collaborative vehicle-road multi-agent reinforcement learning algorithm is used to collaboratively control the traffic lights and vehicles. The traffic lights and vehicles are regarded as multiple agents, and efficient traffic flow control is achieved by finding the optimal strategy through learning.

Specifically, the collaborative vehicle-road multi-agent reinforcement learning algorithm is called CVR-MARL, which stands for CollaborativeVehicle-Road Multi-Agent Reinforcement Learning. The objective function of CVR-MARL is to maximize the cumulative reward of the system. In this model, traffic lights and vehicles are regarded as multiple agents, which need to learn to find optimal strategies to achieve efficient traffic flow control.

The objective function of CVR-MARL is defined as:

The reward function is defined as R(s,a), where s represents the state (feature fusion result based on multi-modal perception), and a represents the action of the agent. By adjusting the action of the agent to maximize the cumulative reward, it is expressed as :

J(θ)＝∑_t R(s_t,a_t)

Among them, J(θ) represents the objective function; s_t represents the state of the agent at time t; a_t represents the action taken by the agent at time t;

Then the loss function L(θ) is expressed as:

L(θ)＝0.5*E[(R(s,a)+γ*max_a'Q(s',a';θ')-Q(s,a;θ))^2]

Among them, E[·] represents the expected value, θ represents the parameters of the current agent, θ' represents the parameters of the target agent, γ is the discount factor, a' represents the action taken by the agent in state s', Q(s, a; θ) is the action value function, used to estimate the cumulative reward of taking action a in state s; Q(s', a'; θ') represents the network's action value evaluation of action a' in state s' , used to measure the quality of action a' in state s'.

Among them, the specific methods to achieve this goal are:

S2.1. Multi-agent collaboration: Centralized training and distributed execution strategies are used to achieve collaboration between agents through centralized training in the training phase. In the execution phase, each agent (including vehicles and traffic lights) Use distributed strategies to make decisions based on local states;

S2.2. Construction of state space: Obtain a comprehensive state space containing multi-modal information through the data of various sensors in step S1, so that the agent can more accurately perceive traffic conditions and construct a state space;

S2.3. Vehicle-road collaboration: Treat vehicles and traffic lights as different intelligent entities to achieve collaborative control between vehicles and traffic lights to improve traffic smoothness.

S3. Preprocess the data collected from various sensors, use the feature fusion method to fuse different modal data together, and build a local state space for each agent.

Specifically, based on the data collected by the multi-modal signal sensing module, a local state space is constructed for each agent (signal light and vehicle). Specifically, CVR-MARL collects multi-modal data from three aspects: vehicles, road conditions and signal lights. Includes the following data:

Vehicle data:

a) Position information: vehicle’s latitude and longitude, heading angle, etc.

b) Speed information: the real-time speed of the vehicle.

c) Acceleration information: real-time acceleration of the vehicle.

d) Vehicle type: such as cars, trucks, buses, etc.

e) Vehicle communication data: communication information between vehicles, such as vehicle-to-vehicle data (V2V), etc.

Traffic data:

a) Road structure: road width, number of lanes, separation zones and other information.

b) Traffic flow: the number and density of vehicles in each lane, etc.

c) Road environment information: road conditions, weather, lighting and other factors.

Traffic light data:

a) Status information: the current status of the signal light (red light, green light, yellow light).

b) Remaining time: the remaining time for the signal light status to change.

c) Signal light control strategy: such as fixed period control, induction control, etc.

d) Vehicle-to-signal light communication data: communication information between vehicles and signal lights, such as vehicle-to-road communication data (V2I), etc.

Based on the collected multimodal data, the state space can be constructed as follows:

s={vehicle data, traffic data, traffic light data}

When constructing the state space, the collected multimodal data need to be preprocessed to eliminate the dimensional and scale differences between the data. For example, the data can be normalized so that it falls within the same range. Next, feature fusion methods are used to fuse data from different modalities together to generate a comprehensive state representation. This comprehensive state representation can make full use of multi-modal information to provide CVR-MARL with richer environmental perception, thereby better realizing vehicle-road collaborative control.

S4. Design feasible action spaces for the traffic light agent and vehicle agent.

Specifically, feasible action spaces are designed for the traffic light agent and the vehicle agent. When designing the action space, we need to consider the control strategy of the signal light so that the action space can adapt to different road conditions. In order to realize the dynamic signal light action space, the following methods are used:

S4.1. Discretize the control strategy of the signal light into a series of optional actions. For example, it can be divided into several discrete actions according to parameters such as phase, duration, and change rate of the signal light. This approach simplifies the representation of the action space and facilitates exploration and optimization by reinforcement learning algorithms.

S4.2. Dynamically adjust the phase setting of the signal light based on real-time traffic data, for example, increase the green light time on lanes with large traffic volume to reduce congestion. In addition, the phase sequence and duration can be adjusted based on factors such as road structure, vehicle type and weather to improve traffic efficiency.

S4.3. Design an adaptive signal light control strategy. For example, sensor control is used at intersections with small traffic volume, and collaborative control is used at intersections with large traffic volume. In addition, the parameters of the control strategy can be dynamically adjusted based on real-time traffic data to adapt to changes in road conditions.

The action space is expressed as:

A＝{action 1, action 2,..., action n}

Among them, each action corresponds to a signal light control strategy or parameter setting. By dynamically adjusting the action space and control strategy, we can achieve intelligent control of traffic lights, thereby improving traffic efficiency and safety.

S5. According to the goal of traffic flow control, design an appropriate reward function for multi-agent reinforcement learning.

Specifically, according to the goals of traffic flow control (such as reducing congestion, reducing emissions, etc.), an appropriate reward function is designed for multi-agent reinforcement learning. The reward function needs to balance various factors to achieve the optimal vehicle-road collaborative control effect. Specifically, the transfer reward function R(s,a,s') is designed as the following form:

R(s,a,s')=w1*T(s,a,s')+w2*D(s,a,s')+w3*S(s,a,s')

in:

s: current status;

a: Action performed by the agent;

s': the new state after executing the action;

w1, w2, w3: weight parameters, used to balance the importance of various indicators;

T(s,a,s'): Traffic efficiency index, such as the average speed or waiting time of vehicles passing through the intersection;

D(s,a,s'): Traffic congestion level indicator, such as the length of vehicle queues at intersections or the number of waiting vehicles;

S(s,a,s'): Traffic safety indicators, such as the probability of traffic accidents or the safe distance between vehicles and pedestrians.

The design of the reward function needs to take into account multiple aspects such as traffic efficiency, congestion, and safety to guide the agent to make decisions that are beneficial to the overall traffic situation. In practical applications, weight parameters and indicator functions can be adjusted according to specific scenarios and needs to achieve better control effects.

S6. Design a communication protocol suitable for vehicle-road collaborative control scenarios.

Specifically, a communication protocol suitable for vehicle-road collaborative control scenarios is designed so that information can be exchanged between agents efficiently and safely. Communication protocols need to consider factors such as low latency, high reliability, and security. We believe that each specific device should adopt different communication methods to ensure that the purpose of the present invention is achieved.

The communication protocol includes communication between vehicles and roadside facilities, communication between signal lights, communication between the central controller and signal lights, and data fusion and processing, specifically:

a) Communication between vehicles and roadside facilities (vehicle-to-road communication): Dedicated short-range communications (DSRC) or vehicle-to-everything (V2X) technology is used to achieve two-way communication between vehicles and roadside facilities (such as traffic lights, sensors, etc.). The vehicle can send its own status information (such as position, speed, driving direction, etc.) to roadside facilities, and at the same time receive instructions from roadside facilities (such as signal light changes, speed limit information, etc.).

b) Communication between signal lights: Information exchange between signal lights can be realized through wireless sensor network (WSN) or cellular network for collaborative control. Through this communication mechanism, adjacent traffic lights can share local traffic information, such as traffic flow, waiting time, etc.

c) Communication between the central controller and the signal lights: The central controller communicates with each signal light through a wired or wireless network. The central controller is responsible for processing information from traffic lights and vehicles, running a multi-agent reinforcement learning algorithm (CVR-MARL), and sending control strategies to the corresponding traffic lights.

d) Data fusion and processing: The multi-modal signal sensing module sends the collected data to the data processing module. The data processing module is responsible for feature fusion of multi-modal data, constructing a state space, and passing it to the multi-agent reinforcement learning algorithm.

S7. Use historical data or simulation environment to train the multi-agent reinforcement learning model and find the optimal strategy.

Specifically, the multi-agent reinforcement learning model is trained using historical data or simulation environment to find the optimal strategy. In practical applications, in addition to deploying the optimal strategy to the signal light control system, we also need to consider dynamic adjustment capabilities to better adapt to changing traffic conditions in practical applications. To achieve this goal, we can adopt the following strategies:

S7.1. Online learning and updating: Through online learning and updating strategies, we can update the reinforcement learning model according to the current traffic conditions in real time. This means that our system can continuously learn and adapt to the actual traffic environment, thereby improving the effectiveness of signal light control.

S7.2. Exploration-exploitation trade-off: During the implementation phase, we need to make a trade-off between exploration and exploitation. By introducing a certain degree of exploration, we can allow the model to continuously try new strategies in practical applications to discover better solutions. However, too much exploration may reduce the stability of the system. Therefore, we need to find a suitable balance between exploration and exploitation.

S7.3. Handling of abnormal situations: In actual applications, some abnormal situations may occur, such as traffic accidents, road closures, etc. In response to these situations, we need to design an exception handling mechanism so that the system can automatically adjust when encountering these problems. For example, when a road closure is detected, the system can automatically adjust the signal light strategy to guide vehicles around.

S7.4. Real-time feedback and adjustment: In order to further improve the dynamic adjustment capability of the system, we can introduce a real-time feedback mechanism into the signal light control system. By collecting real-time traffic data and comparing it with model predictions, the system can adjust itself to better suit actual traffic conditions.

Through the above strategies, our signal light control system will have strong dynamic adjustment capabilities, be able to better adapt to changing traffic conditions in practical applications, and improve the overall signal light control effect.

Although the embodiments of the present invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and purposes of the invention. The scope is defined by the claims and their equivalents.

Claims (7)

1. A signal light collaborative control method based on multi-agent reinforcement learning, characterized in that the method includes: S1. Collect data from various sensors and perform multi-modal definition, and obtain information in real time through data fusion technology; S2. Use the collaborative vehicle-road multi-agent reinforcement learning algorithm to collaboratively control traffic lights and vehicles, and find the optimal strategy through learning to achieve efficient traffic flow control; S3. Preprocess data collected from various sensors, use feature fusion methods to fuse data from different modalities, and build a local state space for each agent; S4. Design an action space for the agent; S5. Design a reward function for multi-agent reinforcement learning based on the goal of traffic flow control; S6. Design communication protocol; S7. Use historical data or simulation environment to train the multi-agent reinforcement learning model and find the optimal strategy; The multi-modal definition in S1 includes visual mode and radar mode; the image data collected by the visual mode through the camera; the distance and speed information collected by the radar module through the radar; the information includes road condition information, vehicle position and velocity; The objective function of the collaborative vehicle-road multi-agent reinforcement learning algorithm is defined as: The reward function is defined as R(s,a), where s represents the state and a represents the action of the agent. By adjusting the action of the agent to maximize the cumulative reward, it is expressed as: J(θ)＝∑_t R(s_t,a_t) Among them, J(θ) represents the objective function; s_t represents the state of the agent at time t; a_t represents the action taken by the agent at time t; Then the loss function L(θ) is expressed as: L(θ)＝0.5*E[(R(s,a)+γ*max_a'Q(s',a';θ')-Q(s,a;θ))^2] Among them, E[·] represents the expected value, θ represents the parameters of the current agent, θ' represents the parameters of the target agent, γ is the discount factor, a' represents the action taken by the agent in state s', Q(s, a; θ) is the action value function, used to estimate the cumulative reward of taking action a in state s; Q(s', a'; θ') represents the network's action value evaluation of action a' in state s' , used to measure the quality of action a' in state s'; The implementation steps of the objective function of the collaborative vehicle-road multi-agent reinforcement learning algorithm include: S2.1, using centralized training and distributed execution strategies, through centralized training in the training phase, collaboration between agents is achieved. In the execution phase, each agent uses a distributed strategy to make decisions based on local states. ; S2.2. Obtain a comprehensive state space containing multi-modal information through the data of various sensors in step S1, so that the agent can more accurately perceive traffic conditions; S2.3. Treat vehicles and traffic lights as different intelligent entities to achieve collaborative control between vehicles and traffic lights to improve traffic smoothness. 2. The signal light collaborative control method based on multi-agent reinforcement learning according to claim 1, characterized in that the data fusion technology is specifically: S1.1. Model the scene as a graph structure; S1.2. Extract features from data of each modality; S1.3. Feature fusion based on graph convolutional neural network; S1.4. Output the reinforcement learning status. 3. The collaborative control method of traffic lights based on multi-agent reinforcement learning according to claim 1, characterized in that the data in step S3 includes vehicle data, road condition data and traffic light data. 4. The traffic light collaborative control method based on multi-agent reinforcement learning and multi-modal signal perception according to claim 3, characterized in that the state space is expressed as follows: s={vehicle data, road condition data, traffic light data}. 5. The signal light collaborative control method based on multi-agent reinforcement learning according to claim 1, characterized in that the step S4 specifically includes: S4.1. Discretize the control strategy of the traffic light into a series of optional actions; S4.2. Dynamically adjust the phase setting of the signal light based on real-time traffic data; S4.3. Design an adaptive traffic light control strategy. 6. The collaborative control method of signal lights based on multi-agent reinforcement learning according to claim 1, characterized in that the communication protocol includes communication between vehicles and roadside facilities, communication between signal lights, communication between the central controller and the signal lights, and Data fusion and processing. 7. The signal light collaborative control method based on multi-agent reinforcement learning according to claim 1, characterized in that the step S7 is specifically: using historical data or simulation environment to train the multi-agent reinforcement learning model to find the optimal The optimal strategy is deployed to the signal light control system.