Abstract
Traffic signal control plays an important role in reducing urban traffic congestion. In complex traffic scenarios, coordinating phase signal control between intersections is a significant challenge. Reinforcement learning is widely used in the field of intelligent traffic signal control because it is good at dealing with sequence decision problems. The current reinforcement learning based approach makes phase decisions through coordinated cooperation. However, existing methods have difficulty with information exchange, because they lack semantic interpretation and explicit quantification of collaborative impact, which results in inefficient or conflicting phase coordination between intersections. Moreover, during the early exploration stage of reinforcement learning training, the phase output of the decision network is unreliable, making it difficult for the model to utilize decision information for self-supervised training. To address these issues, this paper proposes a self-supervised, explicit coordination based multi-agent reinforcement learning approach. Additionally, a phase boosting learning from demonstration method is introduced in the early training stages. Extensive experimental results demonstrate that this method can enhance collaboration among agents, outperforming baseline methods across multiple real-world traffic datasets, while also improving training stability and convergence speed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Reed, T.: INRIX global traffic scorecard (2019)
Robertson, D.I., Bretherton, R.D.: Optimizing networks of traffic signals in real time-the SCOOT method. IEEE Trans. Veh. Technol. 40, 11–15 (1991)
Lowrie, P.R.: Scats, sydney co-ordinated adaptive traffic system: a traffic responsive method of controlling urban traffic (1990)
Ye, B., Weimin, W., Weijie, M.: A two-way arterial signal coordination method with queueing process considered. IEEE Trans. Intell. Transp. Syst. 16(6), 3440–3452 (2015)
Wiering, M.A., Martijn V.O.: Reinforcement Learning. Adaptation, learning, and optimization, p. 729 (2012). https://doi.org/10.1007/978-3-642-27645-3
Buşoniu, L., Babuška, R., De Schutter, B.: Multi-agent reinforcement learning: an overview. In: Srinivasan, D., Jain, L.C. (eds.) Innovations in Multi-Agent Systems and Applications - 1, pp. 183–221. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14435-6_7
El-Tantawy, S., Baher, A., Hossam, A.: Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC): methodology and large-scale application on downtown Toronto. IEEE Trans. Intell. Transp. Syst. 14(3), 1140–1150 (2013)
Chu, T., Wang, J., Codecà, L., et al.: Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Trans. Intell. Transp. Syst. 21(3), 1086–1095 (2019)
Liu, J., Zhang, H., Fu, Z., et al.: Learning scalable multi-agent coordination by spatial differentiation for traffic signal control. Eng. Appl. Artif. Intell. Appl. Artif. Intell. 100, 104165 (2021)
Zhao, W., et al.: IPDALight: Intensity-and phase duration-aware traffic signal control based on reinforcement learning. J. Syst. Architect. 123, 102374 (2022)
Wei, H., et al.: CoLight: learning network-level cooperation for traffic signal control. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management (2019)
Zhu, L., et al.: Meta variationally intrinsic motivated reinforcement learning for decentralized traffic signal control. arXiv preprint arXiv:2101.00746 (2021)
Varaiya, P.: Max pressure control of a network of signalized intersections. Trans. Res. C Emerg. Technol. 36, 177–195 (2013)
Cools, S.-B., Gershenson, C., D’Hooghe, B.: Self-organizing traffic lights: a realistic simulation. In: Prokopenko, M. (ed.) Advances in Applied Self-Organizing Systems, pp. 45–55. Springer, London (2013). https://doi.org/10.1007/978-1-4471-5113-5_3
Wei, H., et al.: IntelliLight: a reinforcement learning approach for intelligent traffic light control. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (2018)
Wei, H., et al.: PressLight: learning max pressure control to coordinate traffic signals in arterial network. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (2019)
Zhang, L., et al.: Expression might be enough: representing pressure and demand for reinforcement learning based traffic signal control. In: International Conference on Machine Learning. PMLR (2022)
Ma, J., Feng, W.: Feudal multi-agent deep reinforcement learning for traffic signal control. In: Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems (2020)
Zhang, Y., Mehul, D., Guillaume, S.: Multi-agent traffic signal control via distributed RL with spatial and temporal feature extraction. In: Melo, F.S., Fang, F. (eds.) Autonomous Agents and Multiagent Systems. Best and Visionary Papers: AAMAS 2022 Workshops, Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20179-0_7
Zheng, G., et al.: Diagnosing reinforcement learning for traffic signal control. arXiv preprint arXiv:1905.04716 (2019)
Oroojlooy, A., et al.: AttendLight: universal attention-based reinforcement learning model for traffic signal control. Adv. Neural Inform. Process. Syst. 33, 4079–4090 (2020)
Acknowledgement
This research was supported by the National Natural Science Foundation of China (62076060, 62072099, 61932007, 61806053).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, Y., Che, Q., Zhou, Y., Wang, W., Jiang, Y. (2024). Explicit Coordination Based Multi-agent Reinforcement Learning for Intelligent Traffic Signal Control. In: Sun, Y., Lu, T., Wang, T., Fan, H., Liu, D., Du, B. (eds) Computer Supported Cooperative Work and Social Computing. ChineseCSCW 2023. Communications in Computer and Information Science, vol 2013. Springer, Singapore. https://doi.org/10.1007/978-981-99-9640-7_1
Download citation
DOI: https://doi.org/10.1007/978-981-99-9640-7_1
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-9639-1
Online ISBN: 978-981-99-9640-7
eBook Packages: Computer ScienceComputer Science (R0)