Optimizing communication in deep reinforcement learning with XingTian

Deep Reinforcement Learning (DRL) achieves great success in various domains. Communication in today's DRL algorithms takes non-negligible time compared to the computation. However, prior DRL frameworks usually focus on computation management while paying little attention to communication optimization, and fail to utilize the opportunity of the communication-computation overlap that hides the communication from the critical path of DRL algorithms. Consequently, communication can take more time than the computation in prior DRL frameworks. In this paper, we present XingTian, a novel DRL framework that co-designs the management of communication and computation in DRL algorithms. XingTian organizes the computation in DRL algorithms in a decentralized way and provides an asynchronous communication channel. XingTian makes the communication execute asynchronously and aggressively and takes advantage of the communication-computation overlapping opportunity from DRL algorithms. Experimental results show that XingTian improves data transmission efficiency and can transmit at least twice as much data per second as the state-of-the-art DRL framework RLLib. DRL algorithms based on XingTian achieve up to 70.71% more throughput than RLLib-based ones with better or similar convergent performance. XingTian maintains high communication efficiency under different scale deployments and the XingTian-based DRL algorithm achieves 91.12% higher throughput than the RLLib-based one when deployed in four machines. XingTian is open-sourced and publicly available at https://github.com/huawei-noah/xingtian.