research-article

A sim2real framework enabling decentralized agents to execute MADDPG tasks

Authors:

Young-Ho Suh,

Sung-Pil Woo,

Hyunhak Kim,

Dong-Hwan ParkAuthors Info & Claims

DIDL '19: Proceedings of the Workshop on Distributed Infrastructures for Deep Learning

Pages 1 - 6

https://doi.org/10.1145/3366622.3368146

Published: 09 December 2019 Publication History

Get Access

Abstract

Multi-agent RL is a process of training the agents to collaborate with others. We argue that an additional 'reality gap' in the system aspects occurs when applying sim2real to the multi-agent RL, especially when performing the 'transferred' collaborative task in the real-world environment. In this paper, we propose an ADO framework enabling decentralized agents to participate in performing collaborative tasks without suffering from the reality gap. Our contribution is threefold. First, we clearly identify and summarize the reality gaps in the context of the sim2real of multi-agent RL. Second, we propose a new system model to deal with system issues derived from when executing collaborative tasks. Third, we design and implement a software framework to support system issues required in developing and executing collaborative tasks in the real world.

References

[1]

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. (2016), 1--4. arXiv:1606.01540

Google Scholar

[2]

Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. 2018. Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience. (2018). arXiv:1810.05687

Google Scholar

[3]

Carl Hewitt. 2011. Actor Model of Computation. Inconsistency Robustness 2011 (2011), 1--25. arXiv:1008.1459

Google Scholar

[4]

Zhang Wei Hong, Yu Ming Chen, Hsuan Kung Yang, Shih Yang Su, Tzu Yun Shann, Yi Hsiang Chang, Brian Hsi Lin Ho, Chih Chieh Tu, Tsu Ching Hsiao, Hsin Wei Hsiao, Sih Pin Lai, Yueh Chuan Chang, and Chun Yi Lee. 2018. Virtual-to-real: Learning to control in visual semantic segmentation. IJCAI International Joint Conference on Artificial Intelligence 2018-July (2018), 4912--4920. arXiv:arXiv:1802.00285v4

Crossref

Google Scholar

[5]

Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. 2018. Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks. (2018). arXiv:1812.07252

Google Scholar

[6]

Svetoslav Kolev and Emanuel Todorov. 2015. Physically consistent state estimation and system identification for contacts. IEEE-RAS International Conference on Humanoid Robots 2015-December (2015), 1036--1043.

Crossref

Google Scholar

[7]

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. (2015). arXiv:1509.02971

Google Scholar

[8]

Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems 2017-December (2017), 6380--6391. arXiv:arXiv:1706.02275v3

Google Scholar

[9]

OpenAI. 2017. Multi Agent Particle Environment. https://github.com/openai/multiagent-particle-envs

Google Scholar

[10]

Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. 2018. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. Proceedings - IEEE International Conference on Robotics and Automation (2018), 3803--3810.

Crossref

Google Scholar

[11]

Marc Shapiro, Nuno Pregui, Carlos Baquero, Marek Zawirski, Carlos Baquero, Marek Zawirski, and Conflict-free Replicated Data. 2011. Conflict-free Replicated Data Types To cite this version:. (2011).

Google Scholar

[12]

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. 2017. Domain randomization for transferring deep neural networks from simulation to the real world. IEEE International Conference on Intelligent Robots and Systems 2017-September (2017), 23--30. arXiv:arXiv:1703.06907v1

Crossref

Google Scholar

Cited By

View all

Schaefer KBrewer RWickwire JScalise RKessens C(2024)Modeling and Simulation Technologies for Effective Multi-agent ResearchVirtual, Augmented and Mixed Reality10.1007/978-3-031-61044-8_7(86-104)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-61044-8_7
Candela EParada LMarques LGeorgescu TDemiris YAngeloudis P(2022)Transferring Multi-Agent Reinforcement Learning Policies for Autonomous Driving using Sim-to-Real2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS47612.2022.9981319(8814-8820)Online publication date: 23-Oct-2022
https://doi.org/10.1109/IROS47612.2022.9981319

Index Terms

A sim2real framework enabling decentralized agents to execute MADDPG tasks
1. Software and its engineering
  1. Software notations and tools
    1. Development frameworks and environments
      1. Application specific development environments
      2. Object oriented frameworks

Recommendations

Multi-agent Reinforcement Learning for Decentralized Stable Matching
Algorithmic Decision Theory
Abstract
In the real world, people/entities usually find matches independently and autonomously, such as finding jobs, partners, roommates, etc. It is possible that this search for matches starts with no initial knowledge of the environment. We propose the ...
Training Cooperative Agents for Multi-Agent Reinforcement Learning
AAMAS '19: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems

Deep Learning and back-propagation has been successfully used to perform centralized training with communication protocols among multiple agents in a cooperative environment. In this paper we present techniques for centralized training of Multi-Agent (...
Shifting Reward Assignment for Learning Coordinated Behavior in Time-Limited Ordered Tasks
Advances in Practical Applications of Agents, Multi-Agent Systems, and Complex Systems Simulation. The PAAMS Collection
Abstract
We propose a variable reward scheme in decentralized multi-agent deep reinforcement learning for a sequential task consisting of a number of subtasks which can be completed when all subtasks are executed in a certain order before a deadline by ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

DIDL '19: Proceedings of the Workshop on Distributed Infrastructures for Deep Learning

December 2019

17 pages

ISBN:9781450370370

DOI:10.1145/3366622

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 December 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

Middleware '19

Sponsor:

Middleware '19: 20th International Middleware Conference

December 9 - 13, 2019

CA, Davis, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
287
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Schaefer KBrewer RWickwire JScalise RKessens C(2024)Modeling and Simulation Technologies for Effective Multi-agent ResearchVirtual, Augmented and Mixed Reality10.1007/978-3-031-61044-8_7(86-104)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-61044-8_7
Candela EParada LMarques LGeorgescu TDemiris YAngeloudis P(2022)Transferring Multi-Agent Reinforcement Learning Policies for Autonomous Driving using Sim-to-Real2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS47612.2022.9981319(8814-8820)Online publication date: 23-Oct-2022
https://doi.org/10.1109/IROS47612.2022.9981319

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Multi-agent Reinforcement Learning for Decentralized Stable Matching

Training Cooperative Agents for Multi-Agent Reinforcement Learning

Shifting Reward Assignment for Learning Coordinated Behavior in Time-Limited Ordered Tasks