A Domain Data Pattern Randomization based Deep Reinforcement Learning method for Sim-to-Real transfer
Authors: 
Peng Gong, Dianxi Shi, Chao Xue, Xucan ChenAuthors Info & Claims
Pages 1 - 7
Abstract
Transferring reinforcement learning policies trained in a physical simulator to the real world is a highly challenging problem, because the gap between the simulation and reality, usually causes the transferred model to perform poorly in the real world. Many algorithms including domain randomization, have been proposed to try to bridge the gap between simulation and reality. However, most of them are to change the value of the corresponding data by superimposing gaussian noise on robot dynamics parameters or environmental data. Such policies often fail to solve the problem of long-term/intermittent missing data patterns caused by sensor failures in the actual operation of the robot. Faced with this problem, we proposed a memory-enhanced domain data pattern randomization method. This method achieves data enhancement by randomizing the distribution pattern of data connection, at the same time, the memory mechanism based on recurrent neural network is introduced into the decision model, to alleviate the jitter of environmental distribution caused by data pattern changes, so as to improve the decision-making ability of the robot in some observable scenes triggered by the change of data pattern.
References
[1]
Adam David Allevato, Elaine Schaertlshort, Mitch Pryor, and Andrea L. Thomaz. 2020. Iterative residual tuning for system identification and sim-to-real robot learning. Autonomous RobotsJune(2020).
[2]
OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, 2020. Learning dexterous in-hand manipulation. The International Journal of Robotics Research 39, 1 (2020), 3–20.
[3]
Karol Arndt, Murtaza Hazara, Ali Ghadirzadeh, and Ville Kyrki. 2020. Meta reinforcement learning for sim-to-real domain adaptation. In 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2725–2731.
[4]
Babak Badnava and Nasser Mozayani. 2019. A new potential-based reward shaping for reinforcement learning agent. arXiv preprint arXiv:1902.06239(2019).
[5]
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, and Dieter Fox. 2019. Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience. In 2019 International Conference on Robotics and Automation (ICRA).
[6]
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078(2014).
[7]
Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, and Wojciech Zaremba. 2016. Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model. arXiv preprint arXiv:1610.03518(2016).
[8]
Nathan Koenig and Andrew Howard. 2004. Design and use paradigms for gazebo, an open-source multi-robot simulator. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566), Vol. 3. IEEE, 2149–2154.
[9]
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971(2015).
[10]
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. 2020. Active domain randomization. (2020), 1162–1176.
[11]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602(2013).
[12]
Igor Mordatch and Pieter Abbeel. 2018. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
[13]
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. 2018. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 1–8.
[14]
Kormushev Petar, Calinon Sylvain, and Caldwell Darwin. 2013. Reinforcement Learning in Robotics: Applications and Real-World Challenges. Robotics 2, 3 (2013), 122–148.
[15]
Riccardo Polvara, Massimiliano Patacchiola, Sanjay Sharma, Jian Wan, Andrew Manning, Robert Sutton, and Angelo Cangelosi. 2018. Toward end-to-end control for UAV autonomous landing via deep reinforcement learning. In 2018 International Conference on Unmanned Aircraft Systems (ICUAS). IEEE, 115–123.
[16]
Aayush Prakash, Shaad Boochoon, Mark Brophy, David Acuna, Eric Cameracci, Gavriel State, Omer Shapira, and Stan Birchfield. 2019. Structured domain randomization: Bridging the reality gap by context-aware synthetic data. In 2019 International Conference on Robotics and Automation (ICRA). IEEE, 7249–7255.
[17]
Andrei A Rusu, Matej Večerík, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. 2017. Sim-to-real robot learning from pixels with progressive nets. (2017), 262–270.
[18]
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. 2014. Deterministic policy gradient algorithms.
[19]
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. 2017. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 23–30.
[20]
Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. 2018. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 969–977.
[21]
Eugene Valassakis, Zihan Ding, and Edward Johns. 2020. Crossing The Gap: A Deep Dive into Zero-Shot Sim-to-Real Transfer for Dynamics. arXiv preprint arXiv:2008.06686(2020).
[22]
Quan Vuong, Sharad Vikram, Hao Su, Sicun Gao, and Henrik I Christensen. 2019. How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?arXiv preprint arXiv:1903.11774(2019).
[23]
Wenhao Yu, C Karen Liu, and Greg Turk. 2018. Policy transfer with strategy optimization. arXiv preprint arXiv:1810.05751(2018).
[24]
Wenshuai Zhao, Jorge Pena Queralta, and Tomi Westerlund. 2020. Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey. arXiv preprint arXiv:2009.13303(2020).
Index Terms
- A Domain Data Pattern Randomization based Deep Reinforcement Learning method for Sim-to-Real transfer
Index terms have been assigned to the content through auto-classification.
Recommendations
DR2L: Surfacing Corner Cases to Robustify Autonomous Driving via Domain Randomization Reinforcement Learning
CSAE '21: Proceedings of the 5th International Conference on Computer Science and Application EngineeringHow to explore corner cases as efficiently and thoroughly as possible has long been one of the top concerns in the context of deep reinforcement learning (DeepRL) autonomous driving. Training with simulated data is less costly and dangerous than ...
Sim-to-Real Deep Reinforcement Learning with Manipulators for Pick-and-Place
Towards Autonomous Robotic SystemsAbstractWhen transferring a Deep Reinforcement Learning model from simulation to the real world, the performance could be unsatisfactory since the simulation cannot imitate the real world well in many circumstances. This results in a long period of fine-...
Deep Reinforcement Learning: From Q-Learning to Deep Q-Learning
Neural Information ProcessingAbstractAs the two hottest branches of machine learning, deep learning and reinforcement learning both play a vital role in the field of artificial intelligence. Combining deep learning with reinforcement learning, deep reinforcement learning is a method ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2021
246 pages
ISBN:9781450388634
DOI:10.1145/3461353
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 September 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- the National Key Research and Development Program of China
Conference
ICIAI 2021
ICIAI 2021: 2021 the 5th International Conference on Innovation in Artificial Intelligence
March 5 - 8, 2021
Xia men, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 92Total Downloads
- Downloads (Last 12 months)13
- Downloads (Last 6 weeks)1
Reflects downloads up to 23 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format