More Web Proxy on the site http://driver.im/

research-article

Open access

VRGym: a virtual testbed for physical and interactive AI

Authors:

Zhenliang Zhang,

Song-Chun ZhuAuthors Info & Claims

ACM TURC '19: Proceedings of the ACM Turing Celebration Conference - China

Article No.: 100, Pages 1 - 6

https://doi.org/10.1145/3321408.3322633

Published: 17 May 2019 Publication History

Abstract

We propose VRGym, a virtual reality (VR) testbed for realistic human-robot interaction. Different from existing toolkits and VR environments, the VRGym emphasizes on building and training both physical and interactive agents for robotics, machine learning, and cognitive science. VRGym leverages mechanisms that can generate diverse 3D scenes with high realism through physics-based simulation. We demonstrate that VRGym is able to (i) collect human interactions and fine manipulations, (ii) accommodate various robots with a ROS bridge, (iii) support experiments for human-robot interaction, and (iv) provide toolkits for training the state-of-the-art machine learning algorithms. We hope VRGym can help to advance general-purpose robotics and machine learning agents, as well as assisting human studies in the field of cognitive science.¹

References

[1]

Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. 2009. A survey of robot learning from demonstration. Robotics and autonomous systems 57, 5 (2009), 469--483.

Digital Library

[2]

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. Openai gym.

[3]

Noam Brown and Tuomas Sandholm. 2018. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science 359, 6374 (2018), 418--424.

[4]

Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. 2015. Benchmarking in Manipulation Research. IEEE Robotics & Automation Magazine 1070, 9932/15 (2015), 36.

[5]

Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3D: Learning from RGB-D Data in Indoor Environments. In International Conference on 3D Vision (3DV).

[6]

Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. 2015. Shapenet: An information-rich 3d model repository.

[7]

Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. 2016. Benchmarking deep reinforcement learning for continuous control. In ICML.

Digital Library

[8]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In CVPR.

[9]

Geoffrey E Hinton and Ruslan R Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. science 313, 5786 (2006), 504--507.

[10]

Chenfanfu Jiang, Siyuan Qi, Yixin Zhu, Siyuan Huang, Jenny Lin, Lap-Fai Yu, Demetri Terzopoulos, and Song-Chun Zhu. 2018. Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars. IJCV 126, 9 (2018), 920--941.

Digital Library

[11]

Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017. AI2-THOR: An interactive 3d environment for visual AI.

[12]

Shane Legg and Marcus Hutter. 2007. Universal intelligence: A definition of machine intelligence. Minds and Machines 17, 4 (2007), 391--444.

[13]

Ian Lenz, Honglak Lee, and Ashutosh Saxena. 2015. Deep learning for detecting robotic grasps. IJRR 34, 4--5 (2015), 705--724.

Digital Library

[14]

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2016. Continuous control with deep reinforcement learning. In ICLR.

[15]

Hangxin Liu, Zhenliang Zhang, Xie Xu, Yixin Zhu, Yue Liu, Yongtian Wang, and Song-Chun Zhu. 2019. High-Fidelity Grasping in Virtual Reality using a Glove-based System. In ICRA.

[16]

Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, and Ken Goldberg. 2017. Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics. In RSS.

[17]

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In ICML.

Digital Library

[18]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529.

[19]

Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisỳ, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. 2017. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science 356, 6337 (2017), 508--513.

[20]

Isaac Newton and John Colson. 1736. The Method of Fluxions and Infinite Series; with Its Application to the Geometry of Curve-lines. Henry Woodfall.

[21]

Siyuan Qi, Siyuan Huang, Ping Wei, and Song-Chun Zhu. 2017. Predicting Human Activities Using Stochastic Grammar. In ICCV.

[22]

Siyuan Qi, Yixin Zhu, Siyuan Huang, Chenfanfu Jiang, and Song-Chun Zhu. 2018. Human-centric Indoor Scene Synthesis Using Stochastic Grammar. In CVPR.

[23]

Deepak Ramachandran and Eyal Amir. 2007. Bayesian inverse reinforcement learning. Urbana 51, 61801 (2007), 1--4.

[24]

Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. 2018. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and service robotics. Springer, 621--635.

[25]

Tianmin Shu, Xiaofeng Gao, Michael S Ryoo, and Song-Chun Zhu. 2017. Learning Social Affordance Grammar from Videos: Transferring Human Interactions to Human-Robot Interactions. In ICRA.

[26]

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. nature 529, 7587 (2016), 484.

[27]

Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. 2017. Semantic Scene Completion From a Single Depth Image. In CVPR.

[28]

Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012. Mujoco: A physics engine for modelbased control. In IROS.

[29]

Michal Valko, Mohammad Ghavamzadeh, and Alessandro Lazaric. 2012. Semi-Supervised Apprenticeship Learning. In EWRL.

[30]

Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. 2016. Dueling Network Architectures for Deep Reinforcement Learning. In ICML.

Digital Library

[31]

Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. 2018. Gibson Env: Real-World Perception for Embodied Agents. In CVPR.

[32]

Lap-Fai Yu, Sai-Kit Yeung, Chi-Keung Tang, Demetri Terzopoulos, Tony F Chan, and Stanley J Osher. 2011. Make it home: automatic optimization of furniture arrangement. TOG 30, 4 (2011), 86.

Digital Library

[33]

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. 2008. Maximum Entropy Inverse Reinforcement Learning. In AAAI.

Digital Library

Cited By

Zhang ZZhang ZJiao ZSu YLiu HWang WZhu S(2024)On the Emergence of Symmetrical Reality2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00084(639-649)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VR58804.2024.00084
Raees MMeijerink ILykourentzou IKhan VPapangelis K(2024)From explainable to interactive AI: A literature review on current trends in human-AI interactionInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103301189(103301)Online publication date: Sep-2024
https://doi.org/10.1016/j.ijhcs.2024.103301
Peng YHan JZhang ZFan LLiu TQi SFeng XMa YWang YZhu S(2024)The Tong Test: Evaluating Artificial General Intelligence Through Dynamic Embodied Physical and Social InteractionsEngineering10.1016/j.eng.2023.07.00634(12-22)Online publication date: Mar-2024
https://doi.org/10.1016/j.eng.2023.07.006
Show More Cited By

Index Terms

VRGym: a virtual testbed for physical and interactive AI

Recommendations

Analysis of MR–VR tele-operation methods for legged-manipulator robots
Abstract
The development of immersive technologies in recent years has facilitated the control and execution of tasks at a high level of complexity in robotic systems. On the other hand, exploration and manipulation tasks in unknown environments have been ...
Haptics-based virtual reality periodontal training simulator

This paper focuses upon the research and development of a prototype dental simulator for training of periodontal procedures. By the use of virtual reality and haptics technology, the periodontal simulator allows trainees to learn performing diagnosis ...
Design and Implementation of Mine Virtual Simulation Training System
ICECC '12: Proceedings of the 2012 International Conference on Electronics, Communications and Control

This paper developed a kind of virtual training system for coal mine workers and technicians by using virtual reality (Virtual reality, VR) technology. The basis of this system is virtual reality and 3D game engine, in the virtual environment, trainees ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ACM TURC '19: Proceedings of the ACM Turing Celebration Conference - China

May 2019

963 pages

ISBN:9781450371582

DOI:10.1145/3321408

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 May 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

ACM TURC 2019

ACM TURC 2019: ACM Turing Celebration Conference - China

May 17 - 19, 2019

Chengdu, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
607
Total Downloads

Downloads (Last 12 months)176
Downloads (Last 6 weeks)17

Reflects downloads up to 14 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang ZZhang ZJiao ZSu YLiu HWang WZhu S(2024)On the Emergence of Symmetrical Reality2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00084(639-649)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VR58804.2024.00084
Raees MMeijerink ILykourentzou IKhan VPapangelis K(2024)From explainable to interactive AI: A literature review on current trends in human-AI interactionInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103301189(103301)Online publication date: Sep-2024
https://doi.org/10.1016/j.ijhcs.2024.103301
Peng YHan JZhang ZFan LLiu TQi SFeng XMa YWang YZhu S(2024)The Tong Test: Evaluating Artificial General Intelligence Through Dynamic Embodied Physical and Social InteractionsEngineering10.1016/j.eng.2023.07.00634(12-22)Online publication date: Mar-2024
https://doi.org/10.1016/j.eng.2023.07.006
Liu HZhang ZJiao ZZhang ZLi MJiang CZhu YZhu S(2024)A Reconfigurable Data Glove for Reconstructing Physical and Virtual GraspsEngineering10.1016/j.eng.2023.01.00932(202-216)Online publication date: Jan-2024
https://doi.org/10.1016/j.eng.2023.01.009
Jiang HWeng DDongye XLuo LZhang Z(2023)Commonsense Knowledge-Driven Joint Reasoning Approach for Object Retrieval in Virtual RealityACM Transactions on Graphics10.1145/361832042:6(1-18)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618320
Zhang ZZhang LWang ZJiao ZHan MZhu YZhu SLiu H(2023)Part-level Scene Reconstruction Affords Robot Interaction2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS55552.2023.10342208(11178-11185)Online publication date: 1-Oct-2023
https://doi.org/10.1109/IROS55552.2023.10342208
Sharma VWalia N(2023)Human Cognition and Artificial Intelligence: A Survey2023 8th International Conference on Communication and Electronics Systems (ICCES)10.1109/ICCES57224.2023.10192616(871-876)Online publication date: 1-Jun-2023
https://doi.org/10.1109/ICCES57224.2023.10192616
Huang SWang ZLi PJia BLiu TZhu YLiang WZhu S(2023)Diffusion-based Generation, Optimization, and Planning in 3D Scenes2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01607(16750-16761)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.01607
Han MZhang ZJiao ZXie XZhu YZhu SLiu H(2022)Scene Reconstruction with Functional Objects for Robot AutonomyInternational Journal of Computer Vision10.1007/s11263-022-01670-0130:12(2940-2961)Online publication date: 20-Sep-2022
https://doi.org/10.1007/s11263-022-01670-0
Kaur DSingh NBanerjee B(2022)A review of platforms for simulating embodied agents in 3D virtual environmentsArtificial Intelligence Review10.1007/s10462-022-10253-x56:4(3711-3753)Online publication date: 10-Sep-2022
https://dl.acm.org/doi/10.1007/s10462-022-10253-x
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents