abstract

Augmented Reality with Multi-view Merging for Robot Teleoperation

Authors:

Bidan Huang,

Nicholas Gerard Timmons,

Qiang LiAuthors Info & Claims

HRI '20: Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction

Pages 260 - 262

https://doi.org/10.1145/3371382.3378336

Published: 01 April 2020 Publication History

Get Access

Abstract

This paper proposes a user-friendly teleoperation interface for manipulation. We provide the user with a view of the scene augmented with depth information from local cameras to provide visibility in occluded areas during manipulation tasks. This gives an improved sense of the 3D environment which results in better task performance. Further, we monitor the pose of the robot's end effector in real-time so that we are able to superimpose a virtual representation into the scene when parts are occluded. The integration of these features enables the user to perform difficult manipulation tasks when the action environment is normally occluded in the main camera view.

We performed preliminary studies with this new setup and users provided positive feedback regarding the proposed augmented reality (AR) system.

References

[1]

Gloria L Calhoun, Mark H Draper, Jeremy T Nelson, and Heath A Ruff. 2006. Advanced display concepts for UAV sensor operations: Landmark cues and picture-in-picture. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 50. SAGE Publications Sage CA: Los Angeles, CA, 121--125.

Crossref

Google Scholar

[2]

Soichiro Iwataki, Hiromitsu Fujii, Alessandro Moro, Atsushi Yamashita, Hajime Asama, and Hiroshi Yoshinada. 2015. Visualization of the surrounding environment and operational part in a 3DCG model for the teleoperation of construction machines. In 2015 IEEE/SICE International Symposium on System Integration (SII). IEEE, 81--87.

Crossref

Google Scholar

[3]

Leonid Keselman, John Iselin Woodfill, Anders Grunnet-Jepsen, and Achintya Bhowmik. 2017. Intel realsense stereoscopic depth cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1--10.

Crossref

Google Scholar

[4]

Fumio Okura, Yuko Ueda, Tomokazu Sato, and Naokazu Yokoya. 2013. Teleoperation of mobile robots by generating augmented free-viewpoint images. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 665--671.

Crossref

Google Scholar

[5]

Luis Enrique Ortiz, Elizabeth V Cabrera, and Luiz M Goncc alves. 2018. Depth data error modeling of the ZED 3D vision sensor from stereolabs. ELCVIA: electronic letters on computer vision and image analysis, Vol. 17, 1 (2018), 0001--15.

Google Scholar

[6]

A Pitman, Curtis M Humphrey, and Julie A Adams. 2007. A picture-in-picture interface for a multiple robot system.

Google Scholar

[7]

Long Qian, Anton Deguet, Zerui Wang, Yun-Hui Liu, and Peter Kazanzides. 2019. Augmented Reality Assisted Instrument Insertion and Tool Manipulation for the First Assistant in Robotic Surgery. In 2019 International Conference on Robotics and Automation (ICRA). IEEE, 5173--5179.

Google Scholar

[8]

Thomas B Sheridan. 1992. Musings on telepresence and virtual presence. Presence: Teleoperators & Virtual Environments, Vol. 1, 1 (1992), 120--126.

Crossref

Google Scholar

[9]

AWW Yew, SK Ong, and AYC Nee. 2017. Immersive augmented reality environment for the teleoperation of maintenance robots. Procedia CIRP, Vol. 61 (2017), 305--310.

Crossref

Google Scholar

Cited By

View all

Kerdvibulvech CChang C(2023)A New Study of Integration Between Social Robotic Systems and the Metaverse for Dealing with Healthcare in the Post-COVID-19 SituationsSocial Robotics10.1007/978-3-031-24670-8_35(392-401)Online publication date: 2-Feb-2023
https://doi.org/10.1007/978-3-031-24670-8_35
Suzuki RKarim AXia THedayati HMarquardt N(2022)Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic InterfacesProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517719(1-33)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517719
Wei DHuang BLi Q(2021)Multi-View Merging for Robot Teleoperation With Virtual RealityIEEE Robotics and Automation Letters10.1109/LRA.2021.31093486:4(8537-8544)Online publication date: Oct-2021
https://doi.org/10.1109/LRA.2021.3109348

Index Terms

Augmented Reality with Multi-view Merging for Robot Teleoperation
1. Hardware
  1. Emerging technologies
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques
  2. Interaction design
    1. Empirical studies in interaction design

Recommendations

Improving Collocated Robot Teleoperation with Augmented Reality
HRI '18: Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction

Robot teleoperation can be a challenging task, often requiring a great deal of user training and expertise, especially for platforms with high degrees-of-freedom (e.g., industrial manipulators and aerial robots). Users often struggle to synthesize ...
Haptics in Augmented Reality
ICMCS '99: Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2

An augmented reality system merges synthetic sensory information into a user's perception of a three-dimensional environment. An important performance goal for an augmented reality system is that the user perceives a single seamless environment. In most ...
Virtual and Augmented Reality with Head-Tracking for Efficient Teleoperation of Groups of Robots
CW '08: Proceedings of the 2008 International Conference on Cyberworlds

This paper deals with the usage of Virtual and Augmented Reality techniques in the field of teleoperation. The global context is a project for collaborative teleoperation of real robots groups. The aim is to enable to a group of teleoprators to control ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

HRI '20: Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction

March 2020

702 pages

ISBN:9781450370578

DOI:10.1145/3371382

General Chairs:
Tony Belpaeme
Ghent University, Belgium
,
James Young
University of Manitoba, Canada
,
Program Chairs:
Hatice Gunes
University of Cambridge, UK
,
Laurel Riek
UC San Diego, USA

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 April 2020

Check for updates

Author Tags

Qualifiers

Abstract

Conference

HRI '20

Sponsor:

HRI '20: ACM/IEEE International Conference on Human-Robot Interaction

March 23 - 26, 2020

Cambridge, United Kingdom

Acceptance Rates

Overall Acceptance Rate 192 of 519 submissions, 37%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
449
Total Downloads

Downloads (Last 12 months)58
Downloads (Last 6 weeks)10

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Kerdvibulvech CChang C(2023)A New Study of Integration Between Social Robotic Systems and the Metaverse for Dealing with Healthcare in the Post-COVID-19 SituationsSocial Robotics10.1007/978-3-031-24670-8_35(392-401)Online publication date: 2-Feb-2023
https://doi.org/10.1007/978-3-031-24670-8_35
Suzuki RKarim AXia THedayati HMarquardt N(2022)Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic InterfacesProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517719(1-33)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517719
Wei DHuang BLi Q(2021)Multi-View Merging for Robot Teleoperation With Virtual RealityIEEE Robotics and Automation Letters10.1109/LRA.2021.31093486:4(8537-8544)Online publication date: Oct-2021
https://doi.org/10.1109/LRA.2021.3109348

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Improving Collocated Robot Teleoperation with Augmented Reality

Haptics in Augmented Reality

Virtual and Augmented Reality with Head-Tracking for Efficient Teleoperation of Groups of Robots