[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3611659.3615685acmconferencesArticle/Chapter ViewAbstractPublication PagesvrstConference Proceedingsconference-collections
research-article
Public Access

Evaluating Augmented Reality Communication: How Can We Teach Procedural Skill in AR?

Published: 09 October 2023 Publication History

Abstract

Augmented reality (AR) has great potential for use in healthcare applications, especially remote medical training and supervision. In this paper, we analyze the usage of an AR communication system to teach a medical procedure, the placement of a central venous catheter (CVC) under ultrasound guidance. We examine various AR communication and collaboration components, including gestural communication, volumetric information, annotations, augmented objects, and augmented screens. We compare how teaching in AR differs from teaching through videoconferencing-based communication. Our results include a detailed medical training steps analysis in which we compare how verbal and visual communication differs between video and AR training. We identify procedural steps in which medical experts give visual instructions utilizing AR components. We examine the change in AR usage and interaction over time and recognize patterns between users. Moreover, AR design recommendations are given based on post-training interviews.

Supplemental Material

MP4 File
Augmented reality collaboration system video

References

[1]
Weronika Celniak and Piotr Augustyniak. 2022. Eye-Tracking as a Component of Multimodal Emotion Recognition Systems. In Information Technology in Biomedicine, Ewa Pietka, Pawel Badura, Jacek Kawa, and Wojciech Wieclawek (Eds.). Springer International Publishing, Cham, 66–75.
[2]
Dragoş Datcu, Stephan Lukosch, and Frances Brazier. 2015. On the Usability and Effectiveness of Different Interaction Types in Augmented Reality. International Journal of Human–Computer Interaction 31, 3 (2015), 193–209.
[3]
Ryan Anthony de Belen, Huyen Nguyen, Daniel Filonik, Dennis Favero, and Tomasz Bednarz. 2019. A systematic review of the current state of collaborative mixed reality technologies: 2013–2018. AIMS Electronics and Electrical Engineering 3 (05 2019), 181–223. https://doi.org/10.3934/ElectrEng.2019.2.181
[4]
Andreas Dünser, Raphaël Grasset, Hartmut Seichter, and Mark Billinghurst. 2007. Applying HCI principles to AR systems design. (2007).
[5]
Barrett Ens, Joel Lanir, Anthony Tang, Scott Bateman, Gun Lee, Thammathip Piumsomboon, and Mark Billinghurst. 2019. Revisiting collaboration through mixed reality: The evolution of groupware. International Journal of Human-Computer Studies 131 (2019), 81–98. https://doi.org/10.1016/j.ijhcs.2019.05.011 50 years of the International Journal of Human-Computer Studies. Reflections on the past, present and future of human-centred technologies.
[6]
Spencer Fowers, Thiago V. Spina, Andrea Britto Mattos Lima, Ben Cutler, David Tittsworth, Wayne Chang, Amber Hoak, Johnny Johnson, Mike Shepperd, and Whitney Hudson. 2022. Participatory Development of a 3D Telemedicine system during Covid: the future of remote consultations. Journal of Plastic, Reconstructive & Aesthetic Surgery (October 2022). https://www.microsoft.com/en-us/research/publication/participatory-development-of-a-3d-telemedicine-system-during-covid-the-future-of-remote-consultations/
[7]
Danilo Gasques, Janet G. Johnson, Tommy Sharkey, Yuanyuan Feng, Ru Wang, Zhuoqun Robin Xu, Enrique Zavala, Yifei Zhang, Wanze Xie, Xinming Zhang, Konrad Davis, Michael Yip, and Nadir Weibel. 2021. ARTEMIS: A Collaborative Mixed-Reality System for Immersive Surgical Telementoring. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 662, 14 pages.
[8]
David Harris, Mark Wilson, and Samuel Vine. 2020. Development and validation of a simulation workload measure: the simulation task load index (SIM-TLX). Virtual Reality (2020), 557–566.
[9]
Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting. 904–908.
[10]
Daria Joanna Hemmerling, Maciej Stroinski, Kamil Kwarciak, Krzysztof Trusiak, Maciej Szymkowski, Weronika Celniak, William Frier, Orestis Georgiou, and Mykola Maksymenko. 2022. Touchless Touch with Biosignal Transfer for Online Communication. In Proceedings of the 2022 International Conference on Multimodal Interaction (Bengaluru, India) (ICMI ’22). Association for Computing Machinery, New York, NY, USA, 579–590. https://doi.org/10.1145/3536221.3556585
[11]
Steven J Henderson and Steven K Feiner. 2011. Augmented reality in the psychomotor phase of a procedural task. In 2011 10th IEEE international symposium on mixed and augmented reality. IEEE, 191–200.
[12]
S. K. Janani and P. Swarnalatha. 2020. Study of Human–Computer Interaction in Augmented Reality. In Soft Computing for Problem Solving, Kedar Nath Das, Jagdish Chand Bansal, Kusum Deep, Atulya K. Nagar, Ponnambalam Pathipooranam, and Rani Chinnappa Naidu (Eds.). Springer Singapore, Singapore, 835–846.
[13]
Changjuan Jing, Johan Potgieter, Frazer Noble, and Ruili Wang. 2017. A comparison and analysis of RGB-D cameras’ depth performance for robotics application. In 2017 24th International Conference on Mechatronics and Machine Vision in Practice (M2VIP). 1–6. https://doi.org/10.1109/M2VIP.2017.8211432
[14]
Jens Kessler, Jessica T. Wegener, Markus W. Hollmann, and Markus F. Stevens. 2016. Teaching concepts in ultrasound-guided regional Anesthesia. Current Opinion in Anaesthesiology 29 (2016), 608–613. Issue 5. https://doi.org/10.1097/ACO.0000000000000381
[15]
Seungwon Kim, Mark Billinghurst, and Kangsoo Kim. 2020. Multimodal interfaces and communication cues for remote collaboration. Journal on Multimodal User Interfaces 14 (12 2020), 313–319. https://doi.org/10.1007/s12193-020-00346-8
[16]
Leo Kobayashi, Xiao Chi Zhang, Scott A Collins, Naz Karim, and Derek L Merck. 2018. Exploratory application of augmented reality/mixed reality devices for acute care procedure training. Western Journal of Emergency Medicine 19, 1 (2018), 158.
[17]
Mathias N. Lystbæk, Peter Rosenberg, Ken Pfeuffer, Jens Emil Grønbæk, and Hans Gellersen. 2022. Gaze-Hand Alignment: Combining Eye Gaze and Mid-Air Pointing for Interacting with Menus in Augmented Reality. 6, ETRA, Article 145 (may 2022), 18 pages. https://doi.org/10.1145/3530886
[18]
Faraz Mahmood, Eitezaz Mahmood, Robert Gregory Dorfman, John Mitchell, Feroze Udin Mahmood, Stephanie B. Jones, and Robina Matyal. 2018. Augmented Reality and Ultrasound Education: Initial Experience. Journal of Cardiothoracic and Vascular Anesthesia 32 (6 2018), 1363–1367. Issue 3. https://doi.org/10.1053/j.jvca.2017.12.006
[19]
Bernardo Marques, Samuel Silva, João Alves, Tiago Araújo, Paulo Dias, and Beatriz Sousa Santos. 2022. A Conceptual Model and Taxonomy for Collaborative Augmented Reality. IEEE Transactions on Visualization and Computer Graphics 28, 12 (2022), 5113–5133. https://doi.org/10.1109/TVCG.2021.3101545
[20]
Carey Mather, Tony Barnett, Vlasti Broucek, Annette Saunders, Darren Grattidge, and Weidong Huang. 2017. Helping hands: using augmented reality to provide remote guidance to health professionals. In Context Sensitive Health Informatics: Redesigning Healthcare Work. IOS Press, 57–62.
[21]
David Mcneill. 1994. Hand and Mind: What Gestures Reveal About Thought. Bibliovault OAI Repository, the University of Chicago Press 27 (06 1994). https://doi.org/10.2307/1576015
[22]
Brendan William Munzer, Mohammad Mairaj Khan, Barbara Shipman, and Prashant Mahajan. 2019. Augmented reality in emergency medicine: a scoping review. Journal of medical Internet research 21, 4 (2019), e12368.
[23]
Delwyn Nicholls, Linda Sweet, and Jon Hyett. 2014. Psychomotor Skills in Medical Ultrasound Imaging. Journal of Ultrasound in Medicine 33, 8 (2014), 1349–1352. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.7863/ultra.33.8.1349https://onlinelibrary.wiley.com/doi/abs/10.7863/ultra.33.8.1349
[24]
Delwyn Nicholls, Linda Sweet, Amanda Muller, and Jon Hyett. 2016. Teaching psychomotor skills in the twenty-first century: Revisiting and reviewing instructional approaches through the lens of contemporary literature. Medical Teacher 38, 10 (2016), 1056–1063.
[25]
Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L. Davidson, Sameh Khamis, Mingsong Dou, Vladimir Tankovich, Charles Loop, Qin Cai, Philip A. Chou, Sarah Mennicken, Julien Valentin, Vivek Pradeep, Shenlong Wang, Sing Bing Kang, Pushmeet Kohli, Yuliya Lutchyn, Cem Keskin, and Shahram Izadi. 2016. Holoportation: Virtual 3D Teleportation in Real-Time. In UIST 2016. https://www.microsoft.com/en-us/research/publication/holoportation-virtual-3d-teleportation-in-real-time/
[26]
Manuel Rebol, Christian Gütl, and Krzysztof Pietroszek. 2021. Real-Time Gesture Animation Generation from Speech for Virtual Human Interaction. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21). Association for Computing Machinery, New York, NY, USA, Article 197, 4 pages. https://doi.org/10.1145/3411763.3451554
[27]
Manuel Rebol, Christian Gütl, and Krzysztof Pietroszek. 2021. Passing a Non-verbal Turing Test: Evaluating Gesture Animations Generated from Speech. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 573–581. https://doi.org/10.1109/VR50410.2021.00082
[28]
Manuel Rebol, Krzysztof Pietroszek, Claudia Ranniger, Colton Hood, Adam Rutenberg, Neal Sikka, and Christian Guetl. 2023. Collaborative system design of mixed reality communication for medical training. (2023), 418–427. https://hdl.handle.net/10125/102680
[29]
Manuel Rebol, Krzysztof Pietroszek, Claudia Ranniger, Colton Hood, Adam Rutenberg, Neal Sikka, David Li, and Christian Gütl. 2022. Mixed Reality Communication for Medical Procedures: Teaching the Placement of a Central Venous Catheter. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 346–354. https://doi.org/10.1109/ISMAR55827.2022.00050
[30]
Manuel Rebol, Alexander Steinmaurer, Florian Gamillscheg, Krzysztof Pietroszek, Christian Gütl, Claudia Ranniger, Colton Hood, Adam Rutenberg, and Neal Sikka. 2023. CPR Emergency Assistance Through Mixed Reality Communication. In Augmented Intelligence and Intelligent Tutoring Systems, Claude Frasson, Phivos Mylonas, and Christos Troussas (Eds.). Springer Nature Switzerland, Cham, 415–429.
[31]
Julian Rodriguez. 2021. A comparison of an RGB-D cameras performance and a stereo camera in relation to object recognition and spatial position determination. ELCVIA Electronic Letters on Computer Vision and Image Analysis 20 (01 2021), 16. https://doi.org/10.5565/rev/elcvia.1238
[32]
Daniel Roth, Kevin Yu, Frieder Pankratz, Gleb Gorbachev, Andreas Keller, Marc Lazarovici, Dirk Wilhelm, Simon Weidert, Nassir Navab, and Ulrich Eck. 2021. Real-time Mixed Reality Teleconsultation for Intensive Care Units in Pandemic Situations. In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). 693–694.
[33]
DS Schoeb, Jessica Schwarz, Simon Hein, D Schlager, Philippe Fabian Pohlmann, Alexander Frankenschmidt, Christian Gratzke, and Arkadiusz Miernik. 2020. Mixed reality for teaching catheter placement to medical students: a randomized single-blinded, prospective trial. BMC medical education 20, 1 (2020), 1–8.
[34]
Peter Schopf and Julia M Jonas. 2023. Affordances of Augmented Reality Systems for Co-Located Collaboration. (2023).
[35]
Ward’s science. 2022. Blue Phantom® Central Line Ultrasound Trainers. https://www.wardsci.com/store/product/29606535/blue-phantom-central-line-ultrasound-trainers [Online; accessed 11-September-2022].
[36]
Mickael Sereno, Xiyao Wang, Lonni Besançon, Michael J. McGuffin, and Tobias Isenberg. 2022. Collaborative Work in Augmented Reality: A Survey. IEEE Transactions on Visualization and Computer Graphics 28, 6 (2022), 2530–2549. https://doi.org/10.1109/TVCG.2020.3032761
[37]
Brian Sites, Brian Spence, John Gallagher, Christopher Wiley, Marc Bertrand, and George Blike. 2007. Sites BD, Spence BC, Gallagher JD, Wiley CW, Bertrand ML, Blike GT. Characterizing novice behavior associated with learning ultrasound-guided peripheral regional anesthesia. Regional anesthesia and pain medicine 32 (03 2007), 107–15. https://doi.org/10.1016/j.rapm.2006.11.006
[38]
Sonosite. 2022. SONOSITE M-TURBO. https://www.sonosite.com/products/sonosite-m-turbo [Online; accessed 11-September-2022].
[39]
Robin Strak, Kevin Yu, Frieder Pankratz, Marc Lazarovici, Benedikt Sandmeyer, Julia Reichling, Simon Weidert, Clemens Kraetsch, Barbara Roegele, Nassir Navab, Ulrich Eck, and Daniel Roth. 2021. Comparison Between Video-Mediated and Asymmetric 3D Teleconsultation During a Preclinical Scenario. In Proceedings of Mensch Und Computer 2021 (Ingolstadt, Germany) (MuC ’21). Association for Computing Machinery, New York, NY, USA, 227–235. https://doi.org/10.1145/3473856.3473883
[40]
Minghui Sun, Xinyu Wu, Zhihua Fan, and Liyan Dong. 2019. Augmented reality based educational design for children. International Journal of Emerging Technologies in Learning (Online) 14, 3 (2019), 51.
[41]
Peng Wang, Xiaoliang Bai, Mark Billinghurst, Shusheng Zhang, Xiangyu Zhang, Shuxia Wang, Weiping He, Yuxiang Yan, and Hongyu Ji. 2021. AR/MR Remote Collaboration on Physical Tasks: A Review. Robotics and Computer-Integrated Manufacturing 72 (2021), 102071. https://doi.org/10.1016/j.rcim.2020.102071
[42]
Pierre Wijdenes, David Borkenhagen, Julie Babione, Irene Ma, and Greg Hallihan. 2018. Leveraging augmented reality training tool for medical education: A case study in central venous catheterization. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. 1–7.
[43]
Jong-Hyun Yoon, Jong-Seung Park, and Mee Young Sung. 2006. Vision-Based Bare-Hand Gesture Interface for Interactive Augmented Reality Applications. In Entertainment Computing - ICEC 2006, Richard Harper, Matthias Rauterberg, and Marco Combetto (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 386–389.
[44]
Kevin Yu, Ulrich Eck, Frieder Pankratz, Marc Lazarovici, Dirk Wilhelm, and Nassir Navab. 2022. Duplicated Reality for Co-located Augmented Reality Collaboration. IEEE Transactions on Visualization and Computer Graphics 28, 5 (2022), 2190–2200.

Cited By

View all
  • (2024)Stand Alone or Stay Together: An In-situ Experiment of Mixed-Reality Applications in Embryonic Anatomy EducationProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687706(1-11)Online publication date: 9-Oct-2024
  • (2024)MR Microsurgical Suture Training System with Level-Appropriate SupportProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642324(1-19)Online publication date: 11-May-2024

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
VRST '23: Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology
October 2023
542 pages
ISBN:9798400703287
DOI:10.1145/3611659
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 October 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Augmented Reality
  2. Remote Collaboration
  3. Telehealth
  4. Volumetric Communication

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

VRST 2023

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)199
  • Downloads (Last 6 weeks)41
Reflects downloads up to 14 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Stand Alone or Stay Together: An In-situ Experiment of Mixed-Reality Applications in Embryonic Anatomy EducationProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687706(1-11)Online publication date: 9-Oct-2024
  • (2024)MR Microsurgical Suture Training System with Level-Appropriate SupportProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642324(1-19)Online publication date: 11-May-2024

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media