More Web Proxy on the site http://driver.im/

research-article

Authoring Interactive and Immersive Experiences Using Programming by Demonstration

Authors:

Edwige Chauvergne,

Arnaud ProuzeauAuthors Info & Claims

IHM '23: Proceedings of the 34th Conference on l'Interaction Humain-Machine

Article No.: 20, Pages 1 - 13

https://doi.org/10.1145/3583961.3583981

Published: 29 May 2023 Publication History

Abstract

Immersive technologies, like virtual and augmented reality, allow engaging the general public in interactive experiences, which are particularly promising in educational and cultural activities (e.g. museums, exhibits). For now, the design of such interactive experiences requires extensive knowledge of expert programming tools, and thus they are not accessible to non-developers. A promising method to ease the prototyping of interactive scenes for non-expert users stands on the concept of programming-by-demonstration. With such an approach, novice users can simply demonstrate an interaction or the expected behavior of a virtual object to build their prototype, without knowing the underlying coding mechanisms. In this paper, we propose an immersive authoring system that bases on that approach. We prototyped two use cases based on this system, as well as a user study. Using our observations and the results of the study, we discuss challenges associated with the design of such systems and provide guidelines for the development of future immersive programming-by-demonstration tools.

Supplemental Material

MP4 File

Presentation video - short version

Download
92.37 MB

References

[1]

Rahul Arora, Rubaiat Habib Kazi, Danny M. Kaufman, Wilmot Li, and Karan Singh. 2019. MagicalHands: Mid-Air Hand Gestures for Animating in VR. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 463–477. https://doi.org/10.1145/3332165.3347942

Digital Library

[2]

Narges Ashtari, Andrea Bunt, Joanna McGrenere, Michael Nebeling, and Parmit K. Chilana. 2020. Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and Opportunities. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376722

Digital Library

[3]

Lee Beever, Serban Pop, and Nigel W. John. 2020. LevelEd VR: A virtual reality level editor and workflow for virtual reality level design. In 2020 IEEE Conference on Games (CoG). IEEE, New York, NY, USA, 136–143. https://doi.org/10.1109/CoG47356.2020.9231769

[4]

Bhaskar Bhattacharya and Eliot H. Winer. 2019. Augmented reality via expert demonstration authoring (AREDA). Computers in Industry 105 (2019), 61–79. https://doi.org/10.1016/j.compind.2018.04.021

Digital Library

[5]

Cynthia Breazeal, Matt Berlin, Andrew Brooks, Jesse Gray, and Andrea L. Thomaz. 2006. Using perspective taking to learn from ambiguous demonstrations. Robotics and Autonomous Systems 54, 5 (2006), 385–393. https://doi.org/10.1016/j.robot.2006.02.004 The Social Mechanisms of Robot Programming from Demonstration.

[6]

Colin Burns, Eric Dishman, William Verplank, and Bud Lassiter. 1994. Actors, Hairdos & Videotape—Informance Design. In Conference Companion on Human Factors in Computing Systems (Boston, Massachusetts, USA) (CHI ’94). Association for Computing Machinery, New York, NY, USA, 119–120. https://doi.org/10.1145/259963.260102

Digital Library

[7]

Yuanzhi Cao, Tianyi Wang, Xun Qian, Pawan S. Rao, Manav Wadhawan, Ke Huo, and Karthik Ramani. 2019. GhostAR: A Time-Space Editor for Embodied Authoring of Human-Robot Collaborative Task with Augmented Reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 521–534. https://doi.org/10.1145/3332165.3347902

Digital Library

[8]

Daniel W. Carruth. 2017. Virtual reality for education and workforce training. In 2017 15th International Conference on Emerging eLearning Technologies and Applications (ICETA). IEEE, New York, NY, USA, 1–6. https://doi.org/10.1109/ICETA.2017.8102472

[9]

Marco Cavallo and Angus G. Forbes. 2019. CAVE-AR: A VR Authoring System to Interactively Design, Simulate, and Debug Multi-user AR Experiences. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, New York, NY, USA, 872–873. https://doi.org/10.1109/VR.2019.8798148

[10]

Zhutian Chen, Yijia Su, Yifang Wang, Qianwen Wang, Huamin Qu, and Yingcai Wu. 2020. MARVisT: Authoring Glyph-Based Visualization in Mobile Augmented Reality. IEEE Transactions on Visualization and Computer Graphics 26, 8 (2020), 2645–2658. https://doi.org/10.1109/TVCG.2019.2892415

Digital Library

[11]

Barrett Ens, Fraser Anderson, Tovi Grossman, Michelle Annett, Pourang Irani, and George Fitzmaurice. 2017. Ivy: Exploring Spatially Situated Visual Programming for Authoring and Understanding Intelligent Environments. In GI ’17: Proceedings of the 43rd Graphics Interface Conference. Canadian Human-Computer Communications Society, Waterloo, Canada.

Digital Library

[12]

Laura Freina and Michela Ott. 2015. A literature review on immersive virtual reality in education: state of the art and perspectives. In The international scientific conference elearning and software for education, Vol. 1. eLSE, Bucharest, Romania, 10–1007.

[13]

Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, Amsterdam, Holland, 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9

[14]

Björn Hartmann, Leith Abdulla, Manas Mittal, and Scott R. Klemmer. 2007. Authoring Sensor-Based Interactions by Demonstration with Direct Manipulation and Pattern Recognition. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’07). Association for Computing Machinery, New York, NY, USA, 145–154. https://doi.org/10.1145/1240624.1240646

Digital Library

[15]

Valentin Heun, James Hobin, and Pattie Maes. 2013. Reality Editor: Programming Smarter Objects. In Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication (Zurich, Switzerland) (UbiComp ’13 Adjunct). Association for Computing Machinery, New York, NY, USA, 307–310. https://doi.org/10.1145/2494091.2494185

Digital Library

[16]

Shotaro Ichikawa, Kazuki Takashima, Anthony Tang, and Yoshifumi Kitamura. 2018. VR Safari Park: A Concept-Based World Building Interface Using Blocks and World Tree. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (Tokyo, Japan) (VRST ’18). Association for Computing Machinery, New York, NY, USA, Article 6, 5 pages. https://doi.org/10.1145/3281505.3281517

Digital Library

[17]

Ananya Ipsita, Hao Li, Runlin Duan, Yuanzhi Cao, Subramanian Chidambaram, Min Liu, and Karthik Ramani. 2021. VRFromX: From Scanned Reality to Interactive Virtual Experience with Human-in-the-Loop. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21). Association for Computing Machinery, New York, NY, USA, Article 289, 7 pages. https://doi.org/10.1145/3411763.3451747

Digital Library

[18]

Ben Lang. 2021. 12 Tools for Painting, Sculpting, & Animating in VR. https://www.roadtovr.com/vr-painting-drawing-modeling-animation-art-tools-quest-pc/. Accessed: 2022-10-05.

[19]

Flavien Lécuyer, Valérie Gouranton, Adrien Reuzeau, Ronan Gaugne, and Bruno Arnaldi. 2019. Create by Doing – Action Sequencing in VR. In Advances in Computer Graphics, Marina Gavrilova, Jian Chang, Nadia Magnenat Thalmann, Eckhard Hitzer, and Hiroshi Ishikawa (Eds.). Springer International Publishing, Cham, 329–335.

[20]

Gun A. Lee, Gerard J. Kim, and Mark Billinghurst. 2005. Immersive Authoring: What You EXperience Is What You Get (WYXIWYG). Commun. ACM 48, 7 (July 2005), 76–81. https://doi.org/10.1145/1070838.1070840

Digital Library

[21]

Germán Leiva, Jens Emil Grønbæk, Clemens Nylandsted Klokmose, Cuong Nguyen, Rubaiat Habib Kazi, and Paul Asente. 2021. Rapido: Prototyping Interactive AR Experiences through Programming by Demonstration. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 626–637. https://doi.org/10.1145/3472749.3474774

Digital Library

[22]

Hao Lü and Yang Li. 2013. Gesture Studio: Authoring Multi-Touch Interactions through Demonstration and Declaration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 257–266. https://doi.org/10.1145/2470654.2470690

Digital Library

[23]

Blair MacIntyre, Maribeth Gandy, Steven Dow, and Jay David Bolter. 2005. DART: A Toolkit for Rapid Design Exploration of Augmented Reality Experiences. ACM Trans. Graph. 24, 3 (July 2005), 932. https://doi.org/10.1145/1073204.1073288

Digital Library

[24]

Sina Masnadi, Andrés N. Vargas González, Brian Williamson, and Joseph J. LaViola. 2020. AffordIt!: A Tool for Authoring Object Component Behavior in VR. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, New York, NY, USA, 740–741. https://doi.org/10.1109/VRW50115.2020.00221

[25]

Leon Müller, Ken Pfeuffer, Jan Gugenheimer, Bastian Pfleging, Sarah Prange, and Florian Alt. 2021. SpatialProto: Exploring Real-World Motion Captures for Rapid Prototyping of Interactive Mixed Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 363, 13 pages. https://doi.org/10.1145/3411764.3445560

Digital Library

[26]

Michael Nebeling and Katy Madier. 2019. 360proto: Making Interactive Virtual Reality & Augmented Reality Prototypes from Paper. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300826

Digital Library

[27]

Michael Nebeling, Janet Nebeling, Ao Yu, and Rob Rumble. 2018. ProtoAR: Rapid Physical-Digital Prototyping of Mobile Augmented Reality Applications. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3173927

Digital Library

[28]

Michael Nebeling and Maximilian Speicher. 2018. The Trouble with Augmented Reality/Virtual Reality Authoring Tools. In 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, New York, NY, USA, 333–337. https://doi.org/10.1109/ISMAR-Adjunct.2018.00098

[29]

Gary Ng, Joon Gi Shin, Alexander Plopski, Christian Sandor, and Daniel Saakes. 2018. Situated Game Level Editing in Augmented Reality. In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction (Stockholm, Sweden) (TEI ’18). Association for Computing Machinery, New York, NY, USA, 409–418. https://doi.org/10.1145/3173225.3173230

Digital Library

[30]

Antti Oulasvirta, Esko Kurvinen, and Tomi Kankainen. 2003. Understanding Contexts by Being There: Case Studies in Bodystorming. Personal Ubiquitous Comput. 7, 2 (jul 2003), 125–134. https://doi.org/10.1007/s00779-003-0238-7

Digital Library

[31]

Arnaud Prouzeau, Yuchen Wang, Barrett Ens, Wesley Willett, and Tim Dwyer. 2020. Corsican Twin: Authoring In Situ Augmented Reality Visualisations in Virtual Reality. In Proceedings of the International Conference on Advanced Visual Interfaces (Salerno, Italy) (AVI ’20). Association for Computing Machinery, New York, NY, USA, Article 11, 9 pages. https://doi.org/10.1145/3399715.3399743

Digital Library

[32]

Bahador Saket, Hannah Kim, Eli T. Brown, and Alex Endert. 2017. Visualization by Demonstration: An Interaction Paradigm for Visual Data Exploration. IEEE Transactions on Visualization and Computer Graphics 23, 1 (2017), 331–340. https://doi.org/10.1109/TVCG.2016.2598839

Digital Library

[33]

Ronell Sicat, Jiabao Li, Junyoung Choi, Maxime Cordeil, Won-Ki Jeong, Benjamin Bach, and Hanspeter Pfister. 2019. DXR: A Toolkit for Building Immersive Data Visualizations. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2019), 715–725. https://doi.org/10.1109/TVCG.2018.2865152

Digital Library

[34]

Richard Stoakley, Matthew J. Conway, and Randy Pausch. 1995. Virtual Reality on a WIM: Interactive Worlds in Miniature. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’95). ACM Press/Addison-Wesley Publishing Co., USA, 265–272. https://doi.org/10.1145/223904.223938

Digital Library

[35]

Ryo Suzuki, Rubaiat Habib Kazi, Li-yi Wei, Stephen DiVerdi, Wilmot Li, and Daniel Leithinger. 2020. RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching. Association for Computing Machinery, New York, NY, USA, 166–181. https://doi.org/10.1145/3379337.3415892

Digital Library

[36]

Anastasios Theodoropoulos and Angeliki Antoniou. 2022. VR Games in Cultural Heritage: A Systematic Review of the Emerging Fields of Virtual Reality and Culture Games. Applied Sciences 12, 17 (2022), 19. https://doi.org/10.3390/app12178476

[37]

Andrés Vargas González, Senglee Koh, Katelynn Kapalo, Robert Sottilare, Patrick Garrity, Mark Billinghurst, and Joseph LaViola. 2019. A Comparison of Desktop and Augmented Reality Scenario Based Training Authoring Tools. In 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, New York, NY, USA, 339–350. https://doi.org/10.1109/ISMAR.2019.00032

[38]

F. Vernier and L. Nigay. 2001. A Framework for the Combination and Characterization of Output Modalities. In Interactive Systems Design, Specification, and Verification, Philippe Palanque and Fabio Paternò (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 35–50.

[39]

Tianyi Wang, Xun Qian, Fengming He, Xiyun Hu, Yuanzhi Cao, and Karthik Ramani. 2021. GesturAR: An Authoring System for Creating Freehand Interactive Augmented Reality Applications. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 552–567. https://doi.org/10.1145/3472749.3474769

Digital Library

[40]

Zeyu Wang, Cuong Nguyen, Paul Asente, and Julie Dorsey. 2021. DistanciAR: Authoring Site-Specific Augmented Reality Experiences for Remote Environments. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, Article 411, 12 pages. https://doi.org/10.1145/3411764.3445552

Digital Library

[41]

E. Yigitbas, J. Klauke, S. Gottschalk, and G. Engels. 2021. VREUD - An End-User Development Tool to Simplify the Creation of Interactive VR Scenes. In 2021 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE Computer Society, Los Alamitos, CA, USA, 1–10. https://doi.org/10.1109/VL/HCC51201.2021.9576372

[42]

Lei Zhang and Steve Oney. 2020. FlowMatic: An Immersive Authoring Tool for Creating Interactive Scenes in Virtual Reality. Association for Computing Machinery, New York, NY, USA, 342–353. https://doi.org/10.1145/3379337.3415824

Digital Library

Cited By

Eroglu SSchmitz PSinke KAnders DKuhlen TWeyers B(2024)Choose Your Reference Frame Right: An Immersive Authoring Technique for Creating Reactive BehaviorProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687744(1-11)Online publication date: 9-Oct-2024
https://dl.acm.org/doi/10.1145/3641825.3687744

Index Terms

Authoring Interactive and Immersive Experiences Using Programming by Demonstration
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Virtual reality
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software prototyping
  2. Software notations and tools
    1. Context specific languages
      1. Programming by example

Recommendations

Immersive interactive technologies and virtual shopping experiences: Differences in consumer perceptions between augmented reality (AR) and virtual reality (VR)
Highlights
- This study investigated the functional mechanisms of AR and VR in impacting user perceptions and behaviors in virtual shopping environments.
Abstract
Based on the concepts of the Reality-Virtuality (RV) continuum and the stimulus-organism-response (SOR) framework, this study investigated differences between AR and VR in their effects on vividness/interactivity, a sense of presence, ...
Experience Design of the Immersive Film: Panoramic View and Interactive Narratives
Design, User Experience, and Usability: Theory and Practice
Abstract
When the immersive filmwork is experienced by the audiences with head-mounted display,the interactive experience of immersive images lies in that the audiences (users) need to be part of the story in terms of the visual perception experience and ...
Analogue and digital immersive experiences: what should digital creators learn from live theatre makers?
EVA '18: Proceedings of the Conference on Electronic Visualisation and the Arts

In every century, there are different apparatuses that are redefining and refining the notion of immersion. After experiencing panoramas, planetariums or even IMAX movies, we are now longing to see whether VR can really fulfil our expectations. We are ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

IHM '23: Proceedings of the 34th Conference on l'Interaction Humain-Machine

April 2023

288 pages

ISBN:9781450398244

DOI:10.1145/3583961

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 May 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Data Availability

Presentation video - short version https://dl.acm.org/doi/10.1145/3583961.3583981#ihm23a-sub4834-cam-i43.mp4

Conference

IHM '23

IHM '23: 34th International Francophone Conference on Human-Computer Interaction

April 3 - 6, 2023

TROYES, France

Acceptance Rates

Overall Acceptance Rate 103 of 199 submissions, 52%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
117
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)7

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Eroglu SSchmitz PSinke KAnders DKuhlen TWeyers B(2024)Choose Your Reference Frame Right: An Immersive Authoring Technique for Creating Reactive BehaviorProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687744(1-11)Online publication date: 9-Oct-2024
https://dl.acm.org/doi/10.1145/3641825.3687744

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents