More Web Proxy on the site http://driver.im/

short-paper

Open access

ROVER: A Standalone Overlay Tool for Questionnaires in Virtual Reality

Authors:

Lucas Küntzer,

Sandra U. Schwab,

Heike Spaderna,

Georg RockAuthors Info & Claims

EICS '24 Companion: Companion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems

Pages 31 - 39

https://doi.org/10.1145/3660515.3661328

Published: 24 June 2024 Publication History

All formats PDF

Abstract

Questionnaires are a prevalent method for evaluating user experience in Virtual Reality (VR) research, yet their integration into VR environments often requires significant technical know-how, limiting broader use. We present ROVER (Rating Overlay for Virtual Environments in Research), a novel, standalone tool designed to immersively embed questionnaires within existing VR applications. Informed by a comprehensive review of existing literature, ROVER was designed and implemented through an iterative, cross-disciplinary approach. Its practicality and usability were evaluated as part of a psychological pilot study involving 21 participants, demonstrating ROVER’s ease of use, reliability in gathering self-reports, and its contribution to the efficiency of assessments. ROVER allows for easy no-code configuration and features an interactive tutorial and improved instructor-participant communication using virtual avatars for assisted onboarding. By releasing ROVER, we aim to facilitate VR user experience research for researchers with various degrees of technical backgrounds and contribute to standardization in the field.

Supplemental Material

MP4 File

Video Figure

Download
40.40 MB

References

[1]

Ana Agić, Lidija Mandić, and Nikolina Stanić Loknar. 2022. Legibility of typefaces and preferences of text/background color variations in virtual environment. In Proceedings - The Eleventh International Symposium GRID 2022. University of Novi Sad, Faculty of technical sciences, Department of graphic engineering and design, Novi Sad, Serbia, 835–841. https://doi.org/10.24867/GRID-2022-p92

[2]

Andrea C. Aleman, Min Wang, and Frank Schaeffel. 2018. Reading and Myopia: Contrast Polarity Matters. Scientific Reports 8, 1 (2018), 10840. https://doi.org/10.1038/s41598-018-28904-x

[3]

Dmitry Alexandrovsky, Susanne Putze, Michael Bonfert, Sebastian Höffner, Pitt Michelmann, Dirk Wenig, Rainer Malaka, and Jan David Smeddinck. 2020. Examining Design Choices of Questionnaires in VR User Studies. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Regina Bernhaupt, Florian ’Floyd’ Mueller, David Verweij, Josh Andres, Joanna McGrenere, Andy Cockburn, Ignacio Avellino, Alix Goguey, Pernille Bjørn, Shengdong Zhao, Briane Paul Samson, and Rafal Kocielnik (Eds.). Association for Computing Machinery, [S.l.], 1–21. https://doi.org/10.1145/3313831.3376260

Digital Library

[4]

Chiara Ancona, Monica Stoppani, Veronica Odazio, Carlo La Spina, Giulia Corradetti, and Francesco Bandello. 2014. Stereo tests as a screening tool for strabismus: which is the best choice?Clinical ophthalmology (Auckland, N.Z.) 8 (2014), 2221–2227. https://doi.org/10.2147/OPTH.S67488

[5]

Anil Ufuk Batmaz, Mayra Donaji Barrera Machuca, Junwei Sun, and Wolfgang Stuerzlinger. 2022. The Effect of the Vergence-Accommodation Conflict on Virtual Hand Pointing in Immersive Displays. In CHI Conference on Human Factors in Computing Systems(ACM Digital Library), Simone Barbosa (Ed.). Association for Computing Machinery, New York,NY,United States, 1–15. https://doi.org/10.1145/3491102.3502067

Digital Library

[6]

J. Birch. 1997. Efficiency of the Ishihara test for identifying red-green colour deficiency. Ophthalmic and Physiological Optics 17, 5 (1997), 403–408. https://doi.org/10.1016/S0275-5408(97)00022-7

[7]

Virginia Braun and Victoria Clarke. 2022. Thematic analysis: A practical guide. SAGE, Los Angeles.

[8]

Jack Brookes, Matthew Warburton, Mshari Alghadier, Mark Mon-Williams, and Faisal Mushtaq. 2020. Studying human behavior with virtual reality: The Unity Experiment Framework. Behavior Research Methods 52, 2 (2020), 455–463. https://doi.org/10.3758/s13428-019-01242-0

[9]

Edwige Chauvergne, Martin Hachet, and Arnaud Prouzeau. 2023. User Onboarding in Virtual Reality: An Investigation of Current Practices. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Albrecht Schmidt, Kaisa Väänänen, Tesh Goyal, Per Ola Kristensson, Anicia Peters, Stefanie Mueller, Julie R. Williamson, and Max L. Wilson (Eds.). ACM, New York, NY, USA, 1–15. https://doi.org/10.1145/3544548.3581211

Digital Library

[10]

DIN Deutsches Institut für Normung e. V.1993-07. DIN 1450 Lettering - Legibility. https://www.din.de/en/getting-involved/standards-committees/natg/publications/wdc-beuth:din21:170093157

[11]

Tilman Dingler, Kai Kunze, and Benjamin Outram. 2018. VR Reading UIs. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, Regan Mandryk and A. Special Interest Group on Computer-HumanC.M. Interaction (Eds.). ACM, Montreal QC, Canada, 1–6. https://doi.org/10.1145/3170427.3188695

Digital Library

[12]

Martin Feick, Niko Kleer, Anthony Tang, and Antonio Krüger. 2020. The Virtual Reality Questionnaire Toolkit. In Adjunct Publication of the 33rd Annual ACM Symposium on User Interface Software and Technology(ACM Digital Library), Shamsi Iqbal (Ed.). Association for Computing Machinery, New York,NY,United States, 68–69. https://doi.org/10.1145/3379350.3416188

Digital Library

[13]

Kraig Finstad. 2010. The Usability Metric for User Experience. Interacting with Computers 22, 5 (2010), 323–327. https://doi.org/10.1016/j.intcom.2010.04.004

Digital Library

[14]

Paul M. Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology 47, 6 (1954), 381–391. https://doi.org/10.1037/h0055392

[15]

Thomas Franke, Christiane Attig, and Daniel Wessel. 2019. A Personal Resource for Technology Interaction: Development and Validation of the Affinity for Technology Interaction (ATI) Scale. International Journal of Human–Computer Interaction 35, 6 (2019), 456–467. https://doi.org/10.1080/10447318.2018.1456150

[16]

Sarah Graf and Valentin Schwind. 2020. Inconsistencies of Presence Questionnaires in Virtual Reality. In Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology (Virtual Event, Canada) (VRST ’20). Association for Computing Machinery, New York, NY, USA, Article 60, 3 pages. https://doi.org/10.1145/3385956.3422105

Digital Library

[17]

Jan P. Gründling, Daniel Zeiler, and Benjamin Weyers. 2022. Answering With Bow and Arrow: Questionnaires and VR Blend Without Distorting the Outcome. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, [S.l.], 683–692. https://doi.org/10.1109/VR51125.2022.00089

[18]

D. Hepperle, T. Dienlin, and M. Wolfel. 2021. Reducing the Human Factor in Virtual Reality Research to Increase Reproducibility and Replicability. In 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE Computer Society, Los Alamitos, CA, USA, 100–105. https://doi.org/10.1109/ISMAR-Adjunct54149.2021.00030

[19]

Kiran Ijaz, Tram Thi Minh Tran, Ahmet Baki Kocaballi, Rafael A. Calvo, Shlomo Berkovsky, and Naseem Ahmadpour. 2022. Design Considerations for Immersive Virtual Reality Applications for Older Adults: A Scoping Review. Multimodal Technologies and Interaction 6, 7 (2022), 60. https://doi.org/10.3390/mti6070060

[20]

International Organization for Standardization. 2011-03. ISO/IEC 25010:2011-03 (E): Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models. https://www.iso.org/standard/35733.html

[21]

J. Deuchler, W. Hettmann, D. Hepperle, and M. Wölfel. 2023. Streamlining Physiological Observations in Immersive Virtual Reality Studies with the Virtual Reality Scientific Toolkit. In 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, Shanghai, China, 485–488. https://doi.org/10.1109/VRW58643.2023.00105

[22]

Jacek Jankowski, Krystian Samp, Izabela Irzynska, Marek Jozwowicz, and Stefan Decker. 2010. Integrating Text with Video and 3D Graphics. In CHI 2010 - we are HCI, Elizabeth Mynatt, Geraldine Fitzpatrick, Scott Hudson, Keith Edwards, and Tom Rodden (Eds.). Association for Computing Machinery, New York, 1321–1330. https://doi.org/10.1145/1753326.1753524

Digital Library

[23]

Behrang Keshavarz and Heiko Hecht. 2011. Validating an efficient method to quantify motion sickness. Human factors 53, 4 (2011), 415–426. https://doi.org/10.1177/0018720811403736

[24]

Jarrod Knibbe, Jonas Schjerlund, Mathias Petraeus, and Kasper Hornbæk. 2018. The Dream is Collapsing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Regan Mandryk, Mark Hancock, Mark Perry, and Anna Cox (Eds.). ACM, [S.l.], 1–13. https://doi.org/10.1145/3173574.3174057

Digital Library

[25]

Tanja Kojić, Robert Spang, Maurizio Vergari, Luis Meier, Sebastian Möller, and Jan-Niklas Voigt-Antons. 2023. Effects of user factors on user experience in virtual reality: age, gender, and VR experience as influencing factors for VR exergames. Quality and User Experience 8, 1 (2023), 1–15. https://doi.org/10.1007/s41233-023-00056-5

[26]

Tanja Kojic, Maurizio Vergari, Sebastian Möller, and Jan-Niklas Voigt-Antons. 2022. Assessing User Experience Of Text Readability With Eye Tracking In Virtual Reality. In Virtual, Augmented and Mixed Reality: Design and Development: 14th International Conference, VAMR 2022, Held as Part of the 24th HCI International Conference, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, Part I. Springer-Verlag, Berlin, Heidelberg, 199–211. https://doi.org/10.1007/978-3-031-05939-1_13

Digital Library

[27]

Lucas Küntzer, Sandra Schwab, Heike Spaderna, and Georg Rock. 2024. Measuring User Experience of Older Adults during Virtual Reality Exergaming. In Proceedings of the 16th International Conference on Quality of Multimedia Experience (QoMEX 2024). IEEE, Karlshamn, Sweden, 1–7.

[28]

Yu Lu, Yong Hu, Xukun Shen, and Zhaojia Chen. 2023. An Immersive Layout Framework for Web Design in Virtual Reality. In CHI’23, Albrecht Schmidt, Kaisa Väänänen, Tesh Goyal, Per Ola Kristensson, and Anicia Peters (Eds.). The Association for Computing Machinery, New York, New York, 1–7. https://doi.org/10.1145/3544549.3585889

Digital Library

[29]

M. Alger. 2015. Visual Design Methods for Virtual Reality. (2015). https://www.semanticscholar.org/paper/Visual-Design-Methods-for-Virtual-Reality-Alger/94d9115ffc204c4a904312d6cecaf3032101009c?sort=is-influential&page=2

[30]

M. Wölfel, D. Hepperle, C. F. Purps, J. Deuchler, and W. Hettmann. 2021. Entering a new Dimension in Virtual Reality Research: An Overview of Existing Toolkits, their Features and Challenges. In 2021 International Conference on Cyberworlds (CW). IEEE, Caen, France, 180–187. https://doi.org/10.1109/CW52790.2021.00038

[31]

Helfried Moosbrugger and Augustin Kelava. 2012. Qualitätsanforderungen an einen psychologischen Test (Testgütekriterien). Springer Berlin Heidelberg, Berlin, Heidelberg, 7–26. https://doi.org/10.1007/978-3-642-20072-4_2

[32]

Franz J. Neyer, Juliane Felber, and Claudia Gebhardt. 2012. Entwicklung und Validierung einer Kurzskala zur Erfassung von Technikbereitschaft. Diagnostica 58, 2 (2012), 87–99. https://doi.org/10.1026/0012-1924/a000067

[33]

Joseph O’Hagan, Julie R. Williamson, Florian Mathis, Mohamed Khamis, and Mark McGill. 2023. Re-Evaluating VR User Awareness Needs During Bystander Interactions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Albrecht Schmidt, Kaisa Väänänen, Tesh Goyal, Per Ola Kristensson, Anicia Peters, Stefanie Mueller, Julie R. Williamson, and Max L. Wilson (Eds.). ACM, New York, NY, USA, 1–17. https://doi.org/10.1145/3544548.3581018

Digital Library

[34]

Susanne Putze, Dmitry Alexandrovsky, Felix Putze, Sebastian Höffner, Jan David Smeddinck, and Rainer Malaka. 2020. Breaking The Experience: Effects of Questionnaires in VR User Studies. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Regina Bernhaupt, Florian ’Floyd’ Mueller, David Verweij, Josh Andres, Joanna McGrenere, Andy Cockburn, Ignacio Avellino, Alix Goguey, Pernille Bjørn, Shengdong Zhao, Briane Paul Samson, and Rafal Kocielnik (Eds.). Association for Computing Machinery, [S.l.], 1–15. https://doi.org/10.1145/3313831.3376144

Digital Library

[35]

Georg Regal, Raimund Schatz, Johann Schrammel, and Stefan Suette. 2018. VRate: A Unity3D Asset for integrating Subjective Assessment Questionnaires in Virtual Environments. In QoMEX 2018. Institute of Electrical and Electronics Engineers, Piscataway, New Jersey, 1–3. https://doi.org/10.1109/QoMEX.2018.8463296

[36]

Georg Regal, Jan-Niklas Voigt-Antons, Steven Schmidt, Johann Schrammel, Tanja Kojić, Manfred Tscheligi, and Sebastian Möller. 2019. Questionnaires embedded in virtual environments: reliability and positioning of rating scales in virtual environments. Quality and User Experience 4, 1 (2019), 1–13. https://doi.org/10.1007/s41233-019-0029-1

[37]

Saeed Safikhani, Michael Holly, Alexander Kainz, and Johanna Pirker. 2021. The Influence of in-VR Questionnaire Design on the User Experience. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology(ACM Digital Library), Yuichi Itoh (Ed.). Association for Computing Machinery, New York,NY,United States, 1–8. https://doi.org/10.1145/3489849.3489884

Digital Library

[38]

Rômulo Santos Silva, Artur Martins Mol, and Lucila Ishitani. 2019. Virtual reality for older users: a systematic literature review. International Journal of Virtual Reality 19, 1 (2019), 11–25. https://doi.org/10.20870/IJVR.2019.19.1.2908

[39]

Schneider, Sven, Kuliga, Saskia, Weiser, René, Kammler, Olaf and Fuchkina, Ekaterina. 2018. VREVAL - A BIM-based Framework for User-centered Evaluation of Complex Buildings in Virtual Environments. In Computing for a better tomorrow - Proceedings of the 36th eCAADe Conference, A. Kepczynska-Walczak and S. Bialkowski (Eds.). Vol. 2. eCAAD, Lodz, Poland, 833–842. https://papers.cumincad.org/cgi-bin/works/paper/ecaade2018_361

[40]

Valentin Schwind, Pascal Knierim, Nico Haas, and Niels Henze. 2019. Using Presence Questionnaires in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300590

Digital Library

[41]

Mel Slater and Anthony Steed. 2000. A Virtual Presence Counter. Presence: Teleoperators and Virtual Environments 9, 5 (2000), 413–434. https://doi.org/10.1162/105474600566925

Digital Library

[42]

Henrik Haugom Solum. 2019. Readability in Virtual reality, an investigation into displaying text in a virtual environment. Ph. D. Dissertation. NTNU. https://ntnuopen.ntnu.no/ntnu-xmlui/handle/11250/2617909

[43]

Andrej Somrak, Matevž Pogačnik, and Jože Guna. 2021. Suitability and Comparison of Questionnaires Assessing Virtual Reality-Induced Symptoms and Effects and User Experience in Virtual Environments. Sensors (Basel, Switzerland) 21, 4 (2021). https://doi.org/10.3390/s21041185

[44]

T. Kojić, D. Ali, R. Greinacher, S. Möller, and J. -N. Voigt-Antons. 2020. User Experience of Reading in Virtual Reality — Finding Values for Text Distance, Size and Contrast. In 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX). IEEE, Athlone, Ireland, 1–6. https://doi.org/10.1109/QoMEX48832.2020.9123091

[45]

Reiya Tamaki and Tatsuo Nakajima. 2021. Shoot Down Drones with Your Answer, an Integration of a Questionnaire into a VR Experience. In Proceedings, SUI 2021, Stephen N. Spencer, Franciscio Ortega, and Robert Teather (Eds.). The Association for Computing Machinery, Inc, New York, New York, 1–2. https://doi.org/10.1145/3485279.3488282

Digital Library

[46]

Thomas Franke, Christiane Attig, and Daniel Wessel. 2017. Assessing Affinity for Technology Interaction – The Affinity for Technology Interaction (ATI) Scale. Scale Description – English and German Scale Version. https://doi.org/10.13140/RG.2.2.28679.50081

[47]

Roshan Venkatakrishnan, Rohith Venkatakrishnan, Chih-Han Chung, Yu-Shuen Wang, and Sabarish Babu. 2022. Investigating a Combination of Input Modalities, Canvas Geometries, and Inking Triggers on On-Air Handwriting in Virtual Reality. ACM Transactions on Applied Perception 19, 4 (2022), 1–19. https://doi.org/10.1145/3560817

Digital Library

[48]

Nadine Wagener, Mareike Stamer, Johannes Schöning, and Johannes Tümler. 2020. Investigating Effects and User Preferences of Extra- and Intradiegetic Virtual Reality Questionnaires. In 26th ACM Symposium on Virtual Reality Software and Technology(ACM Digital Library), Robert J. Teather (Ed.). Association for Computing Machinery, New York,NY,United States, 1–11. https://doi.org/10.1145/3385956.3418972

Digital Library

[49]

Uta Wagner, Mathias N. Lystbæk, Pavel Manakhov, Jens Emil Sloth Grønbæk, Ken Pfeuffer, and Hans Gellersen. 2023. A Fitts’ Law Study of Gaze-Hand Alignment for Selection in 3D User Interfaces. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Albrecht Schmidt, Kaisa Väänänen, Tesh Goyal, Per Ola Kristensson, Anicia Peters, Stefanie Mueller, Julie R. Williamson, and Max L. Wilson (Eds.). ACM, New York, NY, USA, 1–15. https://doi.org/10.1145/3544548.3581423

Digital Library

[50]

Raphael P. Weibel, Jascha Grübel, Hantao Zhao, Tyler Thrash, Dario Meloni, Christoph Hölscher, and Victor R. Schinazi. 2018. Virtual Reality Experiments with Physiological Measures. JoVE (Journal of Visualized Experiments) 138 (2018), e58318. https://doi.org/10.3791/58318

[51]

D. J. Wilkinson, M. Piasecki, and P. J. Atherton. 2018. The age-related loss of skeletal muscle mass and function: Measurement and physiology of muscle fibre atrophy and muscle fibre loss in humans. Ageing research reviews 47 (2018), 123–132. https://doi.org/10.1016/j.arr.2018.07.005

[52]

Ying-Chao Tung and Aaron Hertzmann. 2018. Comparing Spatial Interaction Modalities for 2 D-Widgets in Productivity Applications in Virtual Reality. Technical Report. University of Washington. https://www.semanticscholar.org/paper/Comparing-Spatial-Interaction-Modalities-for-2-in-Tung-Hertzmann/573e00ad4a263089c66bc3a61fa4977ae390ce3f

Cited By

Küntzer LSchwab SSpaderna HRock G(2024)Measuring User Experience of Older Adults during Virtual Reality Exergaming2024 16th International Conference on Quality of Multimedia Experience (QoMEX)10.1109/QoMEX61742.2024.10598263(153-159)Online publication date: 18-Jun-2024
https://doi.org/10.1109/QoMEX61742.2024.10598263

Index Terms

ROVER: A Standalone Overlay Tool for Questionnaires in Virtual Reality
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Using Presence Questionnaires in Virtual Reality
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Virtual Reality (VR) is gaining increasing importance in science, education, and entertainment. A fundamental characteristic of VR is creating presence, the experience of 'being' or 'acting', when physically situated in another place. Measuring presence ...
Effects of Third-Person Locomotion Techniques on Sense of Embodiment in Virtual Reality
MUM '24: Proceedings of the International Conference on Mobile and Ubiquitous Multimedia
Virtual Reality (VR) has enabled novel ways to study embodiment and understand how a virtual avatar may be treated as part of a person’s body. These studies mainly employ virtual bodies perceived from a first-person perspective, given that VR has a ...
Thinning trainer based on forest-growth model, virtual reality and computer-aided virtual environment

Immersive virtual reality is applied in many human activities. This virtual reality can be used as a training tool for thinning operations in forests. The aim of this study is to describe the complex solution of Thinning Trainer, that we developed. This ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

EICS '24 Companion: Companion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems

June 2024

129 pages

ISBN:9798400706516

DOI:10.1145/3660515

Editors:
Michael Nebeling
University of Michigan, USA
,
Lucio Davide Spano
University of Cagliari, Italy
,
José Creissac Campos
University of Minho, Portugal

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 June 2024

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

EICS '24

Sponsor:

SIGCHI

EICS '24: The 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems

June 24 - 28, 2024

Cagliari, Italy

Acceptance Rates

Overall Acceptance Rate 73 of 299 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
258
Total Downloads

Downloads (Last 12 months)258
Downloads (Last 6 weeks)46

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Küntzer LSchwab SSpaderna HRock G(2024)Measuring User Experience of Older Adults during Virtual Reality Exergaming2024 16th International Conference on Quality of Multimedia Experience (QoMEX)10.1109/QoMEX61742.2024.10598263(153-159)Online publication date: 18-Jun-2024
https://doi.org/10.1109/QoMEX61742.2024.10598263

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents