Work in Progress

Online experiments as a supplement of automated driving simulator studies: a methodological insight

Authors:

Martin BaumannAuthors Info & Claims

AutomotiveUI '19: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings

Pages 282 - 286

https://doi.org/10.1145/3349263.3351334

Published: 21 September 2019 Publication History

Get Access

Abstract

Online experiments offer several advantages compared to lab studies. We propose that reaction times in takeover requests in automated driving research can be evaluated using online experiments. Furthermore, we assume that online studies can complement simulator studies to some extent. However, online studies have several pitfalls (e.g. an uncontrolled environment). Affecting factors can occur in hardware, software, and human behavior. We provide a detailed methodology and technology section to reduce possible confounding factors and to increase the validity of automated driving online studies. Furthermore, to test the effect of the proposed methodology, we conducted a user study with a takeover scenario as online (n = 42) and simulator (n = 17) experiment. First results indicate that reaction times did not differ between the simulator and online experiment. Thus, we conclude that with some methodical refinements and practices, similar results can be achieved in online studies compared to simulator studies.

References

[1]

Tab Atkins and fantasai. 2019. CSS Values and Units Module Level 4. (2019). https://www.w3.org/TR/css-values-4/#absolute-lengths

Google Scholar

[2]

SAE On-Road Automated Vehicle Standards Committee and others. 2016. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. SAE Standard J3016 (2016), 01--30.

Google Scholar

[3]

Donald L Fisher, John D Lee, Matthew Rizzo, and Jeffrey Caird. 2011. Handbook of driving simulation for engineering, medicine, and psychology. CRC Press.

Google Scholar

[4]

Paul M Fitts and Michael I Posner. 1967. Human performance. (1967).

Google Scholar

[5]

Vincent Hardy and Sylvain Galineau. 2019. Understanding pixels and other CSS units. (2019). https://webplatform.github.io/docs/tutorials/understanding-css-units/

Google Scholar

[6]

Morten Hertzum and Kasper Hornbæk. 2010. How age affects pointing with mouse and touchpad: A comparison of young, adult, and elderly users. Intl. Journal of Human-Computer Interaction 26, 7 (2010), 703--734.

Crossref

Google Scholar

[7]

Philipp Hock, Johannes Kraus, Franziska Babel, Marcel Walch, Enrico Rukzio, and Martin Baumann. 2018. How to Design Valid Simulator Studies for Investigating User Experience in Automated Driving: Review and Hands-On Considerations. In Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '18). ACM, New York, NY, USA, 105--117.

Digital Library

Google Scholar

[8]

Philipp Hock, Johannes Kraus, Marcel Walch, Nina Lang, and Martin Baumann. 2016. Elaborating feedback strategies for maintaining automation in highly automated driving. In Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 105--112.

Digital Library

Google Scholar

[9]

Robert Kraut, Judith Olson, Mahzarin Banaji, Amy Bruckman, Jeffrey Cohen, and Mick Couper. 2004. Psychological research online: report of Board of Scientific Affairs' Advisory Group on the Conduct of Research on the Internet. American psychologist 59, 2 (2004), 105.

Google Scholar

[10]

Xah Lee. 2015. CSS: What's a CSS Pixel? What's Reference Pixel? http://xahlee.info/js/web_design_screen_density.html. (2015). Accessed: 2019-06-17.

Google Scholar

[11]

Mozilla Inc. 2019a. DevicePixelRatio. (2019). https://developer.mozilla.org/en-US/docs/Web/API/Window/devicePixelRatio

Google Scholar

[12]

Mozilla Inc. 2019b. Using the viewport meta tag to control layout on mobile browsers. (2019). https://developer.mozilla.org/en-US/docs/Mozilla/Mobile/Viewport_meta_tag

Google Scholar

[13]

Richard R Plant, Nick Hammond, and Tom Whitehouse. 2003. How choice of mouse may affect response timing in psychological studies. Behavior Research Methods, Instruments, & Computers 35, 2 (2003), 276--284.

Crossref

Google Scholar

[14]

Ulf-Dietrich Reips. 2000. The Web experiment method: Advantages, disadvantages, and solutions. In Psychological experiments on the Internet. Elsevier, 89--117.

Google Scholar

Cited By

View all

Babel FHock PKraus JBaumann MSakamoto DWeiss AHiatt LShiomi M(2022)It Will Not Take Long! Longitudinal Effects of Robot Conflict Resolution Strategies on Compliance, Acceptance and TrustProceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3523760.3523793(225-235)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.5555/3523760.3523793
Babel FHock PKraus JBaumann M(2022)It Will Not Take Long! Longitudinal Effects of Robot Conflict Resolution Strategies on Compliance, Acceptance and Trust2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)10.1109/HRI53351.2022.9889492(225-235)Online publication date: 7-Mar-2022
https://doi.org/10.1109/HRI53351.2022.9889492
Babel FKraus JHock PAsenbauer HBaumann MBethel CPaiva ABroadbent EFeil-Seifer DSzafir D(2021)Investigating the Validity of Online Robot Evaluations: Comparison of Findings from an One-Sample Online and Laboratory StudyCompanion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3434074.3447141(116-120)Online publication date: 8-Mar-2021
https://dl.acm.org/doi/10.1145/3434074.3447141
Show More Cited By

Index Terms

Online experiments as a supplement of automated driving simulator studies: a methodological insight
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI

Recommendations

How to Design Valid Simulator Studies for Investigating User Experience in Automated Driving: Review and Hands-On Considerations
AutomotiveUI '18: Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

Simulator studies have been conducted in the automotive domain since the 1960s. Recently, automated driving studies have become more popular as real-world automated cars start to emerge but at this time not all levels of automation can be realized. A ...
Designing and deploying online field experiments
WWW '14: Proceedings of the 23rd international conference on World wide web

Online experiments are widely used to compare specific design alternatives, but they can also be used to produce generalizable knowledge and inform strategic decision making. Doing so often requires sophisticated experimental designs, iterative ...
Online controlled experiments: introduction, learnings, and humbling statistics
RecSys '12: Proceedings of the sixth ACM conference on Recommender systems

The web provides an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using controlled experiments (e.g., A/B tests and their generalizations). Whether for front-end user-interface changes, or backend ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

AutomotiveUI '19: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings

September 2019

524 pages

ISBN:9781450369206

DOI:10.1145/3349263

General Chairs:
Christian P. Janssen
Utrecht University, The Netherlands
,
Stella F. Donker
Utrecht University, The Netherlands
,
Program Chairs:
Lewis L. Chuang
LMU Munich, Germany
,
Wendy Ju
Cornell Tech

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 September 2019

Check for updates

Author Tags

Qualifiers

Work in progress

Funding Sources

Carl-Zeiss-Scholarship

Conference

AutomotiveUI '19

Sponsor:

SIGCHI

AutomotiveUI '19: 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

September 21 - 25, 2019

Utrecht, Netherlands

Acceptance Rates

Overall Acceptance Rate 248 of 566 submissions, 44%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
149
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Babel FHock PKraus JBaumann MSakamoto DWeiss AHiatt LShiomi M(2022)It Will Not Take Long! Longitudinal Effects of Robot Conflict Resolution Strategies on Compliance, Acceptance and TrustProceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3523760.3523793(225-235)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.5555/3523760.3523793
Babel FHock PKraus JBaumann M(2022)It Will Not Take Long! Longitudinal Effects of Robot Conflict Resolution Strategies on Compliance, Acceptance and Trust2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)10.1109/HRI53351.2022.9889492(225-235)Online publication date: 7-Mar-2022
https://doi.org/10.1109/HRI53351.2022.9889492
Babel FKraus JHock PAsenbauer HBaumann MBethel CPaiva ABroadbent EFeil-Seifer DSzafir D(2021)Investigating the Validity of Online Robot Evaluations: Comparison of Findings from an One-Sample Online and Laboratory StudyCompanion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3434074.3447141(116-120)Online publication date: 8-Mar-2021
https://dl.acm.org/doi/10.1145/3434074.3447141
Lanzer MBabel FYan FZhang BYou FWang JBaumann M(2020)Designing Communication Strategies of Autonomous Vehicles with Pedestrians: An Intercultural Study12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3409120.3410653(122-131)Online publication date: 21-Sep-2020
https://dl.acm.org/doi/10.1145/3409120.3410653

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

How to Design Valid Simulator Studies for Investigating User Experience in Automated Driving: Review and Hands-On Considerations

Designing and deploying online field experiments

Online controlled experiments: introduction, learnings, and humbling statistics