More Web Proxy on the site http://driver.im/

short-paper

Investigating the Validity of Online Robot Evaluations: Comparison of Findings from an One-Sample Online and Laboratory Study

Authors:

Franziska Babel,

Johannes Kraus,

Hannah Asenbauer,

Martin BaumannAuthors Info & Claims

HRI '21 Companion: Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction

Pages 116 - 120

https://doi.org/10.1145/3434074.3447141

Published: 08 March 2021 Publication History

Abstract

As the number of online studies in the field of human-robot interaction (HRI) increases, the comparability of the results of online studies with those of laboratory experiments needs further investigation. In this one-sample study, 29 participants experienced three different commercially-available service robots first in an online session and then in a lab experiment and evaluated the robots regarding their trust, fear and intention to use the robot. Furthermore, several robot characteristics were evaluated (e.g. humanness, uncanniness). Overall, study results indicate high comparability of findings from online and lab experiments for trust, fear and robot characteristics like humanness and uncanniness. Same relative differences between the robots were found for both presentation methods except for intention to use and robot reliability. This preliminary study provides insights into online study validity and makes recommendations for future research.

References

[1]

Christoph Bartneck, Dana Kulić, Elizabeth Croft, and Susana Zoghbi. 2009. Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International Journal of Social Robotics, Vol. 1, 1 (2009), 71--81.

[2]

Dimitrina S. Dimitrova, Vladimir K. Kaishev, and Senren Tan. 2020. Computing the kolmogorov-smirnov distribution when the underlying cdf is purely discrete, mixed, or continuous. J. Stat. Softw., Vol. 95, 1 (2020), 1--42. https://doi.org/10.18637/jss.v095.i10

[3]

David Feil-Seifer, Kerstin S Haring, Silvia Rossi, Alan R Wagner, and Tom Williams. 2020. Where to Next? The Impact of COVID-19 on Human-Robot Interaction Research. ACM Trans. Human-Robot Interact., Vol. 10, 1 (2020), 1--7. https://doi.org/10.1145/3405450

Digital Library

[4]

Li Gong. 2008. How social is social responses to computers? The function of the degree of anthropomorphism in computer representations. Comput. Human Behav., Vol. 24, 4 (2008), 1494--1509. https://doi.org/10.1016/j.chb.2007.05.007

Digital Library

[5]

Chin Chang Ho and Karl F. MacDorman. 2017. Measuring the Uncanny Valley Effect: Refinements to Indices for Perceived Humanness, Attractiveness, and Eeriness. Int. J. Soc. Robot., Vol. 9, 1 (jan 2017), 129--139. https://doi.org/10.1007/s12369-016-0380--9

[6]

Philipp Hock, Franziska Babel, Enrico Rukzio, Kristin Muehl, and Martin Baumann. 2019. Online experiments as a supplement of automated driving simulator studies: A methodological insight. In Adjun. Proc. - 11th Int. ACM Conf. Automot. User Interfaces Interact. Veh. Appl. AutomotiveUI 2019. Association for Computing Machinery, Inc, 282--286. https://doi.org/10.1145/3349263.3351334

Digital Library

[7]

Philipp Hock, Johannes Kraus, Franziska Babel, Marcel Walch, Enrico Rukzio, and Martin Baumann. 2018. How to design valid simulator studies for investigating user experience in automated driving - Review and hands-on considerations. In Proc. - 10th Int. ACM Conf. Automot. User Interfaces Interact. Veh. Appl. AutomotiveUI 2018. 105--117. https://doi.org/10.1145/3239060.3239066

Digital Library

[8]

Shanee Honig and Tal Oron-Gilad. 2020. Comparing laboratory user studies and video-enhanced web surveys for eliciting user gestures in human-robot interactions. In ACM/IEEE Int. Conf. Human-Robot Interact. 248--250. https://doi.org/10.1145/3371382.3378325

Digital Library

[9]

Jiun-Yin Jian, Ann M Bisantz, and Colin G Drury. 2000. Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics, Vol. 4, 1 (2000), 53--71.

[10]

Cory David Kidd. 2000. Sociable Robots: The Role of Presence and Task in Human-Robot Interaction. MSc Thesis. Master Thesis. https://dspace.mit.edu/handle/1721.1/61863

[11]

Johannes Maria Kraus. 2020. Psychological Processes in the Formation and Calibration of Trust in Automation. Dissertation. Dissertation Ulm University. https://doi.org/10.18725/OPARU-32583

[12]

Johannes Maria Kraus, Franziska Babel, and Martin Baumann. 2019. Initial reactions to an autonomous service robot in public space: an interview study. In AOW Symp. Eng. Psychol. 3.

[13]

Jamy Li. 2015. The benefit of being physically present: A survey of experimental works comparing copresent robots, telepresent robots and virtual agents. Int. J. Hum. Comput. Stud., Vol. 77 (2015), 23--37. https://doi.org/10.1016/j.ijhcs.2015.01.001

Digital Library

[14]

Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In 11th australasian conference on information systems, Vol. 53. Citeseer, 6--8.

[15]

Justinas Mivs eikis, Pietro Caroni, Patricia Duchamp, Alina Gasser, Rastislav Marko, Nelija Mivs eikiene, Frederik Zwilling, Charles de Castelbajac, Lucas Eicher, Michael Frü h, and Hansruedi Frü h. 2020. Lio - A personal robot assistant for human-robot interaction and care applications. arXiv, Vol. 5, 4 (2020), 5339--5346.

[16]

Jordi Pages, Luca Marchionni, and Francesco Ferro. 2016. TIAGo: the modular robot that adapts to different research needs. In Int. Work. Robot Modul. 3--6. https://clawar.org/wp-content/uploads/2016/10/P2.pdf

[17]

Michael Steele and Janet Chaseling. 2006. Powers of discrete goodness-of-fit test statistics for a uniform null against a selection of alternative distributions. Commun. Stat. Simul. Comput., Vol. 35, 4 (2006), 1067--1075. https://doi.org/10.1080/03610910600880666

[18]

Viswanath Venkatesh and Fred D Davis. 2000. A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management science, Vol. 46, 2 (2000), 186--204.

[19]

Sebastian Wallot and Giuseppe Leonardi. 2018. Deriving inferential statistics from recurrence plots: A recurrence-based test of differences between sample distributions and its comparison to the two-sample Kolmogorov-Smirnov test. Chaos, Vol. 28, 085712 (2018). https://doi.org/10.1063/1.5024915

[20]

Sarah N. Woods, Michael L. Walters, Kheng Lee Koay, and Kerstin Dautenhahn. 2006. Methodological issues in HRI: A comparison of live and video-based methods in robot to human approach direction trials. In Proc. - IEEE Int. Work. Robot Hum. Interact. Commun. 51--58. https://doi.org/10.1109/ROMAN.2006.314394

Cited By

KUBOTA TOGAWA K(2024)Influence of the Behavior of a Para-Operated Social Robot on the Impression Interlocutors Have of the Operator: an Online Study近接操作型対話ロボットの振る舞いが操作者の対人印象に与える影響のオンライン調査Journal of Japan Society for Fuzzy Theory and Intelligent Informatics10.3156/jsoft.36.4_69536:4(695-702)Online publication date: 15-Nov-2024
https://doi.org/10.3156/jsoft.36.4_695
Babel FWelsch RMiller LHock PThellman SZiemke T(2024)A Robot Jumping the Queue: Expectations About Politeness and Power During Conflicts in Everyday Human-Robot EncountersProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642082(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642082
Hauser EChan YModak SBiswas JHart J(2024)Vid2Real HRI: Align video-based HRI study designs with real-world settings2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN)10.1109/RO-MAN60168.2024.10731413(542-548)Online publication date: 26-Aug-2024
https://doi.org/10.1109/RO-MAN60168.2024.10731413
Show More Cited By

Index Terms

Investigating the Validity of Online Robot Evaluations: Comparison of Findings from an One-Sample Online and Laboratory Study
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies

Recommendations

Understanding Egyptian Consumers' Intentions in Online Shopping

The purpose of this article is to investigate the factors that impact on Egyptian consumers' attitudes and intentions to use online shopping by integrating the technology acceptance models of Davis, and Fishbein and Ajzen's theory of reasoned action. In ...
Determinants of repurchase intention in online group-buying

A theoretical model integrating the D&M model and trust literature was proposed.The model was tested using data collected customers of a group-buying website.Increasing satisfaction and perceived quality will facilitate repurchase intention.Increasing ...
Distrust and trust in B2C e-commerce: do they differ?
ICEC '06: Proceedings of the 8th international conference on Electronic commerce: The new e-commerce: innovations for conquering current barriers, obstacles and limitations to conducting successful business on the internet

Researchers have not studied e-commerce <u>distrust</u> as much as e-commerce <u>trust</u>. This study examines whether trust and distrust are distinct concepts. If trust and distrust are the same, lack of distrust research matters little. But if they ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

HRI '21 Companion: Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction

March 2021

756 pages

ISBN:9781450382908

DOI:10.1145/3434074

General Chairs:
Cindy Bethel
Mississippi State University, USA
,
Ana Paiva
INESC-ID, IST, University of Lisbon, Portugal & Radcliffe Institute for Advanced Study, Harvard University, USA
,
Program Chairs:
Elizabeth Broadbent
University of Auckland, New Zealand
,
David Feil-Seifer
University of Nevada Reno, USA
,
Daniel Szafir
University of Colorado Boulder, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 March 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

German Ministry of Education and Research

Conference

HRI '21

Sponsor:

HRI '21: ACM/IEEE International Conference on Human-Robot Interaction

March 8 - 11, 2021

CO, Boulder, USA

Acceptance Rates

Overall Acceptance Rate 192 of 519 submissions, 37%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
278
Total Downloads

Downloads (Last 12 months)61
Downloads (Last 6 weeks)7

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

KUBOTA TOGAWA K(2024)Influence of the Behavior of a Para-Operated Social Robot on the Impression Interlocutors Have of the Operator: an Online Study近接操作型対話ロボットの振る舞いが操作者の対人印象に与える影響のオンライン調査Journal of Japan Society for Fuzzy Theory and Intelligent Informatics10.3156/jsoft.36.4_69536:4(695-702)Online publication date: 15-Nov-2024
https://doi.org/10.3156/jsoft.36.4_695
Babel FWelsch RMiller LHock PThellman SZiemke T(2024)A Robot Jumping the Queue: Expectations About Politeness and Power During Conflicts in Everyday Human-Robot EncountersProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642082(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642082
Hauser EChan YModak SBiswas JHart J(2024)Vid2Real HRI: Align video-based HRI study designs with real-world settings2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN)10.1109/RO-MAN60168.2024.10731413(542-548)Online publication date: 26-Aug-2024
https://doi.org/10.1109/RO-MAN60168.2024.10731413
Scholz DKraus JMiller L(2024)Measuring the Propensity to Trust in Automated Technology: Examining Similarities to Dispositional Trust in Other Humans and Validation of the PTT-A ScaleInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2307691(1-24)Online publication date: 14-Feb-2024
https://doi.org/10.1080/10447318.2024.2307691
Benk MKerstan Svon Wangenheim FFerrario A(2024)Twenty-four years of empirical research on trust in AI: a bibliometric review of trends, overlooked issues, and future directionsAI & SOCIETY10.1007/s00146-024-02059-yOnline publication date: 2-Oct-2024
https://doi.org/10.1007/s00146-024-02059-y
Riedelbauch DHöllerich NHenrich D(2023)Benchmarking Teamwork of Humans and Cobots—An Overview of Metrics, Strategies, and TasksIEEE Access10.1109/ACCESS.2023.327160211(43648-43674)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3271602
Tojib DAbdi ETian LRigby LMeads JPrasad T(2023)What’s Best for Customers: Empathetic Versus Solution-Oriented Service RobotsInternational Journal of Social Robotics10.1007/s12369-023-00970-w15:5(731-743)Online publication date: 9-Feb-2023
https://doi.org/10.1007/s12369-023-00970-w
Babel FHock PKraus JBaumann MSakamoto DWeiss AHiatt LShiomi M(2022)Human-Robot Conflict Resolution at an Elevator The Effect of Robot Type, Request Politeness and ModalityProceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3523760.3523857(693-697)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.5555/3523760.3523857
Allan DVonasch ABartneck CSakamoto DWeiss AHiatt LShiomi M(2022)Better than UsProceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3523760.3523792(215-224)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.5555/3523760.3523792
Kunold LOnnasch L(2022)A Framework to Study and Design Communication with Social RobotsRobotics10.3390/robotics1106012911:6(129)Online publication date: 15-Nov-2022
https://doi.org/10.3390/robotics11060129
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents