More Web Proxy on the site http://driver.im/

Article

Common metrics for human-robot interaction

Authors:

Aaron Steinfeld,

Michael GoodrichAuthors Info & Claims

HRI '06: Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction

Pages 33 - 40

https://doi.org/10.1145/1121241.1121249

Published: 02 March 2006 Publication History

Abstract

This paper describes an effort to identify common metrics for task-oriented human-robot interaction (HRI). We begin by discussing the need for a toolkit of HRI metrics. We then describe the framework of our work and identify important biasing factors that must be taken into consideration. Finally, we present suggested common metrics for standardization and a case study. Preparation of a larger, more detailed toolkit is in progress.

References

[1]

ANSI/AIAA. Guide to Human Performance Measurements, AIAA, Washington, DC, 1993.

[2]

Bajcsy, R. Active perception. Proc. IEEE, 76 (1988), 966--1005.

[3]

Bruce, A., Nourbakhsh, I., and Simmons, R., The role of expressiveness and attention in human-robot interaction. In Proc. AAAI Fall Symposium on Emotional and Intelligent II: The Tangled Knot of Social Cognition, (2001).

[4]

Casper, J., and Murphy, R. Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade Center. IEEE Transactions on Systems, Man and Cybernetics B, 33 (2003).

Digital Library

[5]

Cassimatis, N., Trafton, G., Schultz, A., Bugajska, M., and Adams, W., A task domain for combining and evaluating robotics and cognitive modeling techniques. In Proc. NIST Performance Metrics for Intelligent Systems Workshop, (2002).

[6]

Dautenhahn, K., and Werry, I., A quantitative technique for analysing robot-human interactions. In Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (2002).

[7]

Dautenhahn, K., Werry, I., Rae, J., Dickerson, P., Stribling, P., and Odgen, B. Robotic playmates: Analysing interactive competencies of children with autism playing with a mobile robot. In Dautenhahn, K., Bond, A., Canamero, L. and Edmonds, B. eds. Socially Intelligent Agents: Creating Relationships with Computers and Robots, Kluwer Academic Publishers, 2002.

Digital Library

[8]

Draper, J., and Blair, L., Workload, flow and telepresence during teleoperation. In Proc. IEEE International Conference on Robotics and Automation (ICRA), (1996), 1030--1035.

[9]

Draper, J.V., and Kaber, D.B. Human-Robot Interaction. In Mital, A., Ayoub, M.M., Kumar, S., Wang, M.-J. and Landau, K. eds. Industrial and Occupational Ergonomics: Users' Encyclopedia (Encyclopedia of Ergonomics), 1999.

[10]

Drury, J., Scholtz, J., and Yanco, H., Awareness in human-robot interactions. In Proc. IEEE International Conference on Systems, Man and Cybernetics, (2003).

Digital Library

[11]

Endsley, M. Measurement of situation awareness in dynamic systems. Human Factors, 37 (1995), 65--84.

[12]

Fong, T., Nourbakhsh, I., and Dautenhahn, K. A survey of socially interactive robots. Robotics and Autonomous Systems, 42 (2003).

[13]

Fong, T., and Thorpe, C. Vehicle teleoperation interfaces. Autonomous Robots, 11 (2001).

Digital Library

[14]

Fong, T., Thorpe, C., and Baur, C. Robot, asker of questions. Robotics and Autonomous Systems, 42 (2003).

[15]

Goetz, J., and Kiesler, S., Cooperation with a robotic assistant. In Proc. Computer-Human Interaction, (2002).

Digital Library

[16]

Goldberg, K., and Siegwart, R. Beyond Webcams: Introduction to Online Robots. MIT Press, Cambridge, MA, 2002.

Digital Library

[17]

Goodrich, M., and Olsen, D., Seven principles of efficient human robot interaction. In Proc. IEEE International Conference on Systems, Man and Cybernetics, (2003), 3943--3948.

[18]

Granda, T., Kirkpatrick, M., Julien, T., and Peterson, L., The evolutionary role of humans in the human-robot system. In Proc. Human Factors Society 34th Annual Meeting, (1990), 664--668.

[19]

Gutwin, C., Effects of network delay on group work in shared workspaces. In Proc. European Conferences on Computer Supported Work, (2001).

Digital Library

[20]

Hart, S., and Staveland, L. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In Hancock, P. and Meshkati, N. eds. Human Mental Workload, North-Holland Elsevier Science, 1988.

[21]

Huang, H., and Messina, E., Toward a generic model for the autonomy levels for unmanned systems. In Proc. NIST Performance Metrics for Intelligent Systems Workshop, (2003).

Digital Library

[22]

Hughes, S., and Lewis, M., Robotic camera control for remote exploration. In Proc. Conference on Human Factors in Computing Systems (CHI), (Vienna, Austria, 2004), 511--517.

Digital Library

[23]

Jacoff, A., Messina, E., and Evans, J., A reference test course for autonomous mobile robots. In Proc. SPIE-AeroSense Conference, (Orlando, FL, 2001).

Digital Library

[24]

Jacoff, A., Messina, E., and Evans, J., A standard test course for urban search and rescue robots. In Proc. NIST Performance Metrics for Intelligent Systems Workshop, (2000).

Digital Library

[25]

Kaber, D., Onal, E., and Endsley, M. Design of automation for telerobots and the effect on performance, operator situation awareness and subjective workload. Human Factors & Ergonomics in Manufacturing, 10 (2000), 409--430.

[26]

Kaber, D.B., and Chow, M.-Y., Human-robot interaction research and an approach to mobile-telerobot interface design. In Proc. XVth Triennial Congress of the International Ergonomics Association (CD-ROM), (Seoul, Korea, 2003).

[27]

Kaber, D.B., Riley, J., Zhou, R., and Draper, J.V., Effects of visual interface design, control interface type, and control latency on performance, telepresence, and workload in a teleoperation task. In Proc. XIVth Triennial Congress of the International Ergonomics Association and 44th Annual Meeting of the Human Factors and Ergonomics Society, (2000), Human Factors and Ergonomics Society, 503--506.

[28]

Lee, J., and Moray, N. Trust, self-confidence, and operators' adaptation to automation. International Journal of Human-Computer Studies, 40 (1994), 153--184.

Digital Library

[29]

Lee, J., and See, K. Trust in automation: designing for appropriate reliance. Human Factors, 46 (2004), 50--80.

[30]

Lewis, M., Wang, J., Manojlovich, J., Hughes, S., and Liu, X., Experiments with attitude: attitude displays for teleoperation. In Proc. IEEE International Conference on Systems, Man, and Cybernetics, (2003).

[31]

Macedo, J., Kaber, D., Endsley, M., Powanusorn, P., and Myung, S. The effects of automated compensation for incongruent axes on teleoperator performance. Human Factors, 40 (1999), 541--553.

[32]

MacKenzie, S., and Ware, C., Lag as a determinant of Human performance in interactive systems. In Proc. ACM Conference on Human Factors in Computing Systems (INTERCHI), (New York, NY, 1993), ACM SIGCHI, 488--493.

Digital Library

[33]

Newman, W., and Lamming, M. Interactive System Design. Addison-Wesley, Boston, 1995.

Digital Library

[34]

Nielsen, C.W., Ricks, B., Goodrich, M.A., Bruemmer, D., Few, D., and Walton, M., Snapshots for semantic maps. In Proc. IEEE International Conference on Systems, Man and Cybernetics, (The Hague, The Netherlands, 2004).

[35]

Olsen, D., and Goodrich, M., Metrics for evaluating human-robot interactions. In Proc. NIST Performance Metrics for Intelligent Systems Workshop, (2003).

[36]

Parasuraman, R. Human use and abuse of automation. In Mouloua, M. and Koonce, J. eds. Human-Automation Interaction, Lawrence Erlbaum Associates, 1997.

[37]

Parasuraman, R., Sheridan, T., and Wickens, C. A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man and Cybernetics B, 30 (2000), 286--297.

Digital Library

[38]

Rantanen, E., and Nunes, A., Taxonomoies of measures in air traffic control research. In Proc. International Symposium on Aviation Psychology, (2003).

[39]

Ricks, B., Nielsen, C.W., and Goodrich, M.A., Ecological displays for robot interaction: A new perspective. In Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (Sendai, Japan, 2004).

[40]

Rodriguez, G., and Weisbin, C. A new method to evaluate human-robot system performance. Autonomous Robots, 14 (2003), 165--178.

Digital Library

[41]

Sanders, M., and McCormick, E. Human Factors in Engineering and Design. McGraw-Hill, New York, NY, 1993.

[42]

Scholtz, J., Theory and evaluation of human robot interactions. In Proc. Hawaii International Conference on System Science 36, (2003).

Digital Library

[43]

Scholtz, J., Antonishek, B., and Young, J., Evaluating human-robot interfaces: development of a situational awareness assessment methodology. In Proc. Hawaii International Conference on System Science 37, (2004).

Digital Library

[44]

Scholtz, J., Antonishek, B., and Young, J., Evaluation of operator interventions in autonomous off-road driving. In Proc. NIST Performance Metrics for Intelligent Systems Workshop, (2003).

[45]

Scholtz, J., Young, J., Drury, J., and Yanco, H., Evaluation of human-robot interaction awareness in search and rescue. In Proc. IEEE International Conference on Robotics and Automation (ICRA), (2004).

Digital Library

[46]

Schulte, J., Rosenberg, C., and Thrun, S., Spontaneous, short-term interaction with mobile robots in public places. In Proc. IEEE International Conference on Robotics and Automation (ICRA), (1999).

[47]

Sheiknainar, M., Kaber, D., and Chow, M.-Y. Control gain adaptation in virtual reality mediated human-telerobot interaction. Human Factors & Ergonomics in Manufacturing, 15 (2005), 259--274.

Digital Library

[48]

Sheridan, T. Telerobotics, Automation, and Human Supervisory Control. MIT Press, Cambridge, 1992.

Digital Library

[49]

Steinfeld, A., Interface lessons for fully and semi-autonomous mobile robots. In Proc. IEEE International Conference on Robotics and Automation (ICRA), (2004).

[50]

Tittle, J., Woods, D., Roesler, A., Howard, M., and Phillips, F., The role of 2-D and 3-D task performance in the design and use of visual displays. In Proc. Human Factors and Ergonomics Society's 46th Annual Meeting, (2002).

[51]

Verma, V. Anecdotes from Rover Field Operations, Unpublished, 2000.

[52]

Wilson, G. Applied use of cardiac and respiration measures: Practical considerations and precautions. Biological Psychology, 34 (1992).

[53]

Wilson, G. Real-time adaptive aiding using psychological operator state assessment. In Harris, D. ed. Engineering Psychology and Cognitive Ergonomics, Ashgate, Aldershot, UK, 2001.

Digital Library

[54]

Yanco, H., and Drury, J., A taxonomy for human-robot interaction. In Proc. AAAI Fall Symposium on Human-Robot Interaction, (2002), 111--119.

[55]

Yanco, H., Drury, J., and Scholtz, J. Beyond usability evaluation: analysis of human-robot interaction at a major robotics competition. Human-Computer Interaction, 19 (2004), 117--149.

Digital Library

Cited By

Ojetunde SRamnarain U(2025)Like objects or like subjects? Effects of student–robot interaction (SRI) and mathematical ability on students learning outcomesSmart Learning Environments10.1186/s40561-024-00345-212:1Online publication date: 20-Jan-2025
https://doi.org/10.1186/s40561-024-00345-2
Zhang GHansen JMinakata K(2025)An experimental study of gaze-controlled telepresence robotsBehaviour & Information Technology10.1080/0144929X.2025.2470382(1-19)Online publication date: 3-Mar-2025
https://doi.org/10.1080/0144929X.2025.2470382
Apraiz ALasa GMazmela MArana-Arexolaleiba NSerrano Muñoz AElguea ÍEtxabe A(2025)Evaluating the Effect of Speed and Acceleration on Human Factors during an Assembly Task in Human–Robot Interaction (HRI)International Journal of Social Robotics10.1007/s12369-025-01224-717:2(211-256)Online publication date: 19-Feb-2025
https://doi.org/10.1007/s12369-025-01224-7
Show More Cited By

Recommendations

Survey of metrics for human-robot interaction
HRI '13: Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction

This paper examines 29 papers that have proposed or applied metrics for human-robot interaction. The 42 metrics are categorized as to the object being directly measured: the human (7), the robot (6), or the system (29). Systems metrics are further ...
Autonomy and Common Ground in Human-Robot Interaction: A Field Study

In a two-year study of a collaborative human-robot system, researchers observed a science team in Pittsburgh and a robot in Chile.The system was part of a project intended to inform planetary exploration while studying a terrestrial desert. Over two ...
Metrics for Human-Robot Team Design: A Teamwork Perspective on Evaluation of Human-Robot Teams
Metrics for human-robot teaming should extend to teams consisting of multiple human and robotic agents, and to teams working in complex, dynamic work domains. This work proposes that to comprehensively analyze and evaluate multi-human, multi-robot teams, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

HRI '06: Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction

March 2006

376 pages

ISBN:1595932941

DOI:10.1145/1121241

General Chair:
Michael A. Goodrich
Brigham Young University USA
,
Program Chairs:
Alan C. Schultz
Naval Research Laboratory, USA
,
David J. Bruemmer
Idaho National Laboratory, USA

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 March 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

HRI06

Sponsor:

HRI06: International Conference on Human Robot Interaction

March 2 - 3, 2006

Utah, Salt Lake City, USA

Acceptance Rates

Overall Acceptance Rate 268 of 1,124 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

487
Total Citations
View Citations
10,240
Total Downloads

Downloads (Last 12 months)268
Downloads (Last 6 weeks)25

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ojetunde SRamnarain U(2025)Like objects or like subjects? Effects of student–robot interaction (SRI) and mathematical ability on students learning outcomesSmart Learning Environments10.1186/s40561-024-00345-212:1Online publication date: 20-Jan-2025
https://doi.org/10.1186/s40561-024-00345-2
Zhang GHansen JMinakata K(2025)An experimental study of gaze-controlled telepresence robotsBehaviour & Information Technology10.1080/0144929X.2025.2470382(1-19)Online publication date: 3-Mar-2025
https://doi.org/10.1080/0144929X.2025.2470382
Apraiz ALasa GMazmela MArana-Arexolaleiba NSerrano Muñoz AElguea ÍEtxabe A(2025)Evaluating the Effect of Speed and Acceleration on Human Factors during an Assembly Task in Human–Robot Interaction (HRI)International Journal of Social Robotics10.1007/s12369-025-01224-717:2(211-256)Online publication date: 19-Feb-2025
https://doi.org/10.1007/s12369-025-01224-7
Biswas BDooley ECoulthard ERoudaut A(2024)Autoethnography of Living with a Sleep RobotMultimodal Technologies and Interaction10.3390/mti80600538:6(53)Online publication date: 18-Jun-2024
https://doi.org/10.3390/mti8060053
Nagy LAbonyi JRuppert T(2024)Knowledge Graph-Based Framework to Support Human-Centered Collaborative Manufacturing in Industry 5.0Applied Sciences10.3390/app1408339814:8(3398)Online publication date: 17-Apr-2024
https://doi.org/10.3390/app14083398
Sharma UMedasetti UDeemyad TMashal MYadav V(2024)Mobile Robot for Security Applications in Remotely Operated Advanced ReactorsApplied Sciences10.3390/app1406255214:6(2552)Online publication date: 18-Mar-2024
https://doi.org/10.3390/app14062552
Liu RGuo YJin RZhang X(2024)A Review of Natural-Language-Instructed Robot Execution SystemsAI10.3390/ai50300485:3(948-989)Online publication date: 26-Jun-2024
https://doi.org/10.3390/ai5030048
Jensen ESankaranarayanan SHayes B(2024)Automated Assessment and Adaptive Multimodal Formative Feedback Improves Psychomotor Skills Training Outcomes in Quadrotor TeleoperationProceedings of the 12th International Conference on Human-Agent Interaction10.1145/3687272.3688322(185-194)Online publication date: 24-Nov-2024
https://dl.acm.org/doi/10.1145/3687272.3688322
Tsoi NSterneck RZhao XVázquez M(2024)Influence of Simulation and Interactivity on Human Perceptions of a Robot During Navigation TasksACM Transactions on Human-Robot Interaction10.1145/3675784Online publication date: 16-Jul-2024
https://doi.org/10.1145/3675784
van Den broek MMoeslund T(2024)What is Proactive Human-Robot Interaction? - A Review of a Progressive Field and Its DefinitionsACM Transactions on Human-Robot Interaction10.1145/365011713:4(1-30)Online publication date: 23-Apr-2024
https://dl.acm.org/doi/10.1145/3650117
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten