When Should I Lead or Follow: Understanding Initiative Levels in Human-AI Collaborative Gameplay
Pages 2037 - 2056
Abstract
Dynamics in Human-AI interaction should lead to more satisfying and engaging collaboration. Key open questions are how to design such interactions and the role personal goals and expectations play. We developed three AI partners of varying initiative (leader, follower, shifting) in a collaborative game called Geometry Friends. We conducted a within-subjects experiment with 60 participants to assess personal AI partner preference and performance satisfaction as well as perceived warmth and competence of AI partners. Results show that AI partners following human initiative are perceived as warmer and more collaborative. However, some participants preferred AI leaders for their independence and speed, despite being seen as less friendly. This suggests that assigning a leadership role to the AI partner may be suitable for time-sensitive scenarios. We identify design factors for developing collaborative AI agents with varying levels of initiative to create more effective human-AI teams that consider context and individual preference.
Supplemental Material
PDF File
video.zip - Teaser video and subtitles questionnaire.pdf - Study Questionnaire
- Download
- 1015.36 KB
ZIP File
video.zip - Teaser video and subtitles questionnaire.pdf - Study Questionnaire
- Download
- 2.83 MB
References
[1]
Kathleen Allen, Richard Bergin, and Kenneth Pickar. 2004. Exploring trust, group satisfaction, and performance in geographically dispersed and co-located university technology commercialization teams. In VentureWell. Proceedings of Open, the Annual Conference. National Collegiate Inventors & Innovators Alliance, 201.
[2]
Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.
[3]
Zahra Ashktorab, Casey Dugan, James Johnson, Qian Pan, Wei Zhang, Sadhana Kumaravel, and Murray Campbell. 2021. Effects of communication directionality and AI agent differences in human-AI interaction. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.
[4]
Zahra Ashktorab, Q. Vera Liao, Casey Dugan, James Johnson, Qian Pan, Wei Zhang, Sadhana Kumaravel, and Murray Campbell. 2020. Human-AI Collaboration in a Cooperative Game Setting: Measuring Social Perception and Outcomes. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 96 (oct 2020), 20 pages. https://doi.org/10.1145/3415167
[5]
Michael J Baker. 2015. Collaboration in collaborative learning. Interaction Studies 16, 3 (2015), 451–473.
[6]
Victoria Bellou. 2011. Do women followers prefer a different leadership style than men?The International Journal of Human Resource Management 22, 13 (2011), 2818–2833.
[7]
Kirsten Bergmann, Friederike Eyssel, and Stefan Kopp. 2012. A second chance to make a first impression? How appearance and nonverbal behavior affect perceived warmth and competence of virtual agents over time. In Intelligent Virtual Agents: 12th International Conference, IVA 2012, Santa Cruz, CA, USA, September, 12-14, 2012. Proceedings 12. Springer, 126–138.
[8]
Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health 11, 4 (2019), 589–597.
[9]
Ángel Alexander Cabrera, Adam Perer, and Jason I Hong. 2023. Improving human-AI collaboration with descriptions of AI behavior. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1–21.
[10]
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems 32 (2019).
[11]
Alejandro Catala, Fernando Garcia-Sanjuan, Patricia Pons, Javier Jaen, and Jose A Mocholi. 2012. Agoras: Towards Collaborative Game-Based Learning Experiences on Surfaces.International Association for Development of the Information Society (2012).
[12]
Megan Charity, Isha Dave, Ahmed Khalifa, and Julian Togelius. 2022. Baba is Y’all 2.0: Design and Investigation of a Collaborative Mixed-Initiative System. IEEE Transactions on Games (2022).
[13]
Lara Christoforakos, Alessio Gallucci, Tinatini Surmava-Große, Daniel Ullrich, and Sarah Diefenbach. 2021. Can robots earn our trust the same way humans do? A systematic exploration of competence, warmth, and anthropomorphism as determinants of trust development in HRI. Frontiers in Robotics and AI 8 (2021), 640444.
[14]
Shi Feng and Jordan Boyd-Graber. 2019. What can ai do for me? evaluating machine learning interpretations in cooperative play. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 229–239.
[15]
Matthew C Fontaine, Ya-Chuan Hsu, Yulun Zhang, Bryon Tjanaka, and Stefanos Nikolaidis. 2021. On the importance of environments in human-robot coordination. arXiv preprint arXiv:2106.10853 (2021).
[16]
Katy Ilonka Gero, Zahra Ashktorab, Casey Dugan, Qian Pan, James Johnson, Werner Geyer, Maria Ruiz, Sarah Miller, David R Millen, Murray Campbell, 2020. Mental models of AI agents in a cooperative game setting. In Proceedings of the 2020 chi conference on human factors in computing systems. 1–12.
[17]
Matthew Gombolay, Anna Bair, Cindy Huang, and Julie Shah. 2017. Computational design of mixed-initiative human–robot teaming that considers human factors: situational awareness, workload, and workflow preferences. The International journal of robotics research 36, 5-7 (2017), 597–617.
[18]
Fritz Heider and Marianne Simmel. 1944. An experimental study of apparent behavior. The American journal of psychology 57, 2 (1944), 243–259.
[19]
Eric Horvitz. 1999. Uncertainty, action, and interaction: In pursuit of mixed-initiative computing. IEEE Intelligent Systems 14, 5 (1999), 17–20.
[20]
Matthew Johnson and Alonso Vera. 2019. No AI is an island: the case for teaming intelligence. AI magazine 40, 1 (2019), 16–28.
[21]
Hyun-Tae Kim, Du-Mim Yoon, and Kyung-Joong Kim. 2014. Solving Geometry Friends using Monte-Carlo tree search with directed graph representation. In 2014 IEEE Conference on Computational Intelligence and Games. IEEE, 1–2.
[22]
Florian Lehmann. 2023. Mixed-Initiative Interaction with Computational Generative Systems. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1–6.
[23]
Xin Lei and Pei-Luen Patrick Rau. 2021. Effect of relative status on responsibility attributions in human–robot collaboration: Mediating role of sense of responsibility and moderating role of power distance orientation. Computers in Human Behavior 122 (2021), 106820.
[24]
Pedro Faria Lopes, David Jardim, and Isabel Machado Alexandre. 2011. Math4Kids. In 6th Iberian Conference on Information Systems and Technologies (CISTI 2011). IEEE, 1–6.
[25]
Geoff Musick, Thomas A O’Neill, Beau G Schelble, Nathan J McNeese, and Jonn B Henke. 2021. What happens when humans believe their teammate is an AI? An investigation into humans teaming with autonomy. Computers in Human Behavior 122 (2021), 106852.
[26]
Stefanos Nikolaidis, David Hsu, and Siddhartha Srinivasa. 2017. Human-robot mutual adaptation in collaborative tasks: Models and experiments. The International Journal of Robotics Research 36, 5-7 (2017), 618–634.
[27]
Rui Prada, Phil Lopes, João Catarino, Joao Quitério, and Francisco S Melo. 2015. The geometry friends game AI competition. In 2015 IEEE conference on computational intelligence and games (CIG). IEEE, 431–438.
[28]
João Quitério, Rui Prada, and Francisco S Melo. 2015. A reinforcement learning approach for the circle agent of geometry friends. In 2015 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, 423–430.
[29]
José Bernardo Guimarães Rocha. 2009. Geometry friends. Ph. D. Dissertation. Master’s thesis, IST, University of Lisbon.
[30]
Jeremy Roschelle, Stephanie D Teasley, 1995. The construction of shared knowledge in collaborative problem solving. In Computer-supported collaborative learning, Vol. 128. 69–197.
[31]
Ana Salta, Rui Prada, and Francisco Melo. 2019. Solving motion and action planning for a cooperative agent problem using geometry friends. In EPIA Conference on Artificial Intelligence. Springer, 86–97.
[32]
Ana Salta, Rui Prada, and Francisco S Melo. 2020. A Game AI Competition to foster Collaborative AI research and development. IEEE Transactions on Games 13, 4 (2020), 398–409.
[33]
Vida Scarpello and John P Campbell. 1983. Job satisfaction: Are all the parts there?Personnel psychology 36, 3 (1983), 577–600.
[34]
Beau G Schelble, Christopher Flathmann, Nathan J McNeese, Guo Freeman, and Rohit Mallick. 2022. Let’s think together! Assessing shared mental models, performance, and trust in human-agent teams. Proceedings of the ACM on Human-Computer Interaction 6, GROUP (2022), 1–29.
[35]
Astrid Schepman and Paul Rodway. 2020. Initial validation of the general attitudes towards Artificial Intelligence Scale. Computers in human behavior reports 1 (2020), 100014.
[36]
Rui Soares, Francisco Leal, Rui Prada, and Francisco S Melo. 2016. Rapidly-Exploring Random Tree approach for Geometry Friends. In DiGRA/FDG.
[37]
Konrad Sowa, Aleksandra Przegalinska, and Leon Ciechanowski. 2021. Cobots in knowledge work: Human–AI collaboration in managerial professions. Journal of Business Research 125 (2021), 135–142.
[38]
Emma M Van Zoelen, Emilia I Barakova, and Matthias Rauterberg. 2020. Adaptive leader-follower behavior in human-robot collaboration. In 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, 1259–1265.
[39]
Dakuo Wang, Elizabeth Churchill, Pattie Maes, Xiangmin Fan, Ben Shneiderman, Yuanchun Shi, and Qianying Wang. 2020. From human-human collaboration to Human-AI collaboration: Designing AI systems that can work together with people. In Extended abstracts of the 2020 CHI conference on human factors in computing systems. 1–6.
[40]
Isaac Wang and Jaime Ruiz. 2021. Examining the use of nonverbal communication in virtual agents. International Journal of Human–Computer Interaction 37, 17 (2021), 1648–1673.
[41]
John P Wanous, Arnon E Reichers, and Michael J Hudy. 1997. Overall job satisfaction: how good are single-item measures?Journal of applied Psychology 82, 2 (1997), 247.
[42]
Rina R Wehbe, Edward Lank, and Lennart E Nacke. 2017. Left them 4 dead: Perception of humans versus non-player character teammates in cooperative gameplay. In Proceedings of the 2017 conference on designing interactive systems. 403–415.
[43]
Monika Westphal, Michael Vössing, Gerhard Satzger, Galit B Yom-Tov, and Anat Rafaeli. 2023. Decision control and explanations in human-AI collaboration: Improving user perceptions and compliance. Computers in Human Behavior 144 (2023), 107714.
[44]
H James Wilson and Paul R Daugherty. 2018. Collaborative intelligence: Humans and AI are joining forces. Harvard Business Review 96, 4 (2018), 114–123.
[45]
Rui Zhang, Nathan J McNeese, Guo Freeman, and Geoff Musick. 2021. " An Ideal Human" Expectations of AI Teammates in Human-AI Teaming. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1–25.
Index Terms
- When Should I Lead or Follow: Understanding Initiative Levels in Human-AI Collaborative Gameplay
Recommendations
From Human-Human Collaboration to Human-AI Collaboration: Designing AI Systems That Can Work Together with People
CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing SystemsArtificial Intelligent (AI) and Machine Learning (ML) algorithms are coming out of research labs into the real-world applications, and recent research has focused a lot on Human-AI Interaction (HAI) and Explainable AI (XAI). However, Interaction is not ...
"An Ideal Human": Expectations of AI Teammates in Human-AI Teaming
CSCWDriven by state-of-the-art AI technologies, human-AI collaboration has become an important area in computer-supported teamwork research. While human-AI collaboration has been investigated in various domains, more research is needed to explore human ...
Integrating AI in Human-Human Collaborative Ideation
CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing SystemsPeople can generate more innovative ideas when they collaborate with one another, collectively exploring ideas and exchanging viewpoints. Advancements in artificial intelligence have opened up new opportunities in people’s creative activities where ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2024
3616 pages
ISBN:9798400705830
DOI:10.1145/3643834
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- Fundação para a Ciência e a Tecnologia
- Horizon 2020 Framework Programme
- Fundação para a Ciência e a Tecnologia
Conference
DIS '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,158 of 4,684 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 379Total Downloads
- Downloads (Last 12 months)379
- Downloads (Last 6 weeks)85
Reflects downloads up to 15 Jan 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in