More Web Proxy on the site http://driver.im/

research-article

Public Access

Evaluating singleplayer and multiplayer in human computation games

Authors:

Matthew Guzdial,

Mark O. RiedlAuthors Info & Claims

FDG '17: Proceedings of the 12th International Conference on the Foundations of Digital Games

Article No.: 34, Pages 1 - 10

https://doi.org/10.1145/3102071.3102077

Published: 14 August 2017 Publication History

Abstract

Human computation games (HCGs) can provide novel solutions to intractable computational problems, help enable scientific breakthroughs, and provide datasets for artificial intelligence. However, our knowledge about how to design and deploy HCGs that appeal to players and solve problems effectively is incomplete. We present an investigatory HCG based on Super Mario Bros. We used this game in a human subjects study to investigate how different social conditions---singleplayer and multiplayer---and scoring mechanics---collaborative and competitive---affect players' subjective experiences, accuracy at the task, and the completion rate. In doing so, we demonstrate a novel design approach for HCGs, and discuss the benefits and tradeoffs of these mechanics in HCG design.

References

[1]

Jonathan Barone, Colin Bayer, Rowan Copley, Nova Barlow, Matthew Burns, Sundipta Rao, Georg Seelig, Zoran Popović, Seth Cooper, and Nanocrafter Players. 2015. Nanocrafter: Design and Evaluation of a DNA Nanotechnology Game. In Proceedings of the 10th International Conference on the Foundations of Digital Games.

[2]

Luke Barrington, Douglas Turnbull, and Gert Lanckriet. 2012. Game-powered machine learning. Proceedings of the National Academy of Sciences 109, 17 (2012), 6411--6416.

[3]

Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popović, and others. 2010. Predicting protein structures with a multiplayer online game. Nature 466, 7307 (2010), 756--760.

[4]

Seth Cooper, Adrien Treuille, Janos Barbero, Andrew Leaver-Fay, Kathleen Tuite, Firas Khatib, Alex Cho Snyder, Michael Beenen, David Salesin, David Baker, Zoran Popović, and Foldit Players. 2010. The Challenge of Designing Scientific Discovery Games. In 5th International Conference on the Foundations of Digital Games.

Digital Library

[5]

Werner Dietl, Stephanie Dietzel, Michael D. Ernst, Nathaniel Mote, Brian Walker, Seth Cooper, Timothy Pavlik, and Zoran Popović. 2012. Verification Games: Making Verification Fun. In 14th Workshop on Formal Techniques for Java-like Programs.

Digital Library

[6]

Julie S. Downs, Mandy B. Holbrook, Steve Sheng, and Lorrie Faith Cranor. 2010. Are Your Participants Gaming the System?: Screening Mechanical Turk Workers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 2399--2402.

Digital Library

[7]

Katharina Emmerich and Maic Masuch. 2013. Helping friends or fighting foes: The influence of collaboration and competition on player experience. In Proceedings of the 8th International Conference on the Foundations of Digital Games.

[8]

Dion Hoe-Lian Goh, Rebecca P. Ang, Chei Sian Lee, and Alton Y. K. Chua. 2011. Fight or Unite: Investigating Game Genres for Image Tagging. J. Am. Soc. Inf. Sci. Technol. 62, 7 (July 2011), 1311--1324.

Digital Library

[9]

Matthew Guzdial and Mark Riedl. 2016. Conceptually Blended Levels in a Unity Engine. In Playable Experiences at Twelfth Artificial Intelligence and Interactive Digital Entertainment Conference.

[10]

Chien-Ju Ho, Tao-Hsuan Chang, Jong-Chuan Lee, Jane Yung-jen Hsu, and Kuan-Ta Chen. 2010. KissKissBan: A Competitive Human Computation Game for Image Annotation. SIGKDD Explor. Newsl. 12, 1 (Nov. 2010), 21--24.

Digital Library

[11]

Peter Jamieson, Lindsay Grace, and Jack Hall. 2012. Research Directions for Pushing Harnessing Human Computation to Mainstream Video Games. In Meaningful Play.

[12]

Fengfeng Ke and Barbara L. Grabowski. 2007. Gameplaying for maths learning: cooperative or not? British Journal of Educational Technology 38 (2007), 249--259.

[13]

Markus Krause and René Kizilcec. 2015. To Play or Not to Play: Interactions between Response Quality and Task Complexity in Games and Paid Crowdsourcing. In Third AAAI Conference on Human Computation and Crowdsourcing.

[14]

Markus Krause and Jan Smeddinck. 2011. Human computation games: A survey. In 2011 19th European Signal Processing Conference. IEEE, 754--758.

[15]

Markus Krause, Aneta Takhtamysheva, Marion Wittstock, and Rainer Malaka. 2010. Frontiers of a Paradigm: Exploring Human Computation with Digital Games. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '10). ACM, New York, NY, USA, 22--25.

Digital Library

[16]

Chris J Lintott, Kevin Schawinski, Anže Slosar, Kate Land, Steven Bamford, Daniel Thomas, M Jordan Raddick, Robert C Nichol, Alex Szalay, Dan Andreescu, Phil Murray, and Jan Vandenberg. 2008. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society 389, 3 (2008), 1179--1189.

[17]

Heather Logas, Jim Whitehead, Michael Mateas, Richard Vallejos, Lauren Scott, Dan Shapiro, John Murray, Kate Compton, Joseph Osborn, Orlando Salvatore, Zhongpeng Lin, Huascar Sanchez, Michael Shavlovsky, Daniel Cetina, Shayne Clementi, and Chris Lewis. 2014. Software Verification Games: Designing Xylem, The Code of Plants. In Proceedings of the 9th International Conference on the Foundations of Digital Games.

[18]

Regan L. Mandryk and Kori M. Inkpen. 2004. Physiological Indicators for the Evaluation of Co-located Collaborative Play. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (CSCW '04). ACM, New York, NY, USA, 102--111.

Digital Library

[19]

Ei Pa Pa Pe-Than, Dion Hoe-Lian Goh, and Chei Sian Lee. 2013. A typology of human computation games: an analysis and a review of current games. Behaviour & Information Technology (2013).

[20]

Wei Peng and Gary Hsieh. 2012. The influence of competition, cooperation, and player relationship in a motor performance centered computer game. Computers in Human Behavior 28, 6 (2012), 2100--2106.

Digital Library

[21]

Mark Peplow. 2016. Citizen science lures gamers into Sweden's Human Protein Atlas. Nature Biotechnology 34, 5 (2016), 452--452.

[22]

Jan L Plass, Paul A Ofikeefe, Bruce D Homer, Jennifer Case, Elizabeth O Hayward, Murphy Stein, and Ken Perlin. 2013. The impact of individual, competitive, and collaborative mathematics game play on learning, performance, and motivation. Journal of Educational Psychology 105, 4 (2013), 1050.

[23]

Norsaremah Salleh, Emilia Mendes, and John Grundy. 2011. Empirical studies of pair programming for CS/SE teaching in higher education: A systematic literature review. IEEE Transactions on Software Engineering 37, 4 (2011), 509--525.

Digital Library

[24]

Magy Seif El-Nasr, Bardia Aghabeigi, David Milam, Mona Erfani, Beth Lameman, Hamid Maygoli, and Sang Mah. 2010. Understanding and Evaluating Cooperative Games. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 253--262.

Digital Library

[25]

Katharina Siorpaes and Martin Hepp. 2008. Games with a Purpose for the Semantic Web. IEEE Intelligent Systems 23, 3 (2008), 50--60.

Digital Library

[26]

Kristin Siu, Alexander Zook, and Mark O. Riedl. 2014. Collaboration versus Competition: Design and Evaluation of Mechanics for Games with a Purpose. In 9th International Conference on the Foundations of Digital Games.

[27]

Julian Togelius, Sergey Karakovskiy, and Noor Shaker. 2012. 2012 Mario AI Championship. http://www.marioai.org/. (2012).

[28]

Kathleen Tuite. 2014. GWAPs: Games with a Problem. In Proceedings of the 9th International Conference on the Foundations of Digital Games.

[29]

Kathleen Tuite, Noah Snavely, Dun-yu Hsiao, Nadine Tabing, and Zoran Popović. 2011. PhotoCity: Training Experts at Large-scale Image Acquisition Through a Competitive Game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 1383--1392.

Digital Library

[30]

Luis von Ahn and Laura Dabbish. 2004. Labeling Images with a Computer Game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). ACM, New York, NY, USA, 319--326.

Digital Library

[31]

Luis von Ahn and Laura Dabbish. 2008. Designing Games with a Purpose. Commun. ACM 51, 8 (2008), 58--67.

Digital Library

[32]

Rina R. Wehbe and Lennart E. Nacke. 2015. Towards Understanding the Importance of Co-Located Gameplay. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '15). ACM, New York, NY, USA, 733--738.

Digital Library

Cited By

Wang ZLiu JYannakakis G(2022)The Fun Facets of Mario: Multifaceted Experience-Driven PCG via Reinforcement LearningProceedings of the 17th International Conference on the Foundations of Digital Games10.1145/3555858.3563282(1-8)Online publication date: 5-Sep-2022
https://dl.acm.org/doi/10.1145/3555858.3563282
Paranthaman PSarkar ACooper S(2021)Applying Rapid Crowdsourced Playtesting to a Human Computation GameProceedings of the 16th International Conference on the Foundations of Digital Games10.1145/3472538.3472626(1-7)Online publication date: 3-Aug-2021
https://dl.acm.org/doi/10.1145/3472538.3472626
Matin AMaduro Mde Leon Pereira RTremblay-Savard O(2020)Effect of Timer, Top Score and Leaderboard on Performance and Motivation in a Human Computing GameProceedings of the 15th International Conference on the Foundations of Digital Games10.1145/3402942.3403000(1-10)Online publication date: 15-Sep-2020
https://dl.acm.org/doi/10.1145/3402942.3403000
Show More Cited By

Index Terms

Evaluating singleplayer and multiplayer in human computation games
1. Applied computing
  1. Computers in other domains
    1. Personal computers and PC applications
      1. Computer games
2. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
      1. Computer supported cooperative work

Recommendations

Reward Systems in Human Computation Games
CHI PLAY '16: Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play

Human computation games (HCGs) are games in which player interaction is used to solve problems intractable for computers. Most HCGs use simple reward mechanisms such as points or leaderboards, but in contrast, many mainstream games use more complex, and ...
A framework for exploring and evaluating mechanics in human computation games
FDG '17: Proceedings of the 12th International Conference on the Foundations of Digital Games

Human computation games (HCGs) are a crowdsourcing approach to solving computationally-intractable tasks using games. We outline a formal representation of the mechanics in HCGs, providing a structural breakdown to visualize, compare, and explore the ...
Evaluating GameFlow in a Multiplayer Online Strategy Game Under Development
ACSW '20: Proceedings of the Australasian Computer Science Week Multiconference

GameFlow is a widely used model of player enjoyment, with hundreds of applications to designing and evaluating games and game-like experiences since its first publication. Derived from a general set of heuristics for creating enjoyable player ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

FDG '17: Proceedings of the 12th International Conference on the Foundations of Digital Games

August 2017

545 pages

ISBN:9781450353199

DOI:10.1145/3102071

Conference Chair:
Sebastian Deterding
University of York
,
General Chairs:
Alessandro Canossa
Massive, Northeastern University
,
Casper Harteveld
Northeastern University
,
Program Chairs:
Jichen Zhu
Drexel University
,
Miguel Sicart
ITU Copenhagen

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Microsoft: Microsoft
Massive Entertainment: Massive Entertainment

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

FDG'17

Sponsor:

Microsoft
Massive Entertainment

FDG'17: International Conference on the Foundations of Digital Games 2017

August 14 - 17, 2017

Massachusetts, Hyannis

Acceptance Rates

FDG '17 Paper Acceptance Rate 36 of 89 submissions, 40%;

Overall Acceptance Rate 152 of 415 submissions, 37%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
652
Total Downloads

Downloads (Last 12 months)107
Downloads (Last 6 weeks)15

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang ZLiu JYannakakis G(2022)The Fun Facets of Mario: Multifaceted Experience-Driven PCG via Reinforcement LearningProceedings of the 17th International Conference on the Foundations of Digital Games10.1145/3555858.3563282(1-8)Online publication date: 5-Sep-2022
https://dl.acm.org/doi/10.1145/3555858.3563282
Paranthaman PSarkar ACooper S(2021)Applying Rapid Crowdsourced Playtesting to a Human Computation GameProceedings of the 16th International Conference on the Foundations of Digital Games10.1145/3472538.3472626(1-7)Online publication date: 3-Aug-2021
https://dl.acm.org/doi/10.1145/3472538.3472626
Matin AMaduro Mde Leon Pereira RTremblay-Savard O(2020)Effect of Timer, Top Score and Leaderboard on Performance and Motivation in a Human Computing GameProceedings of the 15th International Conference on the Foundations of Digital Games10.1145/3402942.3403000(1-10)Online publication date: 15-Sep-2020
https://dl.acm.org/doi/10.1145/3402942.3403000
Gitarana GFithratu CMuis ADarmakusuma R(2020)Analysis and Evaluation of Player Engagement in Serious Education Game using Game Refinement Theory Case Study: Arithmatopia Game2020 6th International Conference on Interactive Digital Media (ICIDM)10.1109/ICIDM51048.2020.9339633(1-5)Online publication date: 14-Dec-2020
https://doi.org/10.1109/ICIDM51048.2020.9339633
Sarkar ACooper SArnedo JNacke LVanden Abeele VToups Dugas P(2019)Using a Disjoint Skill Model for Game and Task Difficulty in Human Computation GamesExtended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts10.1145/3341215.3356310(661-669)Online publication date: 17-Oct-2019
https://dl.acm.org/doi/10.1145/3341215.3356310
Kim BCheng YLi ZLi RTan CWang SShi YHammer JArnedo JNacke LVanden Abeele VToups Dugas P(2019)Game With A Purpose To Collect Home Audio DataExtended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts10.1145/3341215.3356270(445-452)Online publication date: 17-Oct-2019
https://dl.acm.org/doi/10.1145/3341215.3356270
Luo ZGuzdial MRiedl MDeterding SKhosmood FPirker JApperley T(2019)Making CNNs for video parsing accessibleProceedings of the 14th International Conference on the Foundations of Digital Games10.1145/3337722.3337755(1-10)Online publication date: 26-Aug-2019
https://dl.acm.org/doi/10.1145/3337722.3337755
Simperl EReeves NPhethean CLynes TTinati R(2018)Is Virtual Citizen Science A Game?ACM Transactions on Social Computing10.1145/32099601:2(1-39)Online publication date: 27-Jun-2018
https://dl.acm.org/doi/10.1145/3209960

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents