[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article
Open access

A Scoping Review of Heuristics in Videos Games Research: Definitions, Development, Application, and Operationalisation

Published: 04 October 2023 Publication History

Abstract

Heuristics present a cheap and effective way of evaluating usability. However, in video games, evaluating unique player experiences that are dependent on individual preferences and abilities presents a challenge that goes beyond usability. Video games are more than just functional software, so games heuristics have been adapted to help examine functionality and experience. This paper reports on how papers published in the ACM Digital Library between 2012 and 2022 develop and apply heuristics in video games research. We found that heuristics are often used outside their intended purpose of being used in an expert evaluation. Instead, they are used as survey instruments, interview guides, codes for thematic analysis, and as design guidelines. This research contributes to HCI and video games research by distinguishing the terms design guidelines and design principles from heuristics. We make recommendations for researchers around developing heuristics and conducting video game heuristic evaluations. We propose a method for operationalising heuristics and make recommendations for the implementation of heuristics to improve the quality of video game heuristic reviews.

References

[1]
Andrew Aultman, Spencer Dowie, and Nelly Ann Hamid. 2018. Design heuristics for mobile augmented reality game user interfaces. Conference on Human Factors in Computing Systems-Proceedings 2018-April ( 2018 ), 1-5. https://doi.org/10.1145/3170427.3188580
[2]
Thiago Barcelos, Geiza Costa, Roberto Mu˜ oz, René Noël, and Ismar Silveira. 2013. Informal HCI : What may students learn from playability issues during a game design workshop ? ACM International Conference Proceeding Series ( 2013 ), 116-119. https://doi.org/10.1145/2535597.2535613
[3]
Laura Bartoli, Franca Garzotto, Mirko Gelsomini, Luigi Oliveto, and Matteo Valoriani. 2014. Designing and evaluating touchless playful interaction for ASD children. ACM International Conference Proceeding Series 1 ( 2014 ), 17-26. https://doi.org/10.1145/2593968.2593976
[4]
Clara Bonillo, Teresa Romão, and Eva Cerezo. 2019. Persuasive games in Interactive Spaces: The Hidden Treasure Game. ACM International Conference Proceeding Series ( 2019 ). https://doi.org/10.1145/3335595.3335626
[5]
Lance Bunt, Verona Leendertz, and A. Seugnet Blignaut. 2017. A heuristic evaluation of the design and development of a statistics serious game. ACM International Conference Proceeding Series 1, Episode 1 ( 2017 ), 1-7. https://doi.org/10. 1145/3136907.3136940
[6]
Lewis Carter and Leigh Ellen Potter. 2016. Designing games for presence in consumer virtual reality. SIGMIS-CPR 2016-Proceedings of the 2016 ACM SIGMIS Conference on Computers and People Research ( 2016 ), 141-148. https: //doi.org/10.1145/2890602.2890626
[7]
Konstantinos Chorianopoulos, Michail N. Giannakos, and Nikos Chrisochoides. 2014. Design principles for serious games in mathematics. ACM International Conference Proceeding Series 02-04-Octo ( 2014 ), 2-6. https://doi.org/10. 1145/2645791.2645843
[8]
Heather Desurvire, Martin Caplan, and Jozsef A. Toth. 2004. Using Heuristics to Evaluate the Playability of games. Conference on Human Factors in Computing Systems-Proceedings January 2004 ( 2004 ), 1509-1512. https://doi.org/10. 1145/985921.986102
[9]
Heather Desurvire and Charlotte Wiberg. 2009. Game usability heuristics (PLAY) for evaluating and designing better games: The next iteration. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5621 LNCS, July 2009 ( 2009 ), 557-566. https://doi.org/10.1007/978-3-642-02774-1{_} 60
[10]
Heather Desurvire and Charlotte Wiberg. 2010. User Experience Design for Inexperienced Gamers: GAP-Game Approachability Principles. In Evaluating User Experience in Games, Regina Bernhaupt (Ed.). Springer, 131-148. https://doi.org/10.1007/978-1-84882-963-3
[11]
Heather Desurvire and Charlotte Wiberg. 2015. User Experience Design for Inexperienced Gamers: GAP-Game Approachability Principles. ( 2015 ), 169-186. https://doi.org/10.1007/978-3-319-15985-0{_} 8
[12]
Sha Ding, Ningjiu Tang, Tao Lin, and Shiyuan Zhao. 2009. RTS-GameFlow: A new evaluation framework for RTS games. Proceedings-2009 International Conference on Computational Intelligence and Software Engineering, CiSE 2009 ( 2009 ), 0-3. https://doi.org/10.1109/CISE. 2009.5363526
[13]
Tristan C Endsley, Kelly A Sprehn, Ryan M Brill, Kimberly J Ryan, Emily C Vincent, and James M Martin Draper. 2017. Augmented Reality Design Heuristics: Designing For Dynamic Interactions., 2100-2104 pages. https: //doi.org/10.4324/9781003011293-1-3-4
[14]
Fong Ling Fu, Rong Chang Su, and Sheng Chin Yu. 2009. EGameFlow: A scale to measure learners' enjoyment of e-learning games. Computers and Education 52, 1 ( 2009 ), 101-112. https://doi.org/10.1016/j.compedu. 2008. 07.004
[15]
G. Gabriel Elías Chanchí, Wilmar Yesid Campo Muñoz, and Luis Freddy Muñoz Sanabria. 2019. A videogame as a support tool for dyslexia therapy using the GDevelop platform. ACM International Conference Proceeding Series ( 2019 ), 162-166. https://doi.org/10.1145/3364138.3364172
[16]
Nathan Gale, Pejman Mirza-Babaei, and Isabel Pedersen. 2015. Heuristic guidelines for wearable augmented reality applications. CHI PLAY 2015-Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play ( 2015 ), 529-534. https://doi.org/10.1145/2793107.2810309
[17]
Yan Ru Guo and Dion Hoe Lian Goh. 2016. Experimental Evaluation of Afective Embodied Agents in an Information Literacy Game. In JCDL '16. Newark, NJ, USA, 119-128. https://doi.org/10.1145/2910896.2910897
[18]
Yan Ru Guo and Dion Hoe Lian Goh. 2016. From storyboard to software: User evaluation of an information literacy game. Proceedings of the ACM Symposium on Applied Computing 04-08-Apri ( 2016 ), 199-201. https://doi.org/10.1145/ 2851613.2851909
[19]
Yan Ru Guo and Dion Hoe Lian Goh. 2016. Heuristic evaluation of an information literacy game. Proceedings of the Association for Information Science and Technology 53, 1 ( 2016 ), 1-4. https://doi.org/10.1002/pra2. 2016.14505301107
[20]
Celia Hodent. 2022. Advancing Game UX Maturity. Game Usability 2022 ( 2022 ), 7-24. https://doi.org/10.1201/ 9781003109389-3
[21]
Kathryn Hymes, Jessica Hammer, Hakan Seyalioglu, Carol Dow-Richards, Deidra Brown, Trish Hambridge, Jill Ventrice, Meguey Baker, Yeonsoo Julian Kim, Tim Hutchings, and William S. Evans. 2021. Designing Game-Based Rehabilitation Experiences for People with Aphasia. Proceedings of the ACM on Human-Computer Interaction 5, CHIPLAY ( 2021 ). https://doi.org/10.1145/3474697
[22]
Kyros Jalife, Casper Harteveld, and Christofer Holmgård. 2021. From Flow to Fuse: A Cognitive Perspective. Proceedings of the ACM on Human-Computer Interaction 5, CHIPLAY ( 2021 ). https://doi.org/10.1145/3474683
[23]
Kyros Jalife and Christofer Holmgrd. 2019. Cognitive components of flow states and applications to video game design: A brief framework. C and C 2019-Proceedings of the 2019 Creativity and Cognition ( 2019 ), 570-577. https: //doi.org/10.1145/3325480.3326574
[24]
Kalle Jegers. 2007. Pervasive game flow: Understanding player enjoyment in pervasive gaming. ACM Computers in Entertainment (CIE) 5, 1 ( 2007 ), 1-11. https://doi.org/10.1145/1236224.1236238
[25]
Sumbul Khan, Attila Achenbach, Wei Quin Yow, and Lucienne Blessing. 2021. A Case Study on the Design of Touchscreen-Based User Interfaces for Multilingual Older Adults from Southeast Asian Backgrounds. 5th Asian CHI Symposium 2021 ( 2021 ), 167-173. https://doi.org/10.1145/3429360.3468204
[26]
Kornchulee Khanana and Efie Lai Chong Law. 2013. Designing Children's Digital Games on Nutrition with Playability Heuristics. Conference on Human Factors in Computing Systems-Proceedings 2013-April ( 2013 ), 1071-1076. https: //doi.org/10.1145/2468356.2468548
[27]
Hannu Korhonen and Elina M.I. Koivisto. 2006. Playability heuristics for mobile games. ACM International Conference Proceeding Series 159, March ( 2006 ), 9-16. https://doi.org/10.1145/1152215.1152218
[28]
Andrés Lucero, Jussi Holopainen, Elina Ollila, Riku Suomela, and Evangelos Karapanos. 2013. The Playful Experiences (PLEX) framework as a guide for expert evaluation. Proceedings of the 6th International Conference on Designing Pleasurable Products and Interfaces, DPPI 2013 ( 2013 ), 221-230. https://doi.org/10.1145/2513506.2513530
[29]
Rolf Molich and Jakob Nielsen. 1990. Improving a Human-Computer Dialogue. Commun. ACM 33, 3 ( 1990 ), 338-348.
[30]
Christine Murad, Cosmin Munteanu, Benjamin R. Cowan, and Leigh Clark. 2021. Finding a New Voice: Transitioning Designers from GUI to VUI Design. In CUI2021-3rd Conference on Conversational User Interfaces (CUI '21). ACM, New York, NY, USA, Bilbao (online), Spain, 12 pages. https://doi.org/10.1145/3469595.3469617
[31]
Jakob Nielsen. 1992. Finding usability problems through heuristic evaluation. Conference on Human Factors in Computing Systems-Proceedings ( 1992 ), 373-380. https://doi.org/10.1145/142750.142834
[32]
Jakob Nielsen. 1994. Enhancing the explanatory power of usability heuristics. Conference on Human Factors in Computing Systems-Proceedings ( 1994 ), 152-158. https://doi.org/10.1145/191666.191729
[33]
Jakob Nielsen. 1994. Usability inspection methods. Conference on Human Factors in Computing Systems-Proceedings 1994-April ( 1994 ), 413-414. https://doi.org/10.1145/259963.260531
[34]
Jukob Nielsen and Rolf Molich. 1990. Heuristic Evaluation Of User Interfaces. In CHI 90 Proceedings. 240-256.
[35]
Zoë O'Shea and Jonathan Freeman. 2019. Game design frameworks: Where do we start ? ACM International Conference Proceeding Series April ( 2019 ). https://doi.org/10.1145/3337722.3337753
[36]
Janne Paavilainen. 2010. Critical review on video game evaluation heuristics. Proceedings of the International Academic Conference on the Future of Game Design and Technology ( 2010 ), 56-65. http://dl.acm.org/citation.cfm?id= 1920787 % 5Cnhttp://portal.acm.org/citation.cfm?doid= 1920778. 1920787
[37]
Janne Paavilainen. 2013. SoPlay Heuristics for Design and Evaluation of Social Network Games. CHI 2013 Games User Research Workshop January 2013 ( 2013 ), 6.
[38]
Matthew J Page, Joanne E McKenzie, Patrick M Bossuyt, Isabelle Boutron, Tammy C Hofmann, Cynthia D Mulrow, Larissa Shamseer, Jennifer M Tetzlaf, Elie A Akl, Sue E Brennan, Roger Chou, Julie Glanville, Jeremy M Grimshaw, Asbjørn Hróbjartsson, Manoj M Lalu, Tianjing Li, Elizabeth W Loder, Evan Mayo-Wilson, Steve McDonald, Luke A McGuinness, Lesley A Stewart, James Thomas, Andrea C Tricco, Vivian A Welch, Penny Whiting, and David Moher. 2021. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. British Medical Journal 372 ( 2021 ), 1-9. http://dx.doi.org/10.1136/bmj.n71
[39]
Agnieszka Palalas and Norine Wark. 2017. Design Principles for an Adult Literacy Mobile Learning Solution. ACM International Conference Proceeding Series ( 2017 ). https://doi.org/10.1145/3136907.3136934
[40]
Nathan Partlan, Erica Kleinman, Jim Howe, Sabbir Ahmad, Stacy Marsella, and Magy Seif El-nasr. 2021. Design-Driven Requirements for Computationally Co-Creative Game AI Design Tools. In The 16th International Conference on the Foundations ofDigital Games (FDG'21). ACM, Montreal, QC, Canada, 12 pages. https://doi.org/10.1145/3472538.3472573
[41]
Elaine Pearson and Adel Shaban. 2019. Evaluation of a prototype interactive working memory application for children with learning disabilities. Proceedings of the 16th Web For All 2019 Personalization-Personalizing the Web, W4A 2019 ( 2019 ), 1-2. https://doi.org/10.1145/3315002.3332440
[42]
David Pinelle, Nelson Wong, and Tadeusz Stach. 2008. Heuristic evaluation for games: Usability principles for video game design. Conference on Human Factors in Computing Systems-Proceedings ( 2008 ), 1453-1462. https: //doi.org/10.1145/1357054.1357282
[43]
David Pinelle, Nelson Wong, Tadeusz Stach, and Carl Gutwin. 2009. Usability heuristics for networked multiplayer games. GROUP'09-Proceedings of the 2009 ACM SIGCHI International Conference on Supporting Group Work ( 2009 ), 169-178. https://doi.org/10.1145/1531674.1531700
[44]
George Polya. 1945. "Heuristic reasoning". How to Solve It: A New Aspect of Mathematical Method. Princeton University Press, Princeton. https://doi.org/10.1515/ 9781400828678-039
[45]
Aditya Ponnada and Ajaykumar Kannan. 2013. Evaluation of mobile games using playability heuristics. Conference: Proceedings of the International Conference on Advances in Computing, Communications and Informatics ( 2013 ), 244-247. https://doi.org/10.1007/978-3-319-02958-0{_} 25
[46]
Daniela Quiñones and Cristian Rusu. 2017. How to develop usability heuristics: A systematic literature review. Computer Standards and Interfaces 53, September 2016 ( 2017 ), 89-122. https://doi.org/10.1016/j.csi. 2017. 03.009
[47]
Daniela Quiñones, Cristian Rusu, and Virginica Rusu. 2018. A methodology to develop usability/user experience heuristics. Computer Standards and Interfaces 59, November 2017 ( 2018 ), 109-129. https://doi.org/10.1016/j.csi. 2018. 03. 002
[48]
Florentin Rodio and J. M. Christian Bastien. 2013. Heuristics for Video Games Evaluation., 89-93 pages. https: //doi.org/10.1145/2534903.2534915
[49]
Cristian Rusu, Silvana Roncagliolo, Virginica Rusu, and Cesar Collazos. 2011. A Methodology to Establish Usability Heuristics. ACHI 2011 : The Fourth International Conference on Advances in Computer-Human Interactions ( 2011 ), 59-62.
[50]
Stephen R. Serge, Jonathan A. Stevens, and Latika Eifert. 2016. Make it usable: Highlighting the importance of improving the intuitiveness and usability of a computer-based training simulation. Proceedings-Winter Simulation Conference 2016-Febru ( 2016 ), 1056-1067. https://doi.org/10.1109/WSC. 2015.7408233
[51]
Chen Shu-Hui, Wu Wann-Yih, and Jason Dennison. 2018. Validation of EGameFlow: A self-report scale for measuring user experience in video game play. Computers in Entertainment 16, 3 ( 2018 ). https://doi.org/10.1145/3238249
[52]
Gavin Sim. 2012. Designing the anti-heuristic game: A game which violates heuristics. In IDC 2012, June 12-15. ACM, Bremen, Germany, 308. https://doi.org/10.1145/2307096.2307153
[53]
Gavin Sim, Janet C. Read, and Gilbert Cockton. 2009. Evidence based design of heuristics for computer assisted assessment. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5726 LNCS, PART 1 ( 2009 ), 204-216. https://doi.org/10.1007/978-3-642-03655-2{_} 25
[54]
Maria Carmela Sogono and Deborah Richards. 2013. A design template for multisensory and multimodal games to train and test children for sound localisation acuity. ACM International Conference Proceeding Series ( 2013 ). https: //doi.org/10.1145/2513002.2513005
[55]
Penny Sweetser. 2020. GameFlow 2020 : 15 Years of a Model of Player Enjoyment. PervasiveHealth: Pervasive Computing Technologies for Healthcare ( 2020 ), 705-711. https://doi.org/10.1145/3441000.3441048
[56]
Penny Sweetser and Daniel Johnson. 2019. Gameflow and player experience measures: An initial comparison of conceptual constructs. PervasiveHealth: Pervasive Computing Technologies for Healthcare ( 2019 ), 317-321. https: //doi.org/10.1145/3369457.3369486
[57]
Penny Sweetser, Daniel Johnson, and Jay Kyburz. 2020. Evaluating GameFlow in a Multiplayer Online Strategy Game under Development. PervasiveHealth: Pervasive Computing Technologies for Healthcare ( 2020 ). https://doi.org/10.1145/ 3373017.3373068
[58]
Penelope Sweetser, Daniel Johnson, and Peta Wyeth. 2012. Revisiting the GameFlow Model with Detailed Heuristics. Journal of Creative Technologies 2012, 3 ( 2012 ), 1-16. http://journal.colab.org.nz/article/21
[59]
Penny Sweetser, Daniel Johnson, Peta Wyeth, and Anne Ozdowska. 2012. GameFlow Heuristics for Designing and Evaluating Real-Time Strategy Games. In The 8th Australasian Conference on Interactive Entertainment. The 8th Australasian Conference on Interactive Entertainment, Auckland, NZ.
[60]
Penelope Sweetser, Daniel Johnson, Peta Wyeth, Aiman Anwar, Yan Meng, and Anne Ozdowska. 2017. GameFlow in diferent game genres and platforms. Computers in Entertainment 15, 3 ( 2017 ), 1-24. https://doi.org/10.1145/3034780
[61]
Penny Sweetser and Zane Rogalewicz. 2020. Afording Enjoyment in VR Games: Possibilities, Pitfalls, and Perfection. ACM International Conference Proceeding Series ( 2020 ), 55-64. https://doi.org/10.1145/3441000.3441050
[62]
Penny Sweetser, Zane Rogalewicz, and Qingyang Li. 2019. Understanding enjoyment in VR games with game lfow. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST ( 2019 ), 11-12. https: //doi.org/10.1145/3359996.3364800
[63]
Penelope Sweetser and Peta Wyeth. 2005. GameFlow: a model for evaluating player enjoyment in games. Computers in Entertainment (CIE) 3, 3 ( 2005 ). https://doi.org/10.1145/1077246.1077253
[64]
Ramin Tadayon, Winslow Burleson, and Ashish Amresh. 2012. World of Golf: A socially relevant simulation game. ACM International Conference Proceeding Series ( 2012 ), 83-92. https://doi.org/10.1145/2367616.2367626
[65]
Jean Lee Tan, Dion Hoe-Lian Goh, Rebecca P Ang, and Vivien S Huan. 2010. Usability and Playability Heuristics for Evaluation of an Instructional Game. E-Learn 2010-World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education September 2016 ( 2010 ), 363-373. http://www.learntechlib.org/p/35570
[66]
Gustavo F Tondello, Dennis Kappen, Marim Ganaba, and Lennart E Nacke. 2019. Gameful Design Heuristics: A Gamification Inspection Tool. In Proceedings of HCI International. 224-244.
[67]
Gustavo F. Tondello, Dennis L. Kappen, Elisa D. Mekler, Marim Ganaba, and Lennart E. Nacke. 2016. Heuristic evaluation for gameful design. CHI PLAY 2016-Proceedings of the Annual Symposium on Computer-Human Interaction in Play Companion ( 2016 ), 315-323. https://doi.org/10.1145/2968120.2987729
[68]
Obelema Akobo Wodike, Gavin Sim, and Matthew Horton. 2014. Empowering teenagers to perform a heuristic evaluation of a game. Proceedings of the 28th International BCS Human Computer Interaction Conference: Sand, Sea and Sky-Holiday HCI, HCI 2014 ( 2014 ), 353-358. https://doi.org/10.14236/ewic/hci2014.57

Cited By

View all
  • (2024)Guidelines for Disc Golf Applications and Design Principles for SportsHCI: A Human-Centered ApproachProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701584(396-411)Online publication date: 1-Dec-2024
  • (2024)GameFlow Affordances: Towards a Tool for Designing Gameful ExperiencesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650887(1-7)Online publication date: 11-May-2024
  • (2024)Smartphone Games Heuristics (SmGH) – Towards a Standard Set of Platform-Centric Heuristic for Smartphone Games EvaluationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2356911(1-22)Online publication date: 11-Jun-2024

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction
Proceedings of the ACM on Human-Computer Interaction  Volume 7, Issue CHI PLAY
November 2023
1360 pages
EISSN:2573-0142
DOI:10.1145/3554313
Issue’s Table of Contents
This work is licensed under a Creative Commons Attribution 4.0 International License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 October 2023
Published in PACMHCI Volume 7, Issue CHI PLAY

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Design Guidelines
  2. Design Principles
  3. Heuristics
  4. Video Games

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)406
  • Downloads (Last 6 weeks)59
Reflects downloads up to 11 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Guidelines for Disc Golf Applications and Design Principles for SportsHCI: A Human-Centered ApproachProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701584(396-411)Online publication date: 1-Dec-2024
  • (2024)GameFlow Affordances: Towards a Tool for Designing Gameful ExperiencesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650887(1-7)Online publication date: 11-May-2024
  • (2024)Smartphone Games Heuristics (SmGH) – Towards a Standard Set of Platform-Centric Heuristic for Smartphone Games EvaluationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2356911(1-22)Online publication date: 11-Jun-2024

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Full Access

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media