Abstract
The steadily increasing popularity of computer games has led to the rise of a multi-billion dollar industry. Due to the scale of the computer game industry, developing a successful game is challenging. In addition, prior studies show that gamers are extremely hard to please, making the quality of games an important issue. Most online game stores allow users to review a game that they bought. Such reviews can make or break a game, as other potential buyers often base their purchasing decisions on the reviews of a game. Hence, studying game reviews can help game developers better understand user concerns, and further improve the user-perceived quality of games. In this paper, we perform an empirical study of the reviews of 6224 games on the Steam platform, one of the most popular digital game delivery platforms, to better understand if game reviews share similar characteristics with mobile app reviews, and thereby understand whether the conclusions and tools from mobile app review studies can be leveraged by game developers. In addition, new insights from game reviews could possibly open up new research directions for research of mobile app reviews. We first conduct a preliminary study to understand the number of game reviews and the complexity to read through them. In addition, we study the relation between several game-specific characteristics and the fluctuations of the number of reviews that are received on a daily basis. We then focus on the useful information that can be acquired from reviews by studying the major concerns that users express in their reviews, and the amount of play time before players post a review. We find that game reviews are different from mobile app reviews along several aspects. Additionally, the number of playing hours before posting a review is a unique and helpful attribute for developers that is not found in mobile app reviews. Future longitudinal studies should be conducted to help developers and researchers leverage this information. Although negative reviews contain more valuable information about the negative aspects of the game, such as mentioned complaints and bug reports, developers and researchers should also not ignore the potentially useful information in positive reviews. Our study on game reviews serves as a starting point for other game review researchers, and suggests that prior studies on mobile app reviews may need to be revisited.
Similar content being viewed by others
Notes
Anyone who purchased the game is the owner of the game, but only the people who played the game on that day are counted as the player of the game.
References
Alden (2017) Steamworks Partner Program. https://partner.steamgames.com/steamdirect, (last visited: Mar 29, 2018)
Apple Inc. (2017) Choosing a Membership. https://developer.apple.com/support/compare-memberships/, (last visited: Mar 29, 2018)
Bates D, Mächler M, Bolker B, Walker S (2015) Fitting linear mixed-effects models using lme4. J Stat Softw 67(1):1–48
Becker R, Chernihov Y, Shavitt Y, Zilberman N (2012) An analysis of the Steam community network evolution. In: Proceedings of the 27th Convention of Electrical & Electronics Engineers in Israel (IEEEI). IEEE, pp 1–5
Blackburn J, Kourtellis N, Skvoretz J, Ripeanu M, Iamnitchi A (2014) Cheating in online games: a social network perspective. ACM Trans Internet Technol (TOIT) 13(3):9
Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet Allocation. J Mach Learn Res 3(Jan):993–1022
Chambers C, Feng Wc, Sahu S, Saha D (2005) Measurement-based characterization of a collection of on-line games. In: Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement, USENIX Association, pp 1–1
Chen N, Lin J, Hoi SCH, Xiao X, Zhang B (2014) Ar-miner: Mining Informative reviews for developers from mobile app marketplace. In: Proceedings of the 36th International Conference on Software Engineering (ICSE). ACM, New York, pp 767–778
Cheung GK, Zimmermann T, Nagappan N (2014) The first hour experience: How the initial play can engage (or lose) new players. In: Proceedings of the First ACM SIGCHI Annual Symposium on Computer-human Interaction in Play (CHI PLAY). ACM, New York, pp 57–66
Ciurumelea A, Schaufelbühl A, Panichella S, Gall HC (2017) Analyzing reviews and code of mobile apps for better release planning. In: Proceedings of the 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE, pp 91–102
Cobbett R (2017) From Shareware superstars to the Steam gold rush: How indie conquered the PC. http://www.pcgamer.com/from-shareware-superstars-to-the-steam-gold-rush-how-indie-conquered-the-pc/ (last visited: Mar 29 2018)
Coleman M, Liau TL (1975) A computer readability formula designed for machine scoring. J Appl Psychol 60(2):283
Di Sorbo A, Panichella S, Alexandru CV, Shimagaki J, Visaggio CA, Canfora G, Gall HC (2016) What would users change in my app? Summarizing app reviews for recommending software changes. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, pp 499–510
Fu B, Lin J, Li L, Faloutsos C, Hong J, Sadeh N (2013) Why people hate your app: Making sense of user feedback in a mobile app store. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp 1276–1284
Genc-Nayebi N, Abran A (2017) A systematic literature review: Opinion mining studies from mobile app store user reviews. J Syst Softw (JSS) 125:207–219
Google (2017) How to use the Play Console. https://support.google.com/googleplay/android-developer/answer/6112435?hl=en, (last visited: Mar 29, 2018)
Grano G, Di Sorbo A, Mercaldo F, Visaggio CA, Canfora G, Panichella S (2017) Android apps and user feedback: a dataset for software evolution and quality improvement. In: Proceedings of the 2nd ACM SIGSOFT International Workshop on App Market Analytics. ACM, pp 8–11
Gu X, Kim S (2015) What parts of your apps are loved by users?. In: 30Th international conference on automated software engineering (ASE). IEEE, pp 760–770
Hassan S, Tantithamthavorn C, Bezemer CP, Hassan AE (2018) Studying the dialogue between users and developers of free apps in the google play store. Empir Softw Eng 23(3):1275–1312
Hoon L, Vasa R, Schneider JG, Mouzakis K (2012) A preliminary analysis of vocabulary in mobile app user reviews. In: Proceedings of the 24th Australian Computer-Human Interaction Conference. ACM, pp 245–248
Iacob C, Harrison R (2013) Retrieving and analyzing mobile apps feature requests from online reviews. In: 10Th working conference on mining software repositories (MSR). IEEE, pp 41–44
Khalid H, Shihab E, Nagappan M, Hassan AE (2015) What do mobile app users complain about? IEEE Softw 32(3):70–77
Kincaid J (1975) Derivation of New Readability Formulas: (automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel Research Branch report 8–75, Chief of Naval Technical Training, Naval Air Station Memphis
Lewis C, Whitehead J, Wardrip-fruin N (2010) What went wrong: a taxonomy of video game bugs. In: Proceedings of the 5th International Conference on the Foundations of Digital Games (FDG). ACM, pp 108–115
Lin D, Bezemer CP, Hassan AE (2017a) Studying the urgent updates of popular games on the Steam platform. Empir Softw Eng 22(4):2095–2126
Lin D, Bezemer CP, Hassan AE (2017b) An empirical study of early access games on the Steam platform. Empirical Software Engineering 23(2):771–799
Long JD, Feng d, Cliff N (2003) Ordinal Analysis of Behavioral Data. Wiley, Hoboken
Maalej W, Nabil H (2015) Bug report, feature request, or simply praise? on automatically classifying app reviews.In: 23Rd international requirements engineering conference (RE). IEEE, pp 116–125
Machkovech S (2018) Nintendo: letting our fans review video games might not be a good idea. https://arstechnica.com/gaming/2018/02/nintendo-has-opened-the-doors-to-fans-game-reviews-on-nintendo-com/, (last visited: Mar 29, 2018)
Man Y, Gao C, Lyu MR, Jiang J (2016) Experience Report: Understanding cross-platform app issues from user reviews. In: 27Th international symposium on software reliability engineering (ISSRE). IEEE, pp 138–149
Martens D, Johann T (2017) On the emotion of users in app reviews. In: Proceedings of the 2nd International Workshop on Emotion Awareness in Software Engineering. IEEE Press, pp 8–14
Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2017) A survey of app store analysis for software engineering. IEEE Trans Softw Eng (TSE) PP(99):1–32
McIlroy S, Ali N, Khalid H, Hassan AE (2016) Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews. Empir Softw Eng 21(3):1067–1106
McIlroy S, Shang W, Ali N, Hassan A (2017) Is it worth responding to reviews? a case study of the top free apps in the Google Play store. IEEE Softw 34 (3):64–71
McIntosh S, Kamei Y, Adams B, Hassan AE (2016) An empirical study of the impact of modern code review practices on software quality. Empir Softw Eng 21 (5):2146–2189
Nagappan M, Zimmermann T, Bird C (2013) Diversity in software engineering research. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE). ACM, New York, pp 466–476
Noei E, Syer MD, Zou Y, Hassan AE, Keivanloo I (2017) A study of the relation of mobile device attributes with the user-perceived quality of android apps. Empir Softw Eng 22(6):3088–3116
Orland K (2014) Introducing Steam Gauge: Ars Reveals Steam’s most popular games. http://arstechnica.com/gaming/2014/04/introducing-steam-gauge-ars-reveals-steams-most-popular-games/ (last visited: Mar 29 2018)
Pagano D, Maalej W (2013) User feedback in the appstore: An empirical study. In: 21St international requirements engineering conference (RE). IEEE, pp 125–134
Palomba F, Salza P, Ciurumelea A, Panichella S, Gall H, Ferrucci F, De Lucia A (2017) Recommending and localizing change requests for mobile apps based on user reviews. In: Proceedings of the 39th International Conference on Software Engineering (ICSE). IEEE Press, pp 106–117
Panichella S, Di Sorbo A, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can i improve my app? classifying user reviews for software maintenance and evolution. In: International conference on software maintenance and evolution (ICSME). IEEE, pp 281–290
Pavel Djundik MB (2016) Steam DB - Steam Database. https://steamdb.info/, (last visited: Mar 29, 2018)
SuperData Research (2016) Market Brief — Year in Review 2016. https://web.archive.org/web/20170702202939/https://www.superdataresearch.com/market-data/market-brief-year-in-review/, (last visited: Mar 29, 2018)
Romano J, Kromrey JD, Coraggio J, Skowronek J, Devine L (2006) Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen’s d indices the most appropriate choices. In; Annual meeting of the Southern Association for Institutional Research
Ruiz IJM, Nagappan M, Adams B, Berger T, Dienst S, Hassan AE (2016) Examining the rating system used in mobile-app stores. IEEE Softw 33(6):86–92
Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Softw Eng (TSE) 25(4):557–572
Seaman CB, Shull F, Regardie M, Elbert D, Feldmann RL, Guo Y, Godfrey S (2008) Defect categorization: making use of a decade of widely varying historical data. In: Proceedings of the 2nd International Symposium on Empirical Software Engineering and Measurement (ESEM). ACM, pp 149–157
Sergey G (2016) SteamSpy - All the data and stats about Steam games. http://steamspy.com/, (last visited: Mar 29, 2018)
Sifa R, Drachen A, Bauckhage C (2015) Large-scale cross-game player behavior analysis on steam. In: Artificial Intelligence and Interactive Digital Entertainment International Conference. AAAI Press, pp 198–204
Stern C (2012) what makes a game indie: a universal definition. http://sinisterdesign.net/what-makes-a-game-indie-a-universal-definitio n/, (last visited: Mar 29, 2018)
Takahashi D (2016) PwC: Game Industry to grow nearly 5% annually through 2020. https://venturebeat.com/2016/06/08/the-u-s-and-global-game-industries-will-grow-a-healthy-amount-by-2020-pwc-forecasts/ (last visited: Mar 29 2018)
Valve (2016a) Steam Community. http://steamcommunity.com/, (last visited: Mar 29, 2018)
Valve (2016b) Steam Store. http://store.steampowered.com/, (last visited: Mar 29, 2018)
Vasa R, Hoon L, Mouzakis K, Noguchi A (2012) A preliminary analysis of mobile app user reviews. In: Proceedings of the 24th Australian Computer-Human Interaction Conference. ACM, pp 241–244
Villarroel L, Bavota G, Russo B, Oliveto R, Di Penta M (2016) Release planning of mobile apps based on user reviews. In: Proceedings of the 38th International Conference on Software Engineering (ICSE). ACM, pp 14–24
Washburn Jr, M, Sathiyanarayanan P, Nagappan M, Zimmermann T, Bird C (2016) What went right and what went wrong: An analysis of 155 postmortems from game development. In: Proceedings of the 38th International Conference on Software Engineering (ICSE). IEEE/ACM, pp 280–289
Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bullet 1 (6):80–83
Acknowledgements
We are grateful to Sergey Galyonkin, the owner of Steam Spy, who generously gave us access to all the historical data of Steam collected by Steam Spy for this research.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Massimiliano Di Penta
Rights and permissions
About this article
Cite this article
Lin, D., Bezemer, CP., Zou, Y. et al. An empirical study of game reviews on the Steam platform. Empir Software Eng 24, 170–207 (2019). https://doi.org/10.1007/s10664-018-9627-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-018-9627-4