Experimentation in Early-Stage Video Game Startups: Practices and Challenges

Over the last 40 years, video games have increasingly replaced traditional games as leisure activities and have disrupted how we spend our leisure time. The video game market has become an established and ever-growing global industry for over two decades. In 2022, the global video market was worth USD 42.9 billion, and the revenue is expected to grow with an annual growth rate of 8.74%^{Footnote 1}. Originally, video games refer to the games that do not require a microprocessor and use analogue intensity signals displayed on a cathode ray tube (CRT) [17]. The availability of new imaging technologies, such as consoles, home computers, Virtual Reality (VR) devices, etc., has made the idea of video games more conceptual and less tied to a specific technology [5].

Developing a successful video game is a very demanding and complex process. It involves expertise from various disciplines, e.g. software/game development, arts, animation, sound engineering, etc., which may increase the complexity of communication and coordination [10]. Furthermore, it is unclear whether a game will succeed in the market, which poses a major risk to game publishers when investing in new game development projects. Unlike other software startups, video game startups do not build technological solutions to solve real problems. Instead, they combine art, science, and craft to offer fun, entertainment, and experience through the games [2, 11]. Yet, these requirements have no metrics to be applied, yet they must be validated at each stage of the development process.

An effective adoption and implementation of experimentation is a staged process [13]. In this study, we aim to gain insights into how video game startups approach experimentation to develop games. To guide the study, we explore the research question: How do video game startups use experimentation in practice?

In innovative endeavours, the required knowledge for success is generally unknown [9]. Thus, experimentation is particularly useful for acquiring knowledge and reducing uncertainty. Experimentation is an approach based on continuously identifying critical assumptions, transforming them as hypotheses, and prioritising and testing them with experiments to support or refute them [12]. However, most startups persist with the original ideas rather than experimenting [6, 7, 14].

While research in game startups exists, they are limited to mobile game development. For example, Vanhala et al. [16] analysed six Finnish mobile game startups and found that human capital is the most important element in their business models. Moreover, the key challenge is to raise the awareness of game players. Kasurinen et al. [8] showed that game developers are generally pleased by the tools available to experiment with the concept and build prototypes.

Research also shows that the iterative and incremental nature of agile methods positively impacts communication, game quality, and the ability to find the fun aspects of the mobile game features [10]. In contrast, the agile principle of embracing changes increases the pressure to meet the deadline [1]. Mobile game startups should be cautious in considering the minimum viable product concept. The first version of a game artefact released to the market needs to be of sufficient quality to attract and lock in users for an adequate amount of time to allow for further development of the game [15]. This study aims to complement existing research by investigating how video game startups conduct experimentation.

We performed semi-structured interviews [3] to gain insights into how video game startups conduct experimentation. Interview candidates were identified by the first author collaborating with Blekinge Business Incubator (BBI) in Karlskrona, Sweden. The first interview was with a business coach in the incubator, who provided a list of founders of independent (indie) and internal video game startups operating inside larger companies. The interviews were held and recorded in a video conferencing system (Microsoft Teams), each lasting between 60 and 90 min. The profiles of the interviewees are shown in Table 1. The audio recordings were transcribed and analysed using thematic analysis [4]. The transcripts were sent back to the interviewees for follow-up questions and clarification.

This section reports our findings by describing six experimentation scenarios. All quotes and information herein are derived from the interview transcripts.

4.1 Technical or Digital Prototyping

Our interviews reveal that, in the early stages, the main challenge of game development lies not in the ideation process but in the execution and making the game work. Hence, the first purpose of experimentation is to assess the technical feasibility of the team to develop the game. The game’s initial idea is usually outlined in a game design document and describes the game at a high level from the user’s perspective. The team builds prototypes using a 3D engine, e.g., Unity, to test the game’s complexity and scope. In Mana Brigade, a slightly different approach was taken. This company started out performing experiments with a marching cube algorithm^{Footnote 2}. This algorithm was then implemented in Unity, and the user experience was tested using VR devices.

All interviewees agree that technical experimentation is crucial to evaluate their capability to build the game. For example, if they can solve all problems to build a game or need key people with certain skills and expertise. Technical experimentation also showcases their capabilities to potential investors or publishers.

4.2 Controlled Game Tests

Game startups also experiment with external stakeholders, such as end users or players, to evaluate whether they understand the game’s concepts and mechanics. In the case of The Station, they hired external game companies to test their game: “[The external video game companies] bring in players. We have a questionnaire that we want them to answer that they rate the game [like] ‘Was there anything unclear? What did you not like? What did you like?’ ” (Interviewee C)

4.3 Mock Reviews

In the case of The Station, they asked game journalists to write a mock review and to give a score of their game compared to other games in the same genre. The score was used as an early indicator of what could happen when the game was released. In the case of Mana Brigade, they mentioned that it does not use this approach due to a lack of funding.

4.4 Presenting and Pitching in Game Conferences

Presenting and pitching new games in video game conferences is a good opportunity to validate assumptions about the game, e.g., the basic idea and its potential market. In these events, video game startups can meet and talk to publishers, investors, or game scouts to get investments from them to build the game. Mana Brigade’s first experimentation with external stakeholders was competing in a game competition in 2021. “For the first iteration, we want it to be multiplayer, and [we want] to explore dungeons. It’s like awesome, like real-time events. [But] we got feedback from the [judges] ‘This doesn’t make sense.’ So we took that year to iterate on it, and then we wanted to do like it was still single player, but it was still crafting and then adventuring.” (Interviewee B)

However, explaining and convincing the game concepts and design to publishers is a big problem. Video game startups need to find ways to explain their game and, at the same time, to find the right publishers: “[Publishers] get bombarded with hundreds of game ideas they must go through to find that one good game... One publisher wants a game design document, not a PowerPoint. They don’t care about the pictures, [while others] want many. It’s very hard to know what they want.” (Interviewee B)

4.5 Social Media Engagement

The interviewees expressed that they could use social media platforms, i.e., YouTube or Instagram, to experiment and gain user feedback. For example, by releasing screenshots, images, videos or tutorials on social media and measuring gamers’ reactions to these. However, this may not work for indie game startups. They must balance the effort and resources between developing the game and actively maintaining communication with the community and the users.

4.6 Early Release of Vertical Slice

Releasing a vertical slice^{Footnote 3} on video game platforms like Steam for user testing may allow game startups to build a player base. It may also give them some small revenue to improve the game, but it could harm their reputation. Besides that, they need to find the right audience for their games:“The game industry is so big... maybe 100 [new games are published] every day on Steam. It’s hard to reach and find your audience and see your game. There is so much information [on Steam], and many games [can easily] get drowned.” (Interviewee A)

Table 2 summarises the six practices we identified and their associated challenges. Some of the practices are present in other contexts, e.g. prototyping. Some are adapted to the context of games, e.g. controlled game tests and early release, while some are specific to the game industry, e.g. mock reviews by journalists and presentations in game conferences.

The identified challenges can be related to the experimentation inhibitors experimentation identified by Melegati et al. [13]. Missing skill sets and expertise and lack of funding to hire game testers or journalists relate to the scarcity of technical and development resources. The need for early releases is associated with time pressure and over-focus on customer base growth in the early phase. However, the difficulty of explaining a game’s concepts to publishers might be considered a specific challenge of video game startups. It could be classified as an inhibitor to a valid experiment, as described by Melegati et al. In summary, our study describes the particularities of video game startups and provides evidence to support an existing model in the literature.

This study poses a first step to understanding experimentation within gaming startups. Next, additional video game startups will be studied to further expand on their experimentation practices. We will also expand beyond studying startups that develop games for specific platforms, such as consoles and VR, including other platforms, such as smartphones and tablets. By contrasting and comparing the results, we can improve the generalisability of the findings. Future research could also investigate gaming startups’ use of novel technologies, such as artificial intelligence and how these affect their experimentation.