[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Next Article in Journal
Stability and Bioaccessibility of β-Arbutin in Simulated Digestion Conditions
Previous Article in Journal
Improved Honey Badger Algorithm and Its Application to K-Means Clustering
Previous Article in Special Issue
Reliability of Critical Fitness Tests for Police Officers
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 15
Issue 2
10.3390/app15020719
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Sounds and Natures Do Often Agree: Prediction of Esports Players’ Performance in Fighting Games Based on the Operating Sounds of Game Controllers

by
Yamato Hiratsuka
1,
Kazuki Kuga
1,
Takahiro Miura
2,*,
Tetsuo Tanaka
1 and
Mari Ueda
1,*
1
Kanagawa Institute of Technology, Atsugi 243-0292, Japan
2
Human Augmentation Research Center (HARC), National Institute of Advanced Industrial Science and Technology (AIST), Kashiwa 277-0882, Japan
*
Authors to whom correspondence should be addressed.
Appl. Sci. 2025, 15(2), 719; https://doi.org/10.3390/app15020719
Submission received: 4 November 2024 / Revised: 3 January 2025 / Accepted: 9 January 2025 / Published: 13 January 2025
(This article belongs to the Special Issue Human Performance and Health in Sports)
Browse Figures
Figure 1
<p>Experimental setup for measuring controller operation sounds.</p> "> Figure 2
<p>Difference in maximum sound pressure level <math display="inline"><semantics> <msub> <mi>L</mi> <mi>max</mi> </msub> </semantics></math> for (<b>left</b>) player level and (<b>right</b>) game controller.</p> "> Figure 3
<p>Relationship between <math display="inline"><semantics> <msub> <mi>L</mi> <mi>max</mi> </msub> </semantics></math> and (<b>upper left</b>) QOL score, (<b>upper right</b>) TMD score, (<b>lower left</b>) positive engagement score of the WASEDA, as well as the (<b>lower right</b>) in-game score.</p> ">
Versions Notes

Abstract

:
In research focusing on esports, studies have been conducted on designs that attract competitors, performance estimation, training methods, and motivational factors. However, quantitative and convenient methods for performance evaluation are still in the development stage among the numerous performance evaluation methods. In particular, few method has been developed to objectively measure an individual’s mental state utilizing limited equipment. It has been observed that when players’ performance deteriorates or they are under pressure, they occasionally operate the controller in accordance with their state, resulting in the sound of the controller increasing. Therefore, this study aimed to clarify the relationship between the sound of esports players’ controller operations and their objective as well as subjective metrics, including their emotional state and performance during the game. Initially, the controller sounds of players of various ranks in Super Smash Bros. Ultimate (SSBU) by Nintendo were explored, aiming to elucidate the connection between the operation sounds of adept and intermediate esports competitors and their day-to-day fluctuations in game performance and emotional well-being. The findings revealed a discernible pattern: the more proficient the player, the more resonant the sounds emanating from their controller during gameplay. Furthermore, the operational sounds of skilled players exhibited an escalation when their performance faltered.

1. Introduction

The computer game market continues to rise globally. The increasing trend in games has been further reinforced by the “stay at home” demand associated with the COVID-19 pandemic in 2020 [1,2]. Under these circumstances, esports (electronic sports), which are entertainment, competitions, and sports played utilizing electronic devices, have been gaining popularity [3,4,5]. In particular, competitive games played over computers, whose genres include first-person shooter (FPS), third-person shooter (TPS), digital card game (DCG), real-time strategy (RTS), multi-player online battle arena (MOBA), fighting games, puzzle games, sports games, and music games, have been conducted in many countries [3,6,7,8,9]. It is expected that esports will further develop technologically and competitively by incorporating extended reality (XR) as well as metaverse technologies and that the market size will expand further in the future [10,11,12,13].
Regarding esports research, there have been studies on game designs attracting competitors, becoming esports players, and their characteristics, performance estimation, training methods, motivational factors, and atmospheric and social factors related to esports events [14,15,16,17]. Initially, qualitative studies, such as interviews and questionnaire surveys, have been conducted, primarily to identify the factors related to performance and to evaluate the performance of players [18,19,20,21,22,23]. In the subsequent phase, approaches to cognitive studies have been introduced, and various quantitative studies have reported on the performance metrics and relationship between action performance and personality traits, as well as cognitive functions and mental health aspects, including the stress tolerance of the esports experience [24,25,26,27,28]. To quantitatively evaluate the findings of matches, studies applying Elo and Glicko ratings and their extensions, which are utilized to evaluate performance in board games and various team sports, have been reported [29,30,31], and studies have been conducted to predict the findings [32].
Various methods for evaluating players’ mental states and action performances during gameplay have been reported. Nagorsky et al. reported that, based on a questionnaire survey of esports players, although there are differences among various esports in training methods, there are similarities and commonalities between esports and physical sports performance models [33]. Abramov et al. demonstrated the relationship between the estimated team performance in team games and the emotional state of players during a game, predicted based on the players’ voice communication recordings and game logs [34]. Pluss et al. compared the perceived motor performance in professional and amateur esports players and reported that professionals were less affected by the trade-off between speed and accuracy [16]. In addition, training methods related to esports have been conducted and developed.
Among numerous performance evaluation methods, convenient quantitative methods are still in the development stage. Although previous studies have evaluated performance and emotional states by obtaining detailed game and operation logs, few methods have been developed to objectively measure mental states utilizing limited equipment in esports settings. It has been observed that when players’ performance deteriorates or they are under pressure, they sometimes operate the controller in accordance with their state. During these circumstances, there is an increase in the sound of the controller during its utilization.
Although similar phenomena have been reported for hardware and software keyboards, where stress and other factors can be estimated from the input status [35,36,37,38], there have been minimal reports on game controller operations. Many studies have explored detecting stress and emotional states from keyboard and mouse usage [39,40,41,42], with recent research focusing on stress prediction using machine learning techniques [43,44,45]. Moreover, there have been efforts to detect stress levels during computer usage using physiological indicators such as electroencephalography (EEG), blood volume pulse, and pupil diameter during these operations [46,47,48]. However, to the best of our knowledge, few studies have examined the operations of game controllers.
Thus, by examining the quantitative relationship between controller operation sounds and game performance, it is possible to predict performance based solely on the level of the sounds with regard to the operations of the controller. However, the relationship between the sounds of the controller during operation and player performance remains unclear.
This study aimed to clarify the relationship between the sound of esports players’ controller operations and their objective, as well as subjective metrics, including their emotional state and game performance. Initially, we investigate the sounds of the controller of players exhibiting various skill levels in Super Smash Bros. Ultimate (SSBU), developed by Nintendo [49]. In addition, the association between the control sounds of skilled esports players and their day-to-day variations in game performance and emotional state were investigated. Our research questions are as follows:
Q1. 
How do the sound characteristics of game controller operations reflect players’ gaming skills?
Q2. 
To what extent can the maximum sound pressure level ( L max ) of controller operation sounds predict players’ psychological states and in-game performance?

2. Experiment 1: Differences in Controller Operation Sounds Based on Player Skill

The game employed in this study was SSBU, an action game released by Nintendo in 2018, for which numerous user-sponsored competitions have been held. Unlike other fighting games, players do not compete for a strength gauge; instead, the winner is decided by moving from the game stage to the external stage. This experimental setup was modeled on online esports tournaments.

2.1. Participants

The participants comprised 22 male university students, aged 18 to 24 years, who played SSBU. To reduce potential issues arising from unfamiliarity with game controllers, they were asked to use the game controller they usually utilize: nine participants utilized the GameCube controller, three utilized the HORI Wireless controller, and ten utilized the Switch Pro controller. These controllers are officially certified by the game’s development company, are widely available, and are used by gamers with varying preferences. Based on the participants’ win rates in the online SSBU, the top and bottom 30% of each participant group were defined as advanced and beginner players, respectively. The rest were defined as intermediate players. The win rate for beginners was less than 40%, and the win rate for advanced players was 75% or more.

2.2. Method

Figure 1 illustrates the experimental setup. Participants were tasked with playing against non-player characters wearing headsets (Sony Corporation (Tokyo, Japan), MDR-ZX310). The distance between the participants and the gaming display (BenQ Japan (Tokyo, Japan), ZOWIE XL2411K) was approximately 70 cm. The PCM recorder (Roland Corporation (Hamamatsu, Shizuoka, Japan), R-07) was placed 50 to 70 cm from the ground where the participant held the controller, and almost 30 cm to the left of the center of the controller. The background noise level ( L Aeq , 30 s ) was 41 dB as measured using a sound level meter (RION NL-52). The frequency range analyzed was in the audible range, from 20 Hz to 20 kHz, and the measurement was taken using a sound level meter (Rion NL-52) as a flat frequency response without using auditory filters.
The controller operation sound was continuously recorded while playing the game and the maximum sound pressure level was obtained, L max . This setting was selected as the aim was to measure the instantaneous sound pressure level, as the sounds of the buttons being pressed were intermittent.

2.3. Analysis

An analysis of variance (ANOVA) was employed to examine the primary effects of the player level (beginner, intermediate, and advanced) and the game controller on L max , which was confirmed to be normally distributed based on the Shapiro–Wilk test, and the equality of variances for each factor was also confirmed using the Bartlett test. A post hoc multiple comparison method with the Tukey–Kramer test was utilized to determine significant differences in primary effects. Concurrently, effect sizes, such as partial η 2 and Cohen’s d, were calculated utilizing ANOVA and multiple comparisons. The significance levels were set at 5% and 10%. The effect sizes were determined according to Cohen’s criterion [50].

2.4. Results

First, the Shapiro–Wilk test confirmed that L max may satisfy the normality assumption ( p = 0.899 > 0.05 ). In addition, the Bartlett test confirmed the homoscedasticity of variance in L max for each player’s skill ( p = 0.726 > 0.05 ) and each game controller ( p = 0.787 > 0.05 ). The ANOVA of the differences in L max confirmed a significant primary effect for the player level ( F ( 1 , 2 ) = 82.45 , p = 0.007 < 0.01 , partial η 2 = 0.49 (large)) and game controller ( F ( 1 , 2 ) = 55.93 , p = 0.024 < 0.05 , partial η 2 = 0.39 (large)). The interaction between the player level and game controller was not significant for L max ( F ( 1 , 2 ) = 14.81 , p = 0.31 > 0.05 , partial η 2 = 0.39 (large)).
The figures on the left and right sides in Figure 2 exhibit the maximum sound pressure levels for the players and game controllers, respectively. The findings of multiple comparisons illustrate that L max was significantly smaller for beginners as opposed to intermediate ( t ( 14 ) = 5.40 , p = 0.002 < 0.01 , d = 1.95 (large)) and advanced ( t ( 10 ) = 6.48 , p < 0.001 , d = 2.37 (large)) players. The L max was also significantly different in the game controllers: The operation of GameCube controllers was significantly louder compared to that of Switch Pro controllers ( t ( 17 ) = 2.99 , p = 0.041 < 0.05 , d = 1.60 (Large)).

2.5. Discussion

These findings suggest that more skilled players, such as the advanced or intermediate players in this experiment, produced louder sounds during their game controller operations. This could indicate that the more skillful the player, the quicker they pressed the buttons and pads. In fact, various studies have confirmed and validated that more experienced users are faster and more accurate in operating various interfaces, including hardware and software keyboards, mouse, and styluses [51,52,53,54,55,56,57]. A similar phenomenon may also occur in game controllers. However, such speedy gamepad operations are accurate in experienced players for specific games [58], and there may be a limit to the degree of transference to various types of games. Thus, the external validity of the findings from this study needs to be verified across various genres of games and players with multiple backgrounds.
There are challenges to measuring such pressing motions utilizing image-based measurements as they have proved difficult to obtain for numerous commercial controllers that do not have pressure sensors. From this perspective, measuring the maximum sound pressure level of the controller operation sounds can assist with estimating the pressure exerted when pressing the buttons, as well as the speed at which they are pressed. This can be applied as a new measurement criterion for gaming performance and various interface operations.
Our experimental setup has limitations, particularly in demonstrating only a portion of the effects of the gamepad. In this study, we asked participants to use their familiar game controllers to minimize any artifacts related to unfamiliarity with the device. This approach allowed us to effectively capture the significant main effects of the users’ skill on the maximum sound pressure level. However, we cannot discount the possibility that the findings of this study may be specific to a particular type of gamepad. While the controllers used in this experiment are generally considered consumer standards, further investigation is necessary to confirm our results’ generalizability and explore whether similar trends can be observed with different controllers and within various esports contexts.
Additionally, there is a possibility that the sample size in this study was insufficient. A post hoc power analysis of the ANOVA results, conducted using the Superpower packages (ver. 0.2.0) of the R software [59], indicated that the statistical power exceeded 0.8 for both player skill and the interaction between player skill and gamepad, but not for gamepad alone. When we analyzed the post hoc power of the significant results from the multiple comparisons using the PMCMRplus packages (ver. 1.9.10) of the R software [60], we found that the power exceeded 0.8 whenever a significant difference was observed. Even if the results of this study were insignificant, future studies may not always yield the same consistent results. Further verification is needed to enhance the internal validity of our results across a range of players of various skill levels, using different game controllers.

3. Experiment 2: Differences in Controller Operation Sounds Based on Player Conditions

While experiment 1 focused on the variance in the sound of the controller operated between players, experiment 2 explored the variance in the sound between players and determined the causes. Therefore, the relationship between the condition of the player and the gamepad operation sounds was clarified.

3.1. Participants

Five male university students aged between 18 and 22 years who had played SSBU participated. The player levels of the participants were determined by referring to the methods mentioned in the prior section, with two intermediate players and three advanced players. The experiment was conducted over several days, and beginner players were omitted based on the results presented in Figure 2. We made this exclusion to minimize the learning effect and eliminate any potential artifacts that could have influenced the experiment’s results. All participants utilized the Switch Pro controller for the condition control.

3.2. Method

The same method and setup described in the prior section were employed to obtain the maximum sound pressure level L max by continuously recording the controller operation sounds during the game. In this experiment, the participants were asked to compete against a non-player character for five rounds per day for five days. Concurrently, five different non-player characters were assigned randomly, and the in-game scores of each match were obtained.
Prior to each match, participants were asked to complete the CC-QCSL-45 (Check Catalog for Quality of College Student life 45) [61] and the POMS2 (Profile of Mood States 2nd Edition), Japanese shortened version for adults [62]. After each match, they also answered the WASEDA (Waseda affect scale of exercise and durable activity) [63] and a 5-point evaluation item for the degree of subjective concentration on gameplay, such as “I could concentrate on gameplay”. The QOL score in university life and the TMD (total mood disturbance) score were calculated based on the responses to the CC-QCSL-45 and POMS2, respectively. For the responses to the WASEDA, the scores were calculated for three subscales: negative affect, positive engagement, and tranquility [63].

3.3. Analysis

Before performing the series of analyses, we checked the normality of the data using the Shapiro–Wilk test. We calculated the Pearson’s correlation coefficients r when normality was confirmed, and the Spearman’s rank correlation coefficients ρ when normality was not confirmed. Then, the significance of these coefficients was checked. The significance level was set at 5% and the marginally significant level was set at 10%. Subsequently, for items in which significant variances were found, single regression analyses were utilized to obtain approximate expressions for their relationship with L max . When the L max satisfied or did not satisfy normality, we employed linear regression or robust regression for the MM-type estimators [64,65], respectively. At that time, we employed the Shapiro–Wilk test to check for normality of the data, as mentioned above.

3.4. Results

As the Shapiro–Wilk test confirmed that L max in this experiment did not satisfy the normality assumption ( p = 0.023 < 0.05 ), the following analysis was conducted using Spearman’s correlation coefficient and robust regression analysis. Table 1 lists the correlation coefficients ρ with L max for the subjective evaluation scales and the in-game scores. Notably, there was a significant negative correlation (effect size: large) between pre-experimental mood (QOL and TMD) and L max . Meanwhile, there was no significant correlation between the subjective evaluation results obtained after the match L max . Additionally, the in-game score exhibited a significant negative correlation (effect size: medium) with L max . Note that no significant variation in L max was observed based on the day or the number of experiments.
Figure 3 exhibits the relationship between L max for the three significant items and one non-significant item obtained by robust regression analysis. The pre-game mood indices, such as TMD and QOL scores, as well as the in-game score decreased with increasing sound pressure level, while no explicit relationship was confirmed for the post-game mood. The following regression equations were obtained for the parameters significantly correlated with L max :
QOL = 94.2 − 1.524 L max
TMD = 99.3 − 0.399 L max
In - game score = 93.2 − 0.282 L max
Utilizing these regression equations, it is possible to approximate the pre-game mood of players and the game score to some extent by measuring L max .

3.5. Discussion

The findings suggest that the pre-match mental state may have affected the operation sounds of the controller in particular, and that the operation sound varies depending on the quality of the gameplay. Specifically, for intermediate and advanced esports players, the findings suggest that the sound level increasing and decreasing can correlate with the body condition and in-game results, respectively. Conversely, the post-evaluation items did not differ significantly from the operation sounds as the immersion in the game and in-game performance varied from match to match, which may have affected the mood of the participants. Additionally, no significant correlations were found between the in-game score and post-evaluated mental status obtained by WASEDA. To establish a method for estimating changes in mood post-match, it is necessary to measure and analyze not only the sound of the controls, but also the in-game events after acquiring in-game and operation logs. There are some instances of machine learning models that estimate mood based on players’ voice communication recordings and game logs [34]. Based on these findings, constructing an estimation method utilizing only operation sounds as cues is recommended.
The consistent findings illustrated in this experiment may reflect the fact that participants were recruited with a certain level of experience and skill. In the case of novice players, such consistency is unlikely to emerge due to greater variability. However, to examine what kind of experience is the basis for such consistency, a more detailed examination of the player experience would be required. This could be achieved by measuring the operation sound after investigating the player experience in detail or observing the proficiency trend in the playing situation by controlling the learning process with novices.
In this instance, the scope of application of this method has been described. In this study, the maximum sound pressure level in a low-background-noise environment was measured [17]. In an actual esports event with an on-site venue, the background noise level is extremely high as a result of the sounds from the game, commentary by the event masters, and audience buzzing and roaring. Commercial recorders with highly directional microphones should be utilized to apply the proposed method in such competitions. In the future, the verification of the effectiveness of this method in more realistic settings is required, and if necessary, the method should be improved.
In this study, the sound sources of interest to the gamepad operation sounds were limited. In reality, the sounds produced by players during a game include not only operation sounds but also voices, and there are sometimes cheers from the audience and live broadcasting by masters of ceremonies [17,20,34]. In particular, some studies have reported that the sound communication situation during the game and the atmosphere of the game room are related to gameplay conditions. The study’s experiment did not investigate how these offscreen sounds were related to the controller operation sounds. Understanding the variations in various performance-related metrics such as operation sounds and off-game atmospheric features may provide a more realistic estimation of player conditions.

3.5.1. Limitations

This study focused only on L max per play as an index of operation sound for simplicity of analysis and did not focus on the average noise level of L A eq per play over a certain period of time, nor on sound fluctuations per unit of time of operation sound. For a more detailed analysis and precise estimation of esports player conditions, it is necessary to consider such fluctuation indices of gamepad operation sounds.
Also, our research did not investigate the temporal patterns of the controller sounds. Instead, we concentrated on sound pressure characteristics from the versatility perspective, such as ease of analysis. However, exploring a wider range of participant characteristics, such as skill level, playing experience, and the characters used, may reveal differences in temporal characteristics, particularly when complex controller operations are involved. At this time, temporal characteristics could also reflect variations in controller operation and the effects of fatigue. Therefore, there is potential to uncover a relationship between temporal characteristics of controller operation and the player’s mental state, making this an important area to consider for future research.
We used POMS2 for the pre-evaluation and WASEDA for the post-evaluation of the players’ mental state. However, if we had also used POMS2 for the post-evaluation, we could have made a more direct comparison. We believe that WASEDA is suitable for tracking mood changes related to the experience. Because we used different scales, we might have missed some effects. For future studies, it would be better to design experiments that allow for easier direct comparisons before and after.
The number of participants in the second experiment was small compared to other psychological studies. This was as a result of the limited number of participants who met the study requirements. However, upon examining the correlation coefficients in Table 1, we identified several significant results, with these effect sizes consistently falling between large and medium. For checking the validity of the result, we performed a post hoc power analysis on the correlations with a large effect size using the pwr (ver. 1.3-0) and pwrss (ver. 0.3.1) packages of R [66,67]. We found that the statistical power was greater than 0.8 for the 25 samples collected in this study (5 participants × 5 days = 25 samples). This fact suggests that the internal validity of the correlation between the L max and the QOL or the TMD (POMS2) is somewhat statistically secured. In the results of the robust regression shown in Figure 3, the power analysis based on R 2 showed that the QOL, the TMD (POMS2), and the in-game score all had a power > 0.8, which also indicates that internal validity was secured to some extent, although the sample size was limited. However, even if the correlation and regression analysis results may yield significance and sufficient statistical power, they still have limitations in demonstrating consistent internal validity and establishing a clear causal relationship. Thus, further studies are needed to investigate both the internal and external validities, as well as exact causality, with a larger participant pool. At that time, it would be essential to explore the operation sound features of gaming interfaces with various playing environments (e.g., gamepad, keyboard, etc.), game genres, playing experiences, and players’ characteristics, such as gender and personality traits, in a larger group of esports players with a variety of factors.

3.5.2. Practical Implementations

Finally, we discuss the potential and practical applications of the results obtained in this study. There are two types of event formats in esports competitions: on-site and online [68]. In online competitions, where a quiet environment is somewhat ensured, the local setting where competitors participate acoustically resembles the environment described in this paper. Therefore, our methods can directly be applied to online competition settings and can briefly and indirectly indicate the mental state of the competitors themselves to some extent with objectivity using sound. Also, there have been attempts to display a player’s heart rate and other information online during live gameplay broadcasts. As a result, the techniques presented in this paper could serve to indirectly demonstrate the player’s level of tension to the audience.
On the other hand, when competitions are held in an on-site format, such as in a domed arena, there are likely to be many acoustic factors in the venue, such as the cheers of the audience, the live broadcast of the commentary, and the in-game sound effects. However, most esports players are seated and operating a controller, so they do not move around much. Therefore, our method can work using highly-directional microphones like shotgun microphones and other audio equipment capable of pinpoint recording or extracting keystroke sounds based on frequency characteristics. Given the expected challenges of high background noise and microphone limitations, additional sound extraction techniques and expertise will be needed at each site, highlighting the need for further knowledge development. Also, note that in the case of on-site competitions, the player’s psychological state may be affected not only by the in-game environment but also by external circumstances, so it is necessary to investigate carefully the extent to which keystroke sounds reflect the player’s mental state.

4. Conclusions and Future Work

This study, aimed to establish a simple method for predicting a player’s condition and performance by focusing on the sound of the controller operation in fighting games. This study’s findings can be summarized as follows:
  • Skilled players produce louder sounds during game controller operations as opposed to beginners. This could indicate that the more skillful the player, the quicker they press buttons and pads.
  • The maximum sound pressure level L max of the sound of a player operating a game controller can be utilized for predicting the pre-match mood, including QOL and TMD scores obtained by the CC-QCSL-45 and the POMS2, and the player’s in-game performance.
Future issues include not only field testing of this method at esports competition sites but also studies to develop advanced estimation methods that consider various characteristics of controller operation sounds. Specifically, machine learning models could analyze complex temporal variations in operation sounds using comprehensive feature extraction tools like openSMILE [69,70], potentially improving prediction accuracy beyond conventional manual approaches. Furthermore, when developing a model capable of managing large data sets, it is essential to integrate individual personality traits (e.g., the big five personality traits [71,72,73]), gaming experience, and the type of interface used, while also considering the specific context of the game. By taking such a multifaceted approach, we will construct a prediction model that not only addresses individual differences but also effectively adapts to various game situations, thereby advancing the applicability and robustness of the method in real-world competitive esports scenarios.

Author Contributions

Conceptualization: Y.H., M.U. and T.M.; methodology: Y.H., K.K. and M.U.; validation: T.M., Y.H., T.T. and M.U.; formal analysis: Y.H. and T.M.; investigation: K.K., Y.H., M.U., T.M. and T.T.; resources: M.U. and T.T.; data curation: T.M. and M.U.; writing—original draft preparation: Y.H. and T.M.; writing—review and editing: M.U., K.K. and T.T.; visualization: Y.H. and T.M.; supervision, T.T. and M.U.; project administration: M.U. and T.T.; funding acquisition: M.U., T.M. and T.T. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by JSPS KAKENHI grant numbers: JP21K18485, JP22K02296, and JP24K21471.

Institutional Review Board Statement

This study was conducted with the approval of the Institutional Review Board of Kanagawa Institute of Technology (project code: 20190723-09).

Informed Consent Statement

Informed consent was obtained from all participants.

Data Availability Statement

Data are contained within the article.

Conflicts of Interest

The authors declare that this research was conducted in the absence of any commercial or financial relationships that could be construed as potential conflicts of interest.

References

  1. Famitsu Game Hakusho 2023. 2023. Available online: https://f-ism.net/fgh/2023.html (accessed on 3 January 2025).
  2. Newzoo Global Games Market Report 2020. 2020. Available online: https://newzoo.com/insights/trend-reports/newzoo-global-games-market-report-2020-light-version/ (accessed on 3 January 2025).
  3. Taylor, T.L. Raising the Stakes: E-Sports and the Professionalization of Computer Gaming; MIT Press: Cambridge, MA, USA, 2012. [Google Scholar]
  4. Seo, Y. Electronic sports: A new marketing landscape of the experience economy. J. Mark. Manag. 2013, 29, 1542–1560. [Google Scholar] [CrossRef]
  5. Faust, K.; Meyer, J.; Griffiths, M.D. Competitive and professional gaming: Discussing potential benefits of scientific study. Int. J. Cyber Behav. Psychol. Learn. (IJCBPL) 2013, 3, 67–77. [Google Scholar] [CrossRef]
  6. Apperley, T.H. Genre and game studies: Toward a critical approach to video game genres. Simul. Gaming 2006, 37, 6–23. [Google Scholar] [CrossRef]
  7. Kim, Y.; Ross, S.D. An exploration of motives in sport video gaming. Int. J. Sport. Mark. Spons. 2006, 8, 28–40. [Google Scholar] [CrossRef]
  8. Pizzo, A.D.; Baker, B.J.; Na, S.; Lee, M.A.; Kim, D.; Funk, D.C. eSport vs sport: A comparison of spectator motives. Sport Mark. Q. 2018, 27, 108–123. [Google Scholar]
  9. Bowditch, L.; Chapman, J.; Naweed, A. Do coping strategies moderate the relationship between escapism and negative gaming outcomes in World of Warcraft (MMORPG) players? Comput. Hum. Behav. 2018, 86, 69–76. [Google Scholar] [CrossRef]
  10. Miah, A.; Fenton, A.; Chadwick, S. Virtual reality and sports: The rise of mixed, augmented, immersive, and esports experiences. In 21st Century Sports: How Technologies Will Change Sports in the Digital Age; Springer: Cham, Switzerland, 2020; pp. 249–262. [Google Scholar]
  11. Turkay, S.; Formosa, J.; Cuthbert, R.; Adinolf, S.; Brown, R.A. Virtual Reality Esports-Understanding Competitive Players’ Perceptions of Location Based VR Esports. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Online, 8–13 May 2021; pp. 1–15. [Google Scholar]
  12. Anðelić, B.; Bianco, A.; Maksimović, N.; Todorović, N.; Drid, P. A milestone in the era of esports: The Olympics through the lens of virtual reality. Front. Psychol. 2022, 13, 990189. [Google Scholar] [CrossRef] [PubMed]
  13. Cai, L.; Huang, Z.; Feng, Q.; Chang, X.; Yan, K. Co-transformation of digital health and esport in metaverse: Moderating effects of digital personality on mental health in multiplayer online battle arena (MOBA). Int. J. Environ. Res. Public Health 2022, 20, 760. [Google Scholar] [CrossRef] [PubMed]
  14. Bányai, F.; Griffiths, M.D.; Király, O.; Demetrovics, Z. The psychology of esports: A systematic literature review. J. Gambl. Stud. 2019, 35, 351–365. [Google Scholar] [CrossRef]
  15. Pedraza-Ramirez, I.; Musculus, L.; Raab, M.; Laborde, S. Setting the scientific stage for esports psychology: A systematic review. Int. Rev. Sport Exerc. Psychol. 2020, 13, 319–352. [Google Scholar] [CrossRef]
  16. Pluss, M.; Novak, A.R.; Bennett, K.J.; Panchuk, D.; Coutts, A.J.; Fransen, J. Perceptual-motor abilities underlying expertise in esports. J. Expert. 2020, 3, 133–143. [Google Scholar]
  17. Jang, W.W.; Kim, K.A.; Byon, K.K. Social atmospherics, affective response, and behavioral intention associated with esports events. Front. Psychol. 2020, 11, 551969. [Google Scholar] [CrossRef] [PubMed]
  18. Smith, M.J.; Birch, P.D.; Bright, D. Identifying stressors and coping strategies of elite esports competitors. Int. J. Gaming Comput.-Mediat. Simul. (IJGCMS) 2019, 11, 22–39. [Google Scholar] [CrossRef]
  19. Railsback, D.; Caporusso, N. Investigating the human factors in eSports performance. In Proceedings of the Advances in Human Factors in Wearable Technologies and Game Design: Proceedings of the AHFE 2018 International Conferences on Human Factors and Wearable Technologies, and Human Factors in Game Design and Virtual Environments, Orlando, FL, USA, 21–25 July 2018; Springer: Cham, Swittzerland, 2019; pp. 325–334. [Google Scholar]
  20. Paravizo, E.; de Souza, R.R.L. Playing for real: An exploratory analysis of professional esports athletes’ work. In Proceedings of the 20th Congress of the International Ergonomics Association (IEA 2018) Volume V: Human Simulation and Virtual Environments, Work With Computing Systems (WWCS), Process Control 20; Springer: Cham, Swittzerland, 2019; pp. 507–515. [Google Scholar]
  21. Kou, Y.; Gui, X. Emotion regulation in esports gaming: A qualitative study of league of legends. Proc. ACM Hum.-Comput. Interact. 2020, 4, 1–25. [Google Scholar] [CrossRef]
  22. Poulus, D.; Coulter, T.J.; Trotter, M.G.; Polman, R. Stress and coping in esports and the influence of mental toughness. Front. Psychol. 2020, 11, 521465. [Google Scholar] [CrossRef] [PubMed]
  23. Wong, M.Y.C.; Chung, P.K.; Ou, K.; Leung, K.M. Perception of Hong Kong teenagers and young adults on esports participation: A qualitative study using theory of planned behavior. Front. Psychol. 2021, 12, 650000. [Google Scholar] [CrossRef] [PubMed]
  24. Toth, A.J.; Kowal, M.; Campbell, M.J. The color-word stroop task does not differentiate cognitive inhibition ability among esports gamers of varying expertise. Front. Psychol. 2019, 10, 497707. [Google Scholar] [CrossRef] [PubMed]
  25. Sousa, A.; Ahmad, S.L.; Hassan, T.; Yuen, K.; Douris, P.; Zwibel, H.; DiFrancisco-Donoghue, J. Physiological and cognitive functions following a discrete session of competitive esports gaming. Front. Psychol. 2020, 11, 534802. [Google Scholar] [CrossRef] [PubMed]
  26. Matuszewski, P.; Dobrowolski, P.; Zawadzki, B. The association between personality traits and esports performance. Front. Psychol. 2020, 11, 503779. [Google Scholar] [CrossRef]
  27. Trotter, M.G.; Coulter, T.J.; Davis, P.A.; Poulus, D.R.; Polman, R. Examining the impact of school esports program participation on student health and psychological development. Front. Psychol. 2022, 12, 807341. [Google Scholar] [CrossRef]
  28. Trotter, M.G.; Obine, E.A.; Sharpe, B.T. Self-regulation, stress appraisal, and esport action performance. Front. Psychol. 2023, 14, 1265778. [Google Scholar] [CrossRef] [PubMed]
  29. Al Dafai, S. Conventions within eSports: Exploring Similarities in Design. In Proceedings of the DiGRA/FDG, Dundee, UK, 1–6 August 2016. [Google Scholar]
  30. Dehpanah, A.; Ghori, M.F.; Gemmell, J.; Mobasher, B. The evaluation of rating systems in online free-for-all games. In Advances in Data Science and Information Engineering: Proceedings from ICDATA 2020 and IKE 2020; Springer: Cham, Switzerland, 2021; pp. 131–151. [Google Scholar]
  31. Pradhan, S.; Abdourazakou, Y. “power ranking” professional circuit esports teams using multi-criteria decision-making (mcdm). J. Sports Anal. 2020, 6, 61–73. [Google Scholar] [CrossRef]
  32. Urbaniak, K.; Wątróbski, J.; Sałabun, W. Identification of players ranking in e-sport. Appl. Sci. 2020, 10, 6768. [Google Scholar] [CrossRef]
  33. Nagorsky, E.; Wiemeyer, J. The structure of performance and training in esports. PLoS ONE 2020, 15, e0237584. [Google Scholar] [CrossRef]
  34. Abramov, S.; Korotin, A.; Somov, A.; Burnaev, E.; Stepanov, A.; Nikolaev, D.; Titova, M.A. Analysis of video game players’ emotions and team performance: An esports tournament case study. IEEE J. Biomed. Health Inform. 2021, 26, 3597–3606. [Google Scholar] [CrossRef]
  35. Vizer, L.M.; Zhou, L.; Sears, A. Automated stress detection using keystroke and linguistic features: An exploratory study. Int. J. Hum.-Comput. Stud. 2009, 67, 870–886. [Google Scholar] [CrossRef]
  36. Ghosh, S.; Hiware, K.; Ganguly, N.; Mitra, B.; De, P. Emotion detection from touch interactions during text entry on smartphones. Int. J. Hum.-Comput. Stud. 2019, 130, 47–57. [Google Scholar] [CrossRef]
  37. Sağbaş, E.A.; Korukoglu, S.; Balli, S. Stress detection via keyboard typing behaviors by using smartphone sensors and machine learning techniques. J. Med. Syst. 2020, 44, 68. [Google Scholar] [CrossRef]
  38. Lee, P.M.; Tsui, W.H.; Hsiao, T.C. The influence of emotion on keyboard typing: An experimental study using auditory stimuli. PLoS ONE 2015, 10, e0129056. [Google Scholar] [CrossRef] [PubMed]
  39. Lim, Y.M.; Ayesh, A.; Stacey, M. Continuous stress monitoring under varied demands using unobtrusive devices. Int. J. Hum.-Interact. 2020, 36, 326–340. [Google Scholar] [CrossRef]
  40. Lim, Y.M.; Ayesh, A.; Stacey, M. The effects of typing demand on emotional stress, mouse and keystroke behaviours. In Intelligent Systems in Science and Information 2014: Extended and Selected Results from the Science and Information Conference 2014; Springer: Cham, Switzerland, 2015; pp. 209–225. [Google Scholar]
  41. Lim, Y.M.; Ayesh, A.; Stacey, M. Using mouse and keyboard dynamics to detect cognitive stress during mental arithmetic. In Intelligent Systems in Science and Information 2014: Extended and Selected Results from the Science and Information Conference 2014; Springer: Cham, Switzerland, 2015; pp. 335–350. [Google Scholar]
  42. Pepa, L.; Sabatelli, A.; Ciabattoni, L.; Monteriu, A.; Lamberti, F.; Morra, L. Stress detection in computer users from keyboard and mouse dynamics. IEEE Trans. Consum. Electron. 2020, 67, 12–19. [Google Scholar] [CrossRef]
  43. Subhani, A.R.; Mumtaz, W.; Saad, M.N.B.M.; Kamel, N.; Malik, A.S. Machine learning framework for the detection of mental stress at multiple levels. IEEE Access 2017, 5, 13545–13556. [Google Scholar] [CrossRef]
  44. Zhai, J.; Barreto, A.B.; Chin, C.; Li, C. Realization of stress detection using psychophysiological signals for improvement of human-computer interactions. In Proceedings of the IEEE SoutheastCon, Ft. Lauderdale, FL, USA, 8–10 April 2005; pp. 415–420. [Google Scholar]
  45. Salomao, L.A.T.; Mahfouf, M.; El-Samahy, E.; Ting, C.H. Psychophysiologically based real-time adaptive general type 2 fuzzy modeling and self-organizing control of operator’s performance undertaking a cognitive task. IEEE Trans. Fuzzy Syst. 2016, 25, 43–57. [Google Scholar] [CrossRef]
  46. Cowley, B. Psychophysiology and high-performance cognition—A brief review of the literature. PeerJ Prepr. 2015, 3, e1373v1. [Google Scholar]
  47. Barreto, A.; Zhai, J.; Adjouadi, M. Non-intrusive physiological monitoring for automated stress detection in human-computer interaction. In Proceedings of the Human–Computer Interaction: IEEE International Workshop, HCI 2007, Rio de Janeiro, Brazil, 20 October 2007; Proceedings 4. Springer: Berlin/Heidelberg, Germany, 2007; pp. 29–38. [Google Scholar]
  48. Zhai, J.; Barreto, A.; Chin, C.; Li, C. User stress detection in human-computer interactions. Biomed. Sci. Instrum. 2005, 41, 277–286. [Google Scholar] [PubMed]
  49. Super Smash Bros. Ultimate for the Nintendo Switch System. 2018. Available online: https://www.smashbros.com/en_US/ (accessed on 3 January 2025).
  50. Cohen, J. A power primer. Psychol. Bull. 1992, 112, 155. [Google Scholar] [CrossRef] [PubMed]
  51. Fleetwood, M.D.; Byrne, M.D.; Centgraf, P.; Dudziak, K.; Lin, B.; Mogilev, D. An evaluation of text-entry in Palm OS–Graffiti and the virtual keyboard. In Human Factors and Ergonomics Society Annual Meeting; SAGE Publications: Los Angeles, CA, USA, 2002; Volume 46, pp. 617–621. [Google Scholar]
  52. Sears, A.; Revis, D.; Swatski, J.; Crittenden, R.; Shneiderman, B. Investigating touchscreen typing: The effect of keyboard size on typing speed. Behav. Inf. Technol. 1993, 12, 17–22. [Google Scholar] [CrossRef]
  53. Miura, T.; Yabu, K.I.; Kobayashi, M.; Hiyama, A.; Hirose, M.; Ifukube, T. Learning Processes of Touchscreen Gesture Interaction in Older Adults and Children. In International Conference on Human-Computer Interaction; Springer: Cham, Switzerland, 2023; pp. 125–139. [Google Scholar]
  54. Sutter, C.; Ziefle, M. Psychomotor performance of input device users and optimized cursor control. In Human Factors and Ergonomics Society Annual Meeting; SAGE Publications: Los Angeles, CA, USA, 2006; Volume 50, pp. 742–746. [Google Scholar]
  55. MacKenzie, I.S.; Zhang, S.X.; Soukoreff, R.W. Text entry using soft keyboards. Behav. Inf. Technol. 1999, 18, 235–244. [Google Scholar] [CrossRef]
  56. William Soukoreff, R.; Scott Mackenzie, I. Theoretical upper and lower bounds on typing speed using a stylus and a soft keyboard. Behav. Inf. Technol. 1995, 14, 370–379. [Google Scholar] [CrossRef]
  57. Wood, E.; Willoughby, T.; Rushing, A.; Bechtel, L.; Gilbert, J. Use of computer input devices by older adults. J. Appl. Gerontol. 2005, 24, 419–438. [Google Scholar] [CrossRef]
  58. Baba, D.M. Determinants of video game performance. In Advances in Psychology; Elsevier: Amsterdam, The Netherlands, 1993; Volume 102, pp. 57–74. [Google Scholar]
  59. Lakens, D.; Caldwell, A.R. Simulation-based power analysis for factorial analysis of variance designs. Adv. Methods Pract. Psychol. Sci. 2021, 4, 2515245920951503. [Google Scholar] [CrossRef]
  60. Pohlert, T. PMCMRplus: Calculate Pairwise Multiple Comparisons of Mean Rank Sums Extended. R Package Version 1.9.10. 2023. Available online: https://CRAN.R-project.org/package=PMCMRplus (accessed on 3 January 2025).
  61. Fukumori, H.; Matsushita, T.; Ichimiya, A.; Kajitani, K.; Kumagai, S.; Maruyama, T.; Irie, M.; Nagano, J.; Masaki, Y.; Yamamoto, N.; et al. Development of “The Check Catalogue for Quality of College Student Life 45” for measuring Quality of College Student Life. Jpn. J. Health Promot. 2015, 17, 31–39. [Google Scholar]
  62. Lin, S.; Hsiao, Y.Y.; Wang, M. Test review: The profile of mood states 2nd edition. J. Psychoeduc. Assess. 2014, 32, 273–277. [Google Scholar] [CrossRef]
  63. Arai, H.; Takenaka, K.; Oka, K. Affect scale for acute exercise: Scale development and examination of the exercise-induced affects. Jpn. J. Health Psychol. 2003, 16, 1–10. [Google Scholar] [CrossRef] [PubMed]
  64. Yohai, V.J. High breakdown-point and high efficiency robust estimates for regression. Ann. Stat. 1987, 15, 642–656. [Google Scholar] [CrossRef]
  65. Koller, M.; Stahel, W.A. Sharpening Wald-type inference in robust regression for small samples. Comput. Stat. Data Anal. 2011, 55, 2504–2515. [Google Scholar] [CrossRef]
  66. Champely, S.; Ekstrom, C.; Dalgaard, P.; Gill, J.; Weibelzahl, S.; Anandkumar, A.; Ford, C.; Volcic, R.; De Rosario, H. pwr: Basic Functions For Power Analysis. 2017. Available online: https://cran.r-project.org/web/packages/pwr/ (accessed on 3 January 2025).
  67. Bulus, M. Pwrss: Statistical Power and Sample Size Calculation Tools. Available online: https://CRAN.R-project.org/package=pwrss (accessed on 3 January 2025).
  68. Sjöblom, M.; Macey, J.; Hamari, J. Digital athletics in analogue stadiums: Comparing gratifications for engagement between live attendance and online esports spectating. Internet Res. 2020, 30, 713–735. [Google Scholar] [CrossRef]
  69. Eyben, F.; Wöllmer, M.; Schuller, B. Opensmile: The munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy, 25–29 October 2010; pp. 1459–1462. [Google Scholar]
  70. Eyben, F.; Weninger, F.; Gross, F.; Schuller, B. Recent developments in opensmile, the munich open-source multimedia feature extractor. In Proceedings of the 21st ACM International Conference on Multimedia, Barcelona, Spain, 21–25 October 2013; pp. 835–838. [Google Scholar]
  71. Goldberg, L.R. An alternative “description of personality”: The Big-Five factor structure. In Personality and Personality Disorders; Routledge: London, UK, 2013; pp. 34–47. [Google Scholar]
  72. Roccas, S.; Sagiv, L.; Schwartz, S.H.; Knafo, A. The big five personality factors and personal values. Personal. Soc. Psychol. Bull. 2002, 28, 789–801. [Google Scholar] [CrossRef]
  73. Gosling, S.D.; Rentfrow, P.J.; Swann, W.B., Jr. A very brief measure of the Big-Five personality domains. J. Res. Personal. 2003, 37, 504–528. [Google Scholar] [CrossRef]
Applsci 15 00719 g001
Figure 1. Experimental setup for measuring controller operation sounds.
Figure 1. Experimental setup for measuring controller operation sounds.
Applsci 15 00719 g001
Applsci 15 00719 g002
Figure 2. Difference in maximum sound pressure level L max for (left) player level and (right) game controller.
Figure 2. Difference in maximum sound pressure level L max for (left) player level and (right) game controller.
Applsci 15 00719 g002
Applsci 15 00719 g003
Figure 3. Relationship between L max and (upper left) QOL score, (upper right) TMD score, (lower left) positive engagement score of the WASEDA, as well as the (lower right) in-game score.
Figure 3. Relationship between L max and (upper left) QOL score, (upper right) TMD score, (lower left) positive engagement score of the WASEDA, as well as the (lower right) in-game score.
Applsci 15 00719 g003
Table 1. Spearman’s rank correlation coefficient ρ between L max and each subjective evaluation scale or in-game score. Bold text indicates significant coefficients and large/medium effect size by Cohen’s criterion [50].
Table 1. Spearman’s rank correlation coefficient ρ between L max and each subjective evaluation scale or in-game score. Bold text indicates significant coefficients and large/medium effect size by Cohen’s criterion [50].
Scale/Parameter ρ p
QOL (CC-QCSL-45)−0.570.003**
TMD (POMS2)−0.540.005**
Subjective concentration−0.110.607
WASEDA (negative affect)0.0740.874
WASEDA (positive engagement)−0.0730.560
WASEDA (tranquility)−0.0720.659
In-game score−0.450.021*
**: p < 0.01, *: p < 0.05.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Hiratsuka, Y.; Kuga, K.; Miura, T.; Tanaka, T.; Ueda, M. Sounds and Natures Do Often Agree: Prediction of Esports Players’ Performance in Fighting Games Based on the Operating Sounds of Game Controllers. Appl. Sci. 2025, 15, 719. https://doi.org/10.3390/app15020719

AMA Style

Hiratsuka Y, Kuga K, Miura T, Tanaka T, Ueda M. Sounds and Natures Do Often Agree: Prediction of Esports Players’ Performance in Fighting Games Based on the Operating Sounds of Game Controllers. Applied Sciences. 2025; 15(2):719. https://doi.org/10.3390/app15020719

Chicago/Turabian Style

Hiratsuka, Yamato, Kazuki Kuga, Takahiro Miura, Tetsuo Tanaka, and Mari Ueda. 2025. "Sounds and Natures Do Often Agree: Prediction of Esports Players’ Performance in Fighting Games Based on the Operating Sounds of Game Controllers" Applied Sciences 15, no. 2: 719. https://doi.org/10.3390/app15020719

APA Style

Hiratsuka, Y., Kuga, K., Miura, T., Tanaka, T., & Ueda, M. (2025). Sounds and Natures Do Often Agree: Prediction of Esports Players’ Performance in Fighting Games Based on the Operating Sounds of Game Controllers. Applied Sciences, 15(2), 719. https://doi.org/10.3390/app15020719

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop