Optimized virtual reality design through user immersion level detection with novel feature fusion and explainable artificial intelligence

Introduction

Virtual reality (VR) and immersive technology have revolutionized the way we expose and interact with digital environments (Riches et al., 2023; Jerald, 2015). Virtual reality involves a simulated three-dimensional environment created by computers, which users can navigate and engage with. VR is typically based on wearing a head-mounted display that immerses the user in a virtual world, often accompanied by other sensory inputs such as touch and sound. This technology strives to establish an immersive experience, making the user perceive themselves as being physically located within the virtual setting (Creed et al., 2023; Nasralla, 2021). Immersive technology, on the other hand, encompasses a vast range of technologies that aim to fully engage the user’s senses and create a highly immersive experience. This includes virtual and augmented reality (AR), mixed reality (MR), and other related technologies (Lombard et al., 2015). AR overlays digital content onto the real world, while MR blends virtual and real elements seamlessly. These immersive technologies have found applications in numerous fields, including gaming, entertainment, healthcare, education, and training (Fernandes et al., 2023; Rather et al., 2023), offering exciting possibilities for creating realistic and engaging experiences that were once only limited to our imagination.

VR applications have gained significant attention and popularity in various fields due to their numerous advantages (Ning et al., 2023). VR technology provides an immersive and interactive experience by simulating a realistic three-dimensional environment, enabling users to interact with digital objects and scenarios in real time. One of the critical advantages of VR applications is their ability to enhance training and education. In domains like healthcare, aerospace, and defense, VR enables learners to undergo training for intricate tasks and situations within a secure and manageable setting, thereby diminishing the likelihood of mistakes and mishaps (Creed et al., 2023). Moreover, VR applications have proven valuable in architectural design and urban planning, as they enable architects and urban planners to visualize and explore their designs before the actual construction begins. This leads to better decision-making, improved collaboration, and cost savings. Another advantage of VR applications lies in their potential for therapeutic purposes (Chen, Wu & Huang, 2023). Virtual reality can treat anxiety disorders, phobias, and post-traumatic stress by exposing patients to virtual environments that gradually desensitize them to their fears.

User immersion levels in VR environments play a crucial role in enhancing the design of virtual reality experiences (Valluripally et al., 2023). Immersion refers to the degree to which users feel fully engrossed and connected to the virtual world they are experiencing (Slater & Wilbur, 1997). By understanding and leveraging user immersion levels, designers can create VR environments that provide more captivating and realistic experiences. When users are highly immersed, their cognitive and emotional engagement increases, leading to a heightened sense of presence and an enhanced feeling of being physically present within the virtual environment (Elor & Kurniawan, 2020). This heightened immersion can create a more memorable and impactful VR experience for users. Designers can utilize realistic graphics, interactive elements, spatial audio, and haptic feedback to create a sense of presence and increase immersion levels. By considering user immersion as a critical factor in VR design, designers can create more compelling and engaging virtual reality experiences that effectively transport users to new, immersive digital worlds.

Artificial intelligence techniques have been increasingly employed to detect and assess user immersion levels in VR environments (Hong & Ge, 2022). By leveraging machine learning algorithms, researchers have sought to capture and analyze various user behavioural and physiological signals to gauge the extent of immersion experienced by individuals within the VR setting (Wu & Han, 2023). These techniques encompass a range of approaches, including computer vision, natural language processing, and sensor data analysis. Computer vision algorithms enable the tracking and analyzing of user movements, facial expressions, and eye gaze patterns, providing valuable insights into the user’s engagement and presence in the virtual environment (Ahuja et al., 2023). Integrating artificial intelligence techniques holds great promise for advancing our understanding of user experiences in VR environments and informing the design of immersive virtual experiences that captivate and engage users to a greater degree.

The primary research contributions related to virtual reality are followed as:

• A novel Polynomial Random Forest (PRF) method is proposed, which extracts polynomial and class prediction probability features from the virtual reality experience benchmark dataset, generating a new feature set.

• Advanced deep and machine learning methods are employed for comparisons. K-fold validation, hyperparameters optimization, and computational complexity analysis are conducted to validate the performance.

• An XAI approach is employed to interpret the reasoning behind the decisions made by the proposed model for detecting the user’s immersion level in VR.

The remaining research study is categorized as ‘Literature analysis’ comparatively analyzed the literature for virtual reality applications. Our proposed study methodology is demonstrated in ‘Proposed methodology’. ‘Results and discussion’ is based on the results of applied artificial intelligence approaches. Our study findings are concluded in ‘Conclusions and future work’.

Literature analysis

Literature analysis in VR involves conducting a comprehensive examination of existing research studies. Through a systematic review of the available literature, we can better understand the current state of knowledge, identify research gaps, and uncover valuable insights. The scope of literature analysis on VR applications is broad and encompasses various topics such as gaming, education, healthcare, architecture, and training simulations. The literature summary is provided in Table 1.

In Suhaimi, Mountstephens & Teo (2022), the authors proposed a virtual reality environment aimed at inducing four classes of emotions: happy, sad, terrified, and bored, for the purpose of classifying human emotions. The dataset used in this study comprised a total of 32 subjects, including seven females and 25 males. The VR video dataset was sourced from participants on YouTube and offline recordings, and brain signals were collected using an EEG handset. Various classical machine learning models were applied in this study. The research results indicate that the proposed support vector machine model achieved an accuracy score of 85%, which is considered low.

The research proposed by Cha & Im (2023) applies covariate shift adaptation approaches in the feature and classifier domains to build an fEMG-based FER system that is resistant to electrode shifts. This system is designed for use in VR-based metaverse applications (Saba et al., 2019), enabling reliable facial expression identification even when the electrode placements shift. In this experiment, the fEMG dataset is used, which records eleven expressions from different subjects. The study results demonstrate that without using the suggested method, the classification accuracy decreased from 88% to 79% when the electrode placements were changed. However, when the suggested covariate shift adaptation method was employed, the accuracy dramatically increased to 86%. Although this improvement is significant, it remains relatively low compared to the baseline accuracy.

In Xiao (2023), the authors proposed a unique feature fusion technique for facial recognition in VR. This approach was developed because the conventional method’s face expression detection algorithm (Meethongjan et al., 2013) performs poorly in real-world scenarios. In the experimental study, the authors utilized the FER2013 dataset and extracted convolutional features from VGG16 and ResNet50 networks. These features were then combined with HOG features and classified using support vector machine (SVU) (Sharif et al., 2019). The initial classification accuracy achieved by SVM was 67%. To further enhance the accuracy, the authors employed a GAN network and achieved an accuracy of 72%. However, it is important to note that the performance scores of this study are relatively low.

Cha & Im (2022) proposed to create a new technique for enhancing facial expression recognition (FER) performance by combining labelled datasets from other users. In this study, experiments were conducted using systems for social VR applications based on the fEMG dataset to classify eleven different facial expressions. The results of the study indicate that an accuracy of 89.4% could potentially be achieved by applying the linear discriminant analysis (LDA) adaptation technique. It is important to mention that the level of accuracy achieved does not surpass that found in existing studies.

This research was proposed by Pei et al. (2023) to investigate how EEG signals can be used to distinguish between different emotional states in VR scenarios and to enhance the computational speed and accuracy of emotional valence identification. In this experiment, emotional VR videos, which are largely typical, were used to collect subjects’ emotions. Radial basis function neural networks (RBFNN) and DBF were employed to classify the model accuracy. The study results indicate that the proposed RBFNN achieved the highest accuracy of 91%, which is considered low.

Mateos-García et al. (2023) proposed research to create a system for identifying stress during various driving circumstances in a car by combining the usage of sensors and the opportunities provided by VR. In this experiment, a total of 8 subjects (four male and four female) aged between 24 to 50 years were recruited. The dataset used in this study is titled ”Stress Recognition in Automobile Drivers” and was produced in 2007. It is freely accessible on the PhysioNet platform. Different advanced machine learning models were applied to determine the accuracy, and the model that achieved the highest accuracy was the Decision Tree (DT), which achieved a 90% accuracy rate. However, further improvements are still needed.

Uyanık et al. (2022) proposed an improved method for EEG-based emotion detection using the recently released VREED dataset, which is publicly available. The experiment involved 15 male and female subjects who were exposed to a total of 60 VR videos, each lasting 4 s, belonging to three different categories. All videos were shot in 3D by the Shanghai Film Academy. Various machine learning models were employed to achieve the highest accuracy using differential entropy (DE) features in different classifiers. The model that yielded the highest accuracy was SVM, which achieved a result of 76.22% with a computation time of 2.06 s. However, these results are relatively low and demonstrate a high error rate.

This work was proposed by Geraets et al. (2021) was to measure and develop the ability to recognize emotions. VR permits the administration of dynamic, realistic stimuli within a social situation. In this study, twenty avatars were dispersed throughout the virtual street area. Each avatar displayed emotions, including anger, disgust, fear, happiness, sorrow, surprise, or neutrality, for 10 s whenever a participant moved within a two-meter range of the avatar. The VR activity involved rating the emotions of virtual characters (avatars) in a VR street scene and recording eye-tracking in VR. After this experiment, VR achieved a total recognition accuracy of 75%, matching that of the photo and video task.

Bulagang, Mountstephens & Teo (2021) aimed to test if heart rate (HR) signals can be utilized in a virtual reality (VR) environment to classify four different moods using Russell’s emotion model. In this experiment, a total of 20 subjects (12 male and eight female), ranging in age from 20 to 28 years, participated. The Emotiv E4 wristband sensor was employed to record heart rate data, while a VR headset was used to present videos to the subjects. According to the experimental findings, the accuracy of intra-subject emotion classification across the four classes ranged from 45.4% to 100%. These findings demonstrate the promising potential of HR as an approach for emotion classification in a four-class setting using VR, particularly in predicting emotions for participant-specific data. Intra-subject classification yielded accuracy rates of over 80% using SVM and KNN algorithms.

Zheng, Mountstephens & Teo (2020) proposed a method for recognizing emotions in a virtual environment using pupillometry. The experiment involved using the Pupil Labs app to collect eye-tracking data. As part of the data collection process, the participants’ eyes were calibrated. Initially, all participants wore a VR headset with an eye-tracking attachment. Pupil Capture was utilized to record the students’ data. The experiment stimuli consisted of videos presented in 360 degrees while wearing the VR headset. The pupil diameter was measured using an eye tracker and used as the sole criterion for emotion classification. Three advanced machine-learning techniques were employed for emotion classification. Among all the models, SVM achieved the highest average accuracy of 57%. However, the accuracy for properly identifying emotions from the LA/NV quadrant was relatively poor, around 70%.

The presence of user immersion in VR can be determined in various ways (Gougeh et al., 2022). VR headsets like the HTC Vive, Oculus Rift, and PlayStation are employed to quantify immersion levels. The duration of time users spend in a virtual environment, as well as the occurrence of motion sickness, serves as a self-assessment of their susceptibility to motion sickness while engaging in VR (Kim, 2020). Higher ratings on the scale, which spans from 1 to 10, suggest a higher likelihood of experiencing motion sickness.

Proposed methodology

This section analyzes our proposed research methodology for detecting user immersion levels in virtual reality environments. The step-wise flow of the proposed study is illustrated in Fig. 1. In our study, we collected virtual reality data based on user experiences to conduct experiments. We performed a descriptive and distributional analysis of the data, focusing on feature states. Using a novel PRF method, we extracted polynomial and class prediction probability features from the virtual reality experience data, resulting in a new feature set. This newly generated feature set is then partitioned into training and testing sets in an 80:20 ratio. We employed advanced deep and machine-learning techniques for the newly created data. The performance scores of the superior AI model were evaluated using unseen testing data. Finally, an efficient AI model was developed to detect user immersion levels and enhance the design of virtual reality environments.

Virtual reality experiences data

The virtual reality benchmark dataset (Joshi, 2023) based on user experience is used in our study to conduct experiments. The dataset collection involved 1,000 subjects. This dataset contains values that are derived from people’s physiological reactions and feelings while using VR applications. This benchmark dataset includes information about the users’ preferences and dislikes, as well as details about their heart rate, skin conductance, and moods during virtual reality applications. Table 2 presents the results of the descriptive analysis of the dataset. Additionally, we conducted a data distribution analysis, as depicted in Fig. 2.

Table 2:

The descriptive benchmark dataset features analysis.

Feature name	Data type	Description
User ID	int 64	Each user participating in the VR experience has a specific identifier represented by this feature. To distinguish the data of each user in the dataset, it assigns them a unique ID.
Age	int 64	This feature represents the age of the subjects who participated in VR as an integer value.
Gender	object	This feature identifies the gender of the subject, distinguishing between male and female.
VRHeadset	Object	This feature allows us to determine the type of VR headset the user uses during their VR experience. It could be a VR headset such as the HTC Vive, Oculus Rift, PlayStation VR, or any other type.
Duration	float 64	This feature indicates the amount of time subjects spend in a virtual environment, measured in minutes.
Motion sickness	Int 64	This feature reflects the user’s self-assessment of their susceptibility to motion sickness while engaging in virtual reality (VR). Higher ratings on the scale, which ranges from 1 to 10, indicate a greater risk of experiencing motion sickness.
Immersion level	Int 64	This targeting feature captures the extent to which users felt immersed in the VR experience. It demonstrates users’ engagement and involvement with the VR experience. The user’s level of immersion in the virtual environment is measured on a scale of 1 to 5, with 5 representing total immersion.

DOI: 10.7717/peerjcs.2150/table-2

Novel proposed approach

The working architecture flow of our novel PRF proposed approach is analyzed in this section. The feature generation mechanism is illustrated in Fig. 3. To begin with, the virtual reality data based on user experience is input to a polynomial function for extracting polynomial features (Sukhbaatar et al., 2023), and to a random forest approach for extracting class prediction probability features (Raza et al., 2022b). Then, the features extracted from both approaches are fused together to generate a new feature set. Finally, the newly generated feature set is used to conduct the proposed study experiments.

Polynomial features

The polynomial features are designed to expand the feature space of input data by generating polynomial and interaction features. It creates a new feature matrix where each feature is transformed into all polynomial combinations up to a specified degree. This transformation allows the model to capture non-linear interactions between features, potentially leading to more accurate predictions.

Class prediction probability features

The class prediction probability features compute the probabilities based on the average outputs from all the trees within the random forest model. Specifically, for each tree in the forest, the probability assigned to a particular class for a given sample is calculated as the proportion of training samples of the same class that fall within the same leaf as the input sample. The random forest provides a robust estimation of class probabilities by averaging these individual probabilities across all trees, which is helpful for more accurate predictions

Algorithm 1 outlines the sequential process of the suggested method for combining features.

 
_______________________ 
Algorithm 1 PRF Algorithm_______________________________________________________________ 
Input:  Virtual reality data based on user experience. 
Output:  Novel fused features for detecting user immersion levels. 
initiate; 
1-  Fpol   ←− PoLpolynomial features(V d)     // here V d belong to the Virtual 
reality data and Fpol are predicted features. 
2-  Frf  ←− RFprobability features(V d)   // here V d belong to the Virtual reality 
data and Frf are predicted features. 
3- NFfeatures  ←−∑ 
   {Fpol + Frf}  // NFfeatures ϵ  Novel fused features set 
used for detecting user immersion levels. 
end;__________________________________________________________________________________________________    

Results and discussion

The results and discussion of our research on machine learning-based enhancing virtual reality environment design through user immersion level detection are presented in this section. To validate the effectiveness of our proposed research approach, we conducted a series of experiments using a diverse set of virtual reality scenarios.

Results with novel features

Finally, we evaluated applied deep and machine learning approaches with novel features in this analysis. The performance of various machine learning methods was examined using a novel feature set, and the results are described in Table 8. The evaluation of the applied methods included measures such as recall, precision, and F1 score. The analysis revealed that the SVM method exhibited the lowest accuracy performance score of 0.26 when compared to other methods. On the other hand, the GRU and LR methods achieved good accuracy scores of 0.94 and 0.97, respectively. Surprisingly, the RF approach outperformed the others with a high accuracy score of 0.98 when using the novel feature set. The class-wise performance report demonstrated that the proposed RF approach achieved a precision, recall, and F1 score of 0.99 for each. Based on this research analysis, we conclude that the novel feature set proposed in this study yields excellent results.

Table 7:

Performance comparison analysis of applied approaches with PCA features.

Technique	Accuracy	Target class	Precision	Recall	F1
RF	0.23	1	0.23	0.27	0.25
		2	0.21	0.23	0.22
		3	0.22	0.26	0.24
		4	0.26	0.25	0.26
		5	0.18	0.12	0.14
		Average	0.22	0.23	0.22
LR	0.20	1	0.27	0.35	0.31
		2	0.16	0.26	0.20
		3	0.17	0.20	0.18
		4	0.21	0.21	0.21
		5	0.00	0.00	0.00
		Average	0.16	0.20	0.18
SVM	0.21	1	0.21	0.51	0.29
		2	0.14	0.10	0.12
		3	0.23	0.43	0.30
		4	0.30	0.06	0.10
		5	0.20	0.02	0.04
		Average	0.22	0.23	0.17

DOI: 10.7717/peerjcs.2150/table-7

Table 8:

Performance comparison analysis of applied methods with novel proposed features.

Technique	Accuracy	Target class	Precision	Recall	F1
RF	0.98	1	1.00	1.00	1.00
		2	1.00	0.95	0.97
		3	0.97	1.00	0.99
		4	0.98	0.98	0.98
		5	0.98	1.00	0.99
		Average	0.99	0.99	0.99
LR	0.97	1	1.00	1.00	1.00
		2	1.00	0.92	0.96
		3	0.92	1.00	0.96
		4	0.98	0.98	0.98
		5	0.98	0.98	0.98
		Average	0.98	0.98	0.97
SVM	0.26	1	1.00	0.16	0.28
		2	0.21	1.00	0.34
		3	1.00	0.06	0.11
		4	1.00	0.08	0.15
		5	0.00	0.00	0.00
		Average	0.64	0.26	0.18
GRU	0.94	1	0.88	0.97	0.92
		2	0.97	0.92	0.95
		3	0.97	1.00	0.99
		4	0.94	0.98	0.96
		5	0.97	0.85	0.91
		Average	0.95	0.95	0.94

DOI: 10.7717/peerjcs.2150/table-8

The time series results analysis during the training of the applied neural network approach, GRU, is illustrated in Fig. 4. The analysis demonstrates that during the first four epochs of training, low-performance scores are achieved with high-loss scores. However, after epoch 5, the GRU model extracted the optimal weights and enhanced the performance score. From epochs 7 to 8, the model accuracy ranged from 90% to 94%. This analysis wrap up that while the neural network technique achieved acceptable scores, it did not achieve the highest scores.

The confusion matrix results analysis of the applied method using novel features is demonstrated in Fig. 5. This analysis validates the performance of each applied method on unseen testing data. The analysis shows that the RF technique achieves the lowest error rate during classification, followed by LR and GRU. In comparison, the SVM technique achieved a higher error rate. Based on this research analysis, it can be summed up that the proposed RF techniques achieved the minimum error rates, thus validating their high-performance scores.

Explainable AI analysis

XAI analysis (Albahri et al., 2023) has emerged as a critical area of research aimed at enhancing the transparency, interoperability, and decision-making processes of proposed AI models. One XAI analysis approach utilizes the SHAPE (Significance, Heterogeneity, Accuracy, Pragmatism, and Explanation) chart, as illustrated in Fig. 6.

The SHAPE chart in this analysis provides a structured framework for evaluating and understanding the strengths and limitations of our proposed AI model’s explanations. The analysis demonstrates that the newly created features (f8, f10, f11, f7, and f9) have high relevance when predicting user immersion levels in VR environments. This analysis enhances the interoperability of our proposed RF technique during decision-making processes for unseen testing data.

Conclusions and future work

This study aims to detect user immersion levels in VR using an efficient AI model. We utilized data on user experiences in VR environments to conduct our experiments. We proposed a novel technique called PRF for feature engineering mechanisms. The PRF approach extracts polynomial and class prediction probability features to generate a new feature set. Advanced machine and deep learning methods were employed for comparisons. In-depth experimental investigations have demonstrated that the RF technique, leveraging the newly proposed PRF method, surpassed contemporary studies, attaining a remarkable performance metric of 98%. The K-fold validation, hyperparameters optimization, and computational complexity analysis approaches were applied to validate the performance scores. Additionally, we utilized the XAI approach to interpret the reasoning behind the decisions made by the proposed model for user immersion level detection in VR.

Supplemental Information