Keywords

1 Introduction

Virtual reality (VR) is a technology that enables to create a fully three-dimensional computer-generated environment in which a person can move around as well as interact as if he/she actually were in the virtual space [1]. VR has made great strides in the past 20 years, having great potential as sites for research in social, behavioral and economic sciences, as well as in human-centered computer science [2]. Because VR enables to afford a user to walk through a computer-generated environment that can be controlled to assess hypotheses that are hard to examine systematically in the real world [3].

According to the recent technology, it is possible to develop highly immersive and presence evoking environment; that is, a head-mounted display (HMD) based virtual environment. In this immersive virtual environment, user’s viewpoint is fixed on the eyes of a virtual body which substitute a user’s own biological body with synchronous vasomotor feedback, so that when the user moves in the real world, his/her virtual body moves in the virtual world in real time and synchronously [4].

1.1 The Sense of Embodiment in Virtual Reality

The sense of embodiment (SoE) is an essential component of user experience in immersive virtual environments through the embodied virtual body. The SoE consists of three subcomponents such as the sense of self-location, the sense of agency, and the sense of ownership [5]. The sense of self-location is a determinate volume in space where a user feels located. The self-location, as well as body-space normally coincide in the sense that a user feels self-located inside the user’s own biological body [6]. The sense of agency is a sensation defined as “global motor control, including the subjective experience of action, control, intention, motor selection and the conscious experience of will” [7]. The sense of ownership defined as a user’s self-attribution of a body, influencing by morphological similarities as well as by spatial correlation of the body and so on [8].

To enhance the SoE would be to enhance each of its three subcomponents in order to design more immersive experience with an embodied virtual body in the virtual environment. In this line of the enhancement, it is essential to measure the effect of these subcomponents on different factors.

1.2 The Measurement of the Sense of Embodiment

The measurement of the SoE including its subcomponents usually relies on questionnaires or physiological responses [6, 9, 10]. However, there is no explicit measure of the SoE as well as its subcomponents for the moment, including performance-based as well as behavioral-based measurement.

This study aims to explore the nature of difference on performance as well as on behavior in a different kind of information representation methods such as real environment, VR environment, and Mixed Reality environment to identify some features that enable to be applied for performance-based/behavioral-based measurement for the characterization of the SoE and its subcomponents.

2 Methods

2.1 Participants

Eleven students were recruited from our University. All of the participants were male whose mean age was 21 years (SD = 0.31). All participants had little or no experience with Virtual Reality as well as Mixed Reality application.

2.2 Apparatus

Visual Display/Rendering.

HTC Vive head-mounted display was used for the experimental task. The Vive consists of a headset, two controllers, and two infrared laser emitter units. The headset covers a nominal field of view about 110\( ^\circ \) through two 1080 × 1200 pixel display that is updated at 90 Hz. As for the visual rendering, Unity3D was employed, rendering the graphics at 60 Hz.

3D Motion Tracker.

HTC Vive tracker was used for the experimental task. The Vive tracker is a motion tracker that enables to measure its 3D position as well as orientation in real time. The field of view of the tracker is 270\( ^\circ \). The Tracker weighs is 85 g and the size is 99.65 mm (Diameter) × 42.27 mm (Height).

Data Acquisition.

AIO-160802AY-USB was used for the data acquisition in the experiment. AIO-16080AY-USB is a high-precision analog I/O terminal which has analog input (16 bit, 8ch), and analog output (16 bit, 2ch). The maximum conversion speed of the terminal is 10 \( {\mu s}. \)

Real Task Environment.

A custom experimental task called a “rod tracking task” was developed for this study. The developed experimental environment had a square wave-shaped slit on a paper board made by grid paper to provide a path for a rod tracking by a participant (See Fig. 1). The system captured data regarding trial time; whether the user’s rod was touching a wall along the slit; the vive tracker position as the rod location in \( \text{x},\,\text{y},\,\text{and}\,\text{z} \) coordinates.

Fig. 1.
figure 1

A rod tracking task in a real environment. A LED light turns on when the rod collides with a wall along the slit. The applied voltage was measured on the AIO-16080AY-USB.

Full size image

Virtual Reality-Based Task Environment.

A rod tracking task using VR representation was developed based on the real task environment, utilizing HTC Vive head mounted display (See Fig. 2). The real task environment was 3D reconstructed in a virtual environment. The virtual environment was visually represented through the HMD. However, the paper board in the virtual environment was not haptically represented. The system captured data regarding trial time; whether the user’s rod was touching a wall along the slit in the virtual environment; the vive tracker position as the rod location in \( \text{x},\,\text{y},\,\text{and}\,\text{z} \) coordinates.

Fig. 2.
figure 2

A rod tracking task in a VR-based environment.

Full size image

Mixed Reality-Based Task Environment.

A rod tracking task using MR representation was developed based on the real task environment, utilizing HMD as well as the paper board in the real task environment (See Fig. 3). The real task environment was 3D reconstructed in a virtual environment. The position of the paper board in the real environment was calibrated to the position of the paper board in the virtual environment; that is, the paper board was visually represented in the virtual environment through the HMD, as well as was haptically represented in the real environment. The system captured data regarding trial time; whether the user’s rod was touching a wall along the slit in the virtual/real environment; the vive tracker position as the rod location in \( \text{x},\,\text{y},\,\text{and}\,\text{z} \) coordinates.

Fig. 3.
figure 3

A rod tracking task in a MR-based environment.

Full size image

2.3 Experimental Task

The protocol for the experimental task was identical for all conditions. A start point on the left top of the path on the paper board is touched with a rod grabbing by a participant for three seconds. After counting three seconds, the rod would be moved by the participant to the terminal point on the right bottom of the path on the paper board (trial 1). When it reaches the terminal point, the participant would stay for three seconds, then move back to the start point (trial 2). These trials would be repeatedly conducted for three times; that is, six trials would be performed by participants in total. Participants were tasked with moving the rod from the start point to the terminal point as quickly and accurately as possible, as well as much as possible without touching the wall along the slit with the rod.

2.4 Experiment Design and Independent Variables

To examine the effect of behavioral strategy on performance as well as on behavior in the custom task environments such as real task environment, VR task environment, and MR task environment, a balanced 2 × 3 within subjects factorial experiment design was used, which result in 6 experimental tasks. A summary of the independent variables is shown in Table 1.

Table 1. Summary of Independent variables.
Full size table

Behavioral Strategy.

Behavioral strategy for the experimental task was designed to assist two purposes: (1) to control the participant’s intention to move the rod accurately on the center line of the path, relating body-image construction regarding the correct physical motion, (2) not to provide any cues for controlling the participant’s intention regarding physical motion (non-body-image construction) so that participants are required to conduct the task in standard manner of the experimental task; that is, with moving the rod from the start point to the terminal point as quickly and accurately as possible, as well as much as possible without touching the wall along the slit with the rod. These concepts were illustrated in Fig. 4.

Fig. 4.
figure 4

The two behavioral strategy applied in this study; (a) Participants were required to imagine the straight line on the center of the path, as well as to move on the line as accurately as possible. (b) There is no cue regarding the body-image construction.

Full size image

Information Representation Method.

The three distinct representation method was applied such as Real task environment, VR-based task environment, and MR-based task environment.

2.5 Dependent Variables

To systematically investigate the effect of behavioral strategy in the custom task environments, this study used several dependent measures, which can be categorized into two types of variables: (a) task performance including task completion time and, a probability distribution of time-to-collision; (b) user behavior.

Task Performance.

Task performance includes task completion time, and a probability distribution of time-to-collision. Task completion time was defined as the time from when participants start the first trial to the moment they finish the last trial except for the time for the three seconds counting on each start/terminal point. A probability distribution of time-to-collision is a probability distribution of the time to collision with the wall on the path, which was characterized based on the Gamma distribution (See Fig. 5). The distribution can be represented by the Eq. (1).

Fig. 5.
figure 5

The concept of the probability distribution of time-to-collision.

Full size image
$$ f\left( \varvec{x} \right)\, = \,\frac{1}{{\varGamma \left( k \right)\theta^{k} }}\varvec{x}^{k - 1} e^{{ - \frac{\varvec{x}}{\theta } }} \quad (\varvec{x} > 0) $$
(1)

The gamma distribution is continuous probability distribution which has two parameters such as a shape parameter \( k \), as well as scale parameter \( \theta \). These parameters were estimated using the maximum likelihood estimation method, as well as was used as dependent variables.

User Behavior.

A user’s behavior interacting with the task was analyzed using Singular value decomposition (SVD) method. \( \varvec{x}_{n}^{i} \in R^{N} \) is a time series data of the position of the rod in \( n{\text{th}} \) trial. A set of trial data for subject \( i \) is represented by \( \varvec{X}^{i} \) described as

$$ \varvec{X}^{i} \, = \,\left( {\left\{ {\varvec{x}_{1}^{i} } \right\}^{T} \left\{ {\varvec{x}_{2}^{i} } \right\}^{T} \cdots \left\{ {\varvec{x}_{n}^{i} } \right\}^{T} } \right)^{T} $$
(2)

where \( n \) is the number of trials in the experimental task. A set of behavior data represented by \( \varvec{X}^{i} \) can be formed as matrix \( D \) (See Eq. (3)), which represents matrix for subject group. The number of the data point of each \( \varvec{X}^{i} \) was standardized using interpolation technique.

$$ D\, = \,\left( {\varvec{X}^{1} \varvec{ X}^{2} \cdots \varvec{X}^{M} } \right) $$
(3)

The singular value decomposition (SVD) method was applied to the matrix \( D \) in order to extract similarities of behaviors within the subject group. As a result of the SVD, the matrix \( D \) can be decomposed as follows:

$$ D\, = \,U\,\varSigma \,V^{T} $$
(4)

Where \( U \) is a unitary matrix which has left singular vector \( \varvec{u}_{i} \in R^{n \cdot N} \) as its elements. The matrix V is a unitary matrix which has right singular vector \( \varvec{v}_{i} \in R^{M} \) as its elements. The matrix \( \Sigma \) is a diagonal matrix which has a set of singular values \( \upsigma_{i } \left( {i\, = \,1,2, \cdots , M} \right) \) as its diagonal elements. The value of similarities was decomposed as the \( \upsigma_{i } \) so that the behavior of individual \( \varvec{X}^{i} \) can be represented using left singular vector \( \varvec{u}_{i} \), and right singular vector \( \varvec{v}_{i} \) as follows:

$$ \varvec{X}^{i} \, = \,\mathop \sum \limits_{j = 1}^{n}\upsigma_{j} v_{i,j} \varvec{u}_{i} \varvec{ } $$
(5)

If the value of 1st singular value \( \sigma_{1} \) is high, it indicates that decomposed time series data are similar; that is, participants did almost same physical motion in all of the trials. The similarity based on the singular value was defined as a dependent variable.

2.6 Procedure

Participants were required to read and signed an informed consent. After completion of the introduction paperwork, short training was conducted to familiarize participants with interacting with the custom experimental environment. Participants were informed of their goal to move as quickly and accurately in each trial.

Following the training session, the experimental tasks began. Each participant completed 6 tasks in a randomized order, with each task consisting of 6 trials.

2.7 Research Question

This study seeks to examine the effect of different types of behavioral strategy on performance as well as on behavior in three types of different information representation method such as real task environment, VR-based task environment, and MR-based task environment. We assumed that the result from the experiment in the real environment could be defined as the normal state of the SoE. The result from the all the other data in the virtual environment would be compared to the result from the real environment in order to identify some features that enable to be applied for performance-based/behavioral-based measurement for the characterization of the SoE and its subcomponents.

3 Results

3.1 Task Performance

Task Completion Time.

A 2 × 3 ANOVA with Behavioral Strategy (Body-image, None body-image) and Representation method (RL, VR, MR) as within-subject factors revealed a main effect on Representation method, \( \;F\left( {2,20} \right)\, = \,5.017,\,p\, < \,.038,\,\eta_{p}^{2} \, = \,.334 \). A post hoc comparison (Ryan’s method,α = .05) indicated that there is significant difference between RL (\( M\, = \,84.76\,{\text{s}},\,SD\, = \,8.09\,{\text{s}} \)) and MR (\( M\, = \,97.98\,{\text{s}},\,SD\, = \,2.96\,{\text{s}} \)) condition. However, there is no significant difference between VR and MR condition, as well as between RL and VR condition. The main effect was not qualified by an interaction between Representation method and Behavioral strategy, \( F\left( {2,20} \right)\, = \,1.001,\,p\, < \,.355,\,\eta_{p}^{2} \, = \,.091 \). The results showed in Table 2.

Table 2. The result of the ANOVA on the task completion time.
Full size table

Table 3 showed the result of descriptive statistics for the task completion time.

Table 3. The descriptive statistics for the task completion time.
Full size table

The descriptive plot for the time completion time was illustrated in Fig. 6.

Fig. 6.
figure 6

The descriptive plot for the time completion time. The error bars represent standard error of means.

Full size image

Shape Parameter of the Gamma Distribution.

A 2 × 3 ANOVA with Behavioral Strategy (Body-image, None body-image) and Representation method (RL, VR, MR) as within-subject factors revealed a main effect on Representation method, \( F\left( {2,20} \right)\, = \,6.327,\,p\, < \,.025,\,\eta_{p}^{2} \, = \,.388 \). A post hoc comparison (Ryan’s method,α = .05) indicated that there is significant difference between VR (\( M\, = \,10.52,\,SD\, = \,0.95 \)) and MR (\( M\, = \,2.19,\,SD\, = \,0.31 \)) condition, as well as between RL (\( M\, = \,2.63,\,SD\, = \,1.49 \)) and VR (\( M\, = \,10.52,\,SD\, = \,0.95 \)) condition. However, there is no significant difference between RL and MR condition. The main effect was not qualified by an interaction between Representation method and Behavioral strategy, \( F\left( {2,20} \right)\, = \,.277,\,p\, < \,.648,\,\eta_{p}^{2} \, = \,.027 \). The results showed in Table 4.

Table 4. The result of the ANOVA on the shape parameter of the Gamma distribution.
Full size table

Table 5 showed the result of descriptive statistics for the shape parameter of the Gamma distribution.

Table 5. The descriptive statistics for the shape parameter of the Gamma distribution.
Full size table

The descriptive plot for the shape parameter of the Gamma distribution was illustrated in Fig. 7.

Fig. 7.
figure 7

The descriptive plot for the shape parameter of the Gamma distribution. The error bars represent standard error of means.

Full size image

Scale Parameter of the Gamma Distribution.

A 2 × 3 ANOVA with Behavioral Strategy (Body-image, None body-image) and Representation method (RL, VR, MR) as within-subject factors revealed a main effect on Representation method,\( F\left( {2,20} \right)\, = \,4.808,\,p\, < \,.027,\,\eta_{p}^{2} \, = \,.325 \). A post hoc comparison (Ryan’s method,α = .05) indicated that there is significant difference between VR (M = 3.19, SD = 0.54) and MR (M = 5.25, SD = 0.39) condition, as well as between RL (M = 5.14, SD = 0.59) and VR (M = 3.19, SD = 0.54) condition. However, there is no significant difference between RL and MR condition. The main effect was not qualified by an interaction between Representation method and Behavioral strategy, \( F\left( {2,20} \right)\, = \,1.456,\,p\, < \,.259\,\eta_{p}^{2} \, = \,.127 \). The results showed in Table 6.

Table 6. The result of the ANOVA on the scale parameter of the Gamma distribution.
Full size table

Table 7 showed the result of descriptive statistics for scale parameter of the Gamma distribution.

Table 7. The descriptive statistics for the scale parameter of the Gamma distribution.
Full size table

The descriptive plot for the scale parameter of the Gamma distribution was illustrated in Fig. 8.

Fig. 8.
figure 8

The descriptive plot for the scale parameter of the Gamma distribution. The error bars represent standard error of means.

Full size image

3.2 User Behavior

Cumulative Contribution Ratio for Singular Value (Mode 1).

A 2 × 3 ANOVA with Behavioral Strategy (Body-image, None body-image) and Representation method (RL, VR, MR) as within-subject factors revealed no significant difference. The results showed in Table 8. The ANOVA was performed using nine participants data since there was missing data on two participants out of eleven.

Table 8. The result of the ANOVA on the cumulative contribution ratio for singular value (mode 1)
Full size table

Table 9 showed the result of descriptive statistics for the cumulative contribution ratio for singular value (mode 1).

Table 9. The descriptive statistics for the scale parameter of the cumulative contribution ratio for singular value (mode 1)
Full size table

The descriptive plot for the scale parameter of the cumulative contribution ratio for singular value (mode 1) in Fig. 9.

Fig. 9.
figure 9

The descriptive plot for the scale parameter of the cumulative contribution ratio for singular value (mode 1). The error bars represent standard error of means.

Full size image

Cumulative Contribution Ratio for Singular Value (Mode 1 and Mode 2).

A 2 × 3 ANOVA with Behavioral Strategy (Body-image, None body-image) and Representation method (RL, VR, MR) as within-subject factors revealed a main effect on Behavioral strategy,\( F\left( {1,8} \right)\, = \,9.171,\,p\, < \,.016,\,\eta_{p}^{2} \, = \,.534 \). The main effect was not qualified by an interaction between Representation method and Behavioral strategy, \( F\left( {2,16} \right)\, = \,.134,\,p\, < \,.813\,\eta_{p}^{2} \, = \,.017 \). The results showed in Table 10. The ANOVA was performed using nine participants data since there was missing data on two participants out of eleven.

Table 10. The result of the ANOVA on the cumulative contribution ratio for singular value (mode 1 and mode 2)
Full size table

Table 11 showed the result of descriptive statistics for the cumulative contribution ratio for singular value (mode 1 and mode 2).

Table 11. The result of the ANOVA on the cumulative contribution ratio for singular value (mode 1 and mode 2)
Full size table

The descriptive plot for the scale parameter of the cumulative contribution ratio for singular value (mode 1 and mode 2) in Fig. 10.

Fig. 10.
figure 10

The descriptive plot for the scale parameter of the cumulative contribution ratio for singular value (mode 1 and mode 2). The error bars represent standard error of means.

Full size image

4 Discussion

4.1 Task Performance

Task Completion Time.

Research in multimodal feedback suggest improvements in performance when additional modalities are used for user activities [11]. In this case, performance improvements were expected with additional modality, namely haptic feedback. However, there was no significant difference between VR and MR condition. In addition, although the haptic feedback in RL and MR conditions was totally same, there was a significant difference between RL and MR condition. These results indicate that the visual feedback affects the performance; that is, the consistency of visual feedback must be considered.

Shape and Scale Parameter of the Gamma Distribution.

Scale parameter \( \theta \) has the effect of stretching or compressing the range (time-to-collision) of the Gamma distribution. Whereas, shape parameter \( k \) controls the shape of the family of distributions. The fundamental shapes of the Gamma distribution are characterized by values of \( k \); (1) When \( k \) < 1: the Gamma distribution can be exponentially formed as well as asymptotic to both the vertical and horizontal axes, (2) When \( k \) = 1: A Gamma distribution with shape parameter \( k \) = 1 as well as scale parameter \( \theta \) is the same as an exponential distribution of scale parameter \( \theta \), (3) \( {\text{When}}\,k \) > 1: the Gamma distribution assumes a unimodal, namely skewed shape. The skewness reduces when \( k \) increases.

The result of these parameters indicates that the Gamma distribution on VR condition assumes more skewed than that of RL and MR conditions; that is, a collision to the wall during each trial on VR task occurs much more constantly than all the other task. According to the participants’ impression when they play the task, they reported they could conduct the task easily when there is haptic feedback.

4.2 User Behavior

Cumulative Contribution Ratio for Singular Value (Mode 1).

The result indicates that extracted motion pattern has similarity in 1st mode singular value. However, the cumulative contribution ratio for 1st mode was about between 53.54–66.35% in all conditions so that it would not be possible to reconstruct original motion data only using 1st mode data. The remaining about 40% data would be expected to spread out in other mode singular value; that is, feature difference between conditions would be shown in later mode value.

Cumulative Contribution Ratio for Singular Value (Mode 1 and Mode 2).

The result indicate that decomposed motion pattern has the difference in Behavioral strategy factor. We performed ANOVA on the other combination in cumulative contribution ratio, however, there was no significant difference on all of factors; that is, feature difference was spread out by 2nd mode singular value (88.68 ~ 95.88%). We did not perform deeper analysis on difference in this study. However, these behavioral differences must be considered as a feature to identify the quality of the SoE as well as its subcomponents.

5 Conclusion

In this study, we examined the effect of different types of behavioral strategy on performance as well as behavior in three types of different information representation method such as real task environment, VR-based task environment, and MR-based task environment in order to identify some features that enable to be applied for performance-based/behavioral-based measurement for the characterization of the SoE and its sub-components. Especially, we focused to explore what kind of difference it will reveal regarding performance as well as behavior data.

As the results, we found that there was a significant difference in task performance such as time completion time, and parameter of time-to-collision distribution, as well as on user behavior such as decomposed motion data. However, since this study was conducted as a preliminary study, more large number of samples should be analyzed as well as compared to the result of traditional questionnaire measurement of the SoE and its subcomponents as future work.