1 Introduction

Several robotic devices, such as exoskeletons, have been proposed and validated in the last decade (Molteni et al. 2018). One of the most notable applications for these robots lies in the realm of physical rehabilitation, where they offer crucial motor assistance to the joints of impaired limbs, facilitating the completion of a movement or task. In fact, rehabilitation robotics is a dynamic field of research, actively pursued due to its potential to enhance joint movement, alleviating the physical workload of physiotherapists, improving exercise repeatability, and capturing valuable biomechanical data from patients. Robotic rehabilitation devices for the upper limb can be used to train unilateral and bilateral movements (Gupta et al. 2020; Shah et al. 2023). While unilateral training involves isolated activities with the affected limb, bilateral training focuses on simultaneous movements with both limbs, utilizing the healthy limb to support the impaired one and promote functional recovery (Richardson et al. 2021). While both types of training are effective, there is a preference for bilateral training, aligning with the fact that most Activities of Daily Living (ADLs) require the use of both limbs (Swinnen and Wenderoth 2004; Hatem et al. 2016; Barak-Ventura et al. 2022). Moreover, patients’ engagement can be enhanced by integrating the manipulation of familiar objects into the training process (Olafsdottir et al. 2020), as well as by recalling the correct actions associated with stimuli from memory (Zeelenberg et al. 2016).

While robotic-assisted exercises offer numerous advantages, their performance may introduce both physical and cognitive load effects, attributed to the exercise program itself and the generally heavy, bulky, and cumbersome structure of the exoskeleton, covering a significant portion of the impaired limb(s). Nevertheless, integrating immersive tools like Virtual Reality (VR), Mixed Rality (MR) and gamification can alleviate these challenges by serving as a distraction from physical effort and potential discomfort. While virtual reality (VR) generates immersive simulated environments, which may induce feelings of dizziness (Caserman et al. 2021), mixed reality (MR) creates environments where virtual objects coexist with the real world, grounding users in the real world and facilitating seamless interactions between real and virtual elements (Liberatore and Wagner 2021). Gamification is another technique that can serve as a motivational factor to enhance engagement in intensive, specific, and repetitive rehabilitation training exercises (Alfieri et al. 2022). Robotic and VR technologies can significantly enhance upper limb rehabilitation by fostering the development of multimodal, functional, task-oriented, and patient-centered programs. These advanced methods may surpass the traditional segmental and biomechanical approaches commonly used in neurologic and orthopedic rehabilitation. This is supported by the fact that conventional rehabilitation protocols typically involve providing patients with explicit instructions on how to perform movements, directing their attention to their body and specific exercises, and inducing an ‘internal’ focus of attention. However, current neuroscientific knowledge emphasizes that early introduction of functional exercises with an ‘external’ focus-shifting the patient’s attention to the effects their actions have on the external environment-can improve the efficiency of rehabilitation interventions.

The literature indicates a growing interest in integrating gamified VR applications into robot-assisted rehabilitation therapies. However, most developments have been non-immersive, utilizing conventional graphical displays like computer monitors (Catalán et al. 2023). To our knowledge, none of these applications have been fully or partially immersive through the use of Head Mounted Displays (HMDs). Furthermore, there is currently no immersive VR application designed for bilateral robot-assisted training that integrates objects used in rehabilitation therapies as interactive tools within the virtual environment. Based on this considerations, exploring this avenue in MR-based solutions opens up new possibilities to elevate the effectiveness of bilateral robot-assisted training.

This paper introduces the design and evaluation of gamified MREs for shoulder bilateral training, assisted by a 6 Degrees of Freedom (DOF) exoskeleton called FLOAT (Buccelli et al. 2022). FLOAT is specifically engineered to support natural shoulder movements and reintegrate the shoulder into complex functional tasks early in the rehabilitation process. By allowing patients to move beyond the constraints of a seated position, FLOAT enables interaction with the environment and physical objects, thereby expanding treatment possibilities and fostering an occupational approach to rehabilitation. The integration of FLOAT with the MREs enhances the ability to manipulate and move objects, offering innovative treatment opportunities that are not typically available in conventional rehabilitation settings.

The gamified MREs were designed to guide and motivate symmetrical and asymmetrical shoulder movements using a dowel rod, while FLOAT was programmed to assist the movement of the affected limb. Additionally, the system was programmed to generate meaningful performance metrics through kinematic analysis of hand movements. To evaluate the system, a pilot study involving twenty-one healthy adults was conducted. The objectives of this study were to assess the usability of the system, compare metrics obtained from Hololens2 measurements with those from Vicon, establish normal reference values for the performance metrics for future patient evaluations, and identify potential improvements.

The remainder of the paper is structured as follows: Sect. 2 provides a review of relevant literature; Sect. 3 outlines the materials and methods employed in the study; Sect. 4 presents the results obtained; Sect. 5 offers a broader discussion of the findings; Sect. 6 addresses the applicability, limitations, and potential future developments; and Sect. 7 concludes the paper.

2 Related works

In recent years, studies have explored the potential of incorporating VR, Augmented Reality (AR) and MR solutions into conventional rehabilitation therapies to enhance upper limb recovery (Leong et al. 2022; Howard and Davis 2022; Hao et al. 2023), or to provide visual feedback on patient performance to healthcare professionals (Pezzera et al. 2020). Interestingly, despite the importance of bilateral functional tasks in rehabilitation, most VR, AR and MR solutions for upper-limb rehabilitation focus solely on unilateral training to restore the functions of the affected limb (Han et al. 2017; Burke et al. 2010; Desai et al. 2016). Another common feature of these systems is the interaction/manipulation of virtual objects through a virtual arm/hand (Fernandez et al. 2020) or object (Ozkul et al. 2020), which moves within the environment based on hand or joint movements, without incorporating the use of real objects. However, it’s widely recognized that holding a real object in hand(s) stimulates the transmission of information through afferent nerves, enhancing proprioception and thereby contributing to shoulder stability and the performance of more precise movements (Ager et al. 2020).

Activities of daily living (ADLs) entail a continuous stream of visual and cognitive stimuli from the external environment, requiring individuals to respond adeptly, synchronizing and executing their movements accurately. Consequently, within the field of rehabilitation robotics, there has been a growing interest in integrating gamified VR solutions to enhance stimulation, interaction, and movement during the rehabilitation process. These VR solutions tipically utilize non-immersive technologies, such as computer or tablet screens, to present visual and cognitive stimuli through gamified 2D environments and thus promote rehabilitation for various body parts including the hand (Huang et al. 2017; Nehrujee et al. 2021), ankle (Prieto et al. 2020), elbow (Ozkul et al. 2020) or upper limb (Fernandez et al. 2020; Catalán et al. 2023). While non-immersive VR solutions offer advantages such as ease of use, portability, cost-effectiveness, and reduced motion sickness, they may also result in limited immersion and realism, as well as constraints on range of motion and potential distractions. However, the emergence of new MR technologies, such as Microsoft Hololens HMDs, provides the opportunity to create partially immersive 3D environments where virtual objects are overlaid onto the real world (Dhillon and Tinmaz 2024). This characteristic allows users to perceive rich 3D graphic elements while maintaining contact with the real-life setting, potentially reducing motion sickness (Kirollos et al. 2023) and increasing user immersion and engagement. To date, there has been limited exploration of integrating gamified MR environments into rehabilitation robotic systems, particularly ones that integrate both tangible and simulated components within the interaction.

3 Materials and methods

3.1 Gamified MR robot-assisted training system

The system consists of the upper-limb exoskeleton FLOAT designed to promote and accelerate the motor and functional recovery of the shoulder joint complex following post-traumatic or post-surgical injuries (Buccelli et al. 2022), see Fig. 1. Complementing this exoskeleton are gamified MREs, strategically designed to captivate and direct patients through upper limb rehabilitative exercises. These gamified MREs, viewed through an untethered MR headset (Hololens 2) utilized for hand motion tracking, are personalized to each patient’s abilities and are rich with visual and cognitive stimuli. Interacting with the MREs involves using a physical and lightweight dowel rod grasped with both hands, fostering improved proprioception and enabling the unaffected limb to guide and control movements of the impaired limb (Kisner and Colby 2012).

The core concept is to treat FLOAT and HoloLens as independent standalone devices and maintain their functionality separately. In this way, patients can perform the shoulder exercises with or without assistance, and with or without gamified MREs. Additionally, by adhering to this approach, the devices remain disconnected from each other, preventing them from being perceived as a singular biomedical system. Consequently, there’s no need to conduct a system risk analysis with both devices interconnected.

The design and feature details of the robotic assistance and the gamified MREs are explained next.

Fig. 1
figure 1

Bilateral shoulder exercise using a dowel rod and performed with the Gamified MR Robot-Assisted Training System (left), composed of the arm exoskeleton FLOAT and HoloLens v2, and Mixed Reality world perceived through HoloLens (right)

Full size image

3.1.1 Robotic assistance

It was provided through the upper limb robotic device FLOAT, which has five rotational joints, with each degree of freedom representing a movement of the shoulder complex, see right image of Fig. 2. Its control unit is shown on the left of Fig. 2. The first joint of FLOAT assists the scapula protraction/retraction and the second joint the scapula elevation/depression. The last three joints assist the glenohumeral movements: horizontal abduction/adduction, flexion/extension, and internal/external rotation, respectively. The five DOFs of the exoskeleton are sufficient to move the patient upper arm in the Cartesian space, guaranteeing the possibility to perform most of the ADLs. The length of the rigid exoskeleton links can be easily adjusted to match the size of the patient anthropometric characteristics. The exoskeleton includes a passive and polyarticulated arm and a mobile support with a telescopic column, allowing the patient to move freely within a certain area and have a richer and more natural interaction with the surrounding environment. However, this advantage was not utilized in the present study, as the gamified MREs were programmed to facilitate shoulder exercises while seated or standing, eliminating the requirement for walking.

Fig. 2
figure 2

Picture of the upper limb exoskeleton FLOAT, comprising an orthopedic torso harness and arm brace (right), and its control unit (left) (Buccelli et al. 2022)

Full size image

The control unit of FLOAT enables health professionals to choose the modality of operation of the exoskeleton, set all the parameters, and design a new exercise. FLOAT can work in three modalities: Kinematic, Assistive Mode, and Repeat trajectory. In this paper, the evaluation of the gamified MR solution was achieved with the exoskeleton working in Assistive Mode with a medium level of robotic assistance. In this mode, the local control is responsible for compensating the friction torques at every single joint by calculating the motor torque as a function of dynamic variables, as described in Buccelli et al. (2022). The robot assistance is programmed to support the upper limb in performing vertical motions upward. This is achieved by increasing the torque at joint 4 based on the joint angle and its velocity.

3.1.2 Gamified MREs

The design of the gamified MREs for robot-assisted upper limb exercises was based on an user-centered design approach, where users’ needs were the primary guiding factors (Triberti et al. 2015). The design process consisted of three phases, each involving primary users (patients and clinicians) and secondary users (developers). The initial phase focused on identifying requirements, followed by the design and development phase, and culminating in the evaluation of prototypes in the final phase.

As a result of this process, two gamified MREs (Game A and Game B) were designed for robot-assisted upper limb exercises, incorporating the following general features:

  1. 1.

    They elicit bilateral shoulder movements through the interaction of a physical object (a dowel rod) with virtual elements within the games. The use of the dowel rod as a tangible object provides greater proprioceptive and kinesthetic feedback. Additionally, it enables the healthy arm to facilitate movement for the robot-assisted impaired arm (Barak-Ventura et al. 2022; Basti et al. 2016).

  2. 2.

    Game A promotes symmetrical flexion-extension movements, while Game B asymmetrical flexion-extension movements, without imposing a time limit.

  3. 3.

    Both games have “Match” as the Game rule and “Move” as the Play rule (Djaouti et al. 2011), where the Game rule is the goal to be reached and the Play rule determines the means to achieve this goal.

  4. 4.

    They can be played from a sitting or standing position since it is the virtual environment that moves from left to right (Game A), or vice-versa (Game B).

  5. 5.

    During the play game, the robot assists the impaired arm movement and also restricts the use of the trunk to prevent compensation for arm movement.

  6. 6.

    Quantitative metrics are calculated during the game play.

Next, we delve into the game mechanics and the underlying design rationale for Game A and B.

Game A for symmetrical shoulder movements The game involves sequentially placing five bridges at a specific height to enable a hen to walk over them and reach her chicks (Game rule: “Match”), see Fig. 3a. Players manipulate a virtual rod to elevate the bridges from the bottom (Play rule: “Move”), controlled by moving a real dowel rod with palms facing downward. Each bridge placement demands symmetrical shoulder flexion with fully extended elbows, followed by returning the arms to the resting position. Thus, to win the game, players must complete five sets of symmetrical shoulder flexion-extension movements. The game offers two levels of difficulty: in Level 1, shoulder flexion ranges from 0 (resting position) to 70 degrees, while in Level 2, shoulder flexion extends from 0 to 90 degrees. To encourage symmetrical shoulder movements, the virtual rod’s movement is only possible if the real dowel rod maintains a horizontal configuration, with a deviation allowed of \(\pm 5\) degrees. Successfully placing a bridge awards points and the opportunity to collect a coin. Coins can be retrieved using the virtual rod while returning the arms to the resting position, ensuring consistent arm positioning after each bridge placement. Cheerful sounds accompany the correct placement of bridges, collection of coins, and when the hen finds her chicks. Additionally, a background soundtrack enhances the gaming experience alongside the sound effects.

Game B for asymmetrical shoulder movements The game involves players sequentially rotating three handles (30 degrees clockwise or counterclockwise) to adjust an object to specific rotations/positions, enabling a child to walk over them and reach a treasure hunt (Game rule: “Match”), see Fig. 3b. Players control the rotation of virtual handles by manipulating a real dowel rod (Play rule: “Move”), held with palms facing downward. To initiate handle rotation and promote asymmetrical shoulder movements, players must first align the position and rotation of the real dowel rod with the virtual handle, with a tolerance of \(\pm 5\) cm and \(\pm 5\) degrees, respectively. Rotating each handle requires asymmetrical extension/flexion with fully extended elbows at shoulder height, followed by returning the arms to the rest position. Thus, players must complete five sets of asymmetrical shoulder flexion-extension movements to win the game. Successfully placing an object earns points and the chance to collect a gem. Gems can be retrieved using the virtual rod while returning the arms to the resting position, ensuring players consistently return their arms to the resting position. Cheerful sounds accompany correct object placements, gem collections, and when the child reaches the treasure. Additionally, a background soundtrack enhances the gaming experience alongside the sound effects.

Fig. 3
figure 3

Complete view of the virtual environment (left image) and first-person mixed-reality view (right image) of a Game A and b Game B

Full size image

An application for each gamified MRE was created in the Unity graphics engine (version 2020.3.33f1) and then deployed to Hololens. We employed a sampling frequency of 50 Hz for both graphics rendering and data acquisition.

3.2 Participants

Twenty-one healthy right-handed subjects with no existing arm injuries were involved (8 females and 13 male, 29.1±5.6 years of average age). The study was performed according to the Helsinki Declaration and was approved by the Ethic Committee of Liguria Region (IIT-REHAB-FH01). All participants signed a consent before accomplished the experiment.

3.3 Procedure

This study used a repeated measured design where each participant completed robotic-assisted bilateral symmetrical and asymmetrical upper limb exercises with and without the guidance of the gamified MREs. Before starting the experiment, anthropometric measurements of each participant were taken to adjust the length of the robot’s links. Subsequently, participants donned the exoskeleton, and additional adjustments were made to ensure their comfort. Following this, and before each experimental condition (with and without gamified MREs), participants received specific instructions on how to perform each type of exercise. After that, they had the opportunity to conduct pilot trials.

An important instruction given to the participants before starting the exercises under either condition was to keep their elbows fully extended throughout the trials. For the exercises involving the gamified MREs, participants were instructed to perform the movements with the velocity, range, and rhythm needed to follow and fulfill the dynamics of the games. For the exercises without gamified MREs, participants were asked to repeat each demonstrated movement ten times. The instructor, with his back to the participants, showed the movements, which the participants then observed and replicated. The type of movement, range, and number of repetitions were consistent with those used with gamified MREs, as explained below.

During the robotic-assisted shoulder exercises with gamified MREs, participants played Game A at Level 1 and 2, and Game B twice, in random order. Thus, each participant completed a total of thirty shoulder movements: twenty symmetrical shoulder exercises with Game A (ten at level 1 from 0 to \(70^{\circ }\) and ten at level 2 from 0 to \(90^{\circ }\)) and ten asymmetrical shoulder exercises with Game B (from 0 to \(90^{\circ }\)). Each game application could be launched either via a voice command or from the Hololens main menu. Following the exercises, participants filled out paper-and-pencil questionnaires. The entire session lasted approximately 40 min.

When performing robotic-assisted shoulder exercises without gamified MREs, each participant completed a total of thirty shoulder movements: twenty symmetrical shoulder exercises (ten from 0 to \(70^{\circ }\), similar to Game A-Level 1, and ten from 0 to \(90^{\circ }\), similar to Game A-Level 2), and ten asymmetrical shoulder exercises (from 0 to \(90^{\circ }\), similar to Game B). Following the completion of these exercises, participants were asked to fill out paper-and-pencil questionnaires. The entire task lasted approximately 25 min.

Ten participants began with the robotic-assisted exercises with gamified MREs, followed by robotic-assisted exercises without gamified MREs. Conversely, the remaining eleven participants completed the exercises in the reverse order.

3.4 Measurements

The experiment gathered both objective and subjective measurements. Subjective measures comprised qualitative data collected through questionnaires, capturing participants’ reflections on their experience with the Gamified MR robot-assisted training system. It also included their perceptions of usability, workload, and overall experience during robotic-assisted exercises, both with and without gamified MREs. Objective measures focused on four metrics that offer insights into upper limb movement and exercise performance. Further details about the subjective and objective measures are provided below.

3.4.1 Subjective measures

Usability:

of the Gamified MR robot-assisted training system was evaluated through the System Usability Scale (SUS) questionnaire (Brooke 1996). This questionnaire is a ten-item scale that provides a global view of subjective assessments of usability (ranging from 0 to 100). Each item was rated using a five-point Likert scale from ‘strongly disagree’’ (one point) to ‘strongly agree’ (five point). Higher scores indicate better usability, e.g., the 70 s is acceptable but 90 s is truly superior, and scores below 50 indicates unacceptable low levels of usability Bangor et al. (2008).

User workload:

of participants during robot-assisted shoulder exercises with and without gamified MREs was assessed by using the NASA Task Load Index (NASA-TLX) (Hart and Staveland 1988). The questionnaire evaluates six subscales to obtain an estimated total workload of the users during the task performance. The subscales are mental demand, temporal demand, physical demand, effort, perceived performance, and frustration of interactions. Ratings were made within a 100-points range with 5-point steps, where 0 is ‘very low’ and 100 is ‘very high’. Participants also rated the relative importance of each workload dimension for the task, by making a series of 15 pairwise comparisons between dimensions. The number of times each dimension is chosen was the weighted score. A weighted workload score for each dimension is calculated by multiplying its scale score and weighting score. An overall workload score was obtained from the sum of the weighted workload score of all six dimensions and then dividing the result by 15 to get a value from 0 to 100.

User experience:

of participants during robot-assisted shoulder exercises with and without gamified MREs was assessed through the User Experience Questionnaire (UEQ) (Laugwitz et al. 2008). This questionnaire is a widely used evaluation tool for interactive products and consists of 26 items formed by 7-point likert scale. It measures six factors of user experience: Attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty. The factors are scaled from −3 to +3, where −3 represents the most negative answer, 0 a neutral answer, and +3 the most positive answer. Perspicuity, Efficiency and Dependability are pragmatic quality aspects (goal-directed), while Stimulation and Novelty are hedonic quality aspects (not goal-directed). Scores above 1 are considered as positive evaluation.

3.4.2 Objective measures

The hand paths and velocity profiles were derived from data recorded by the Hololens 2 and the Vicon capture system. Additionally, metrics such as reach time, hand path ratio, peak hand velocity, and hand movement smoothness were calculated. These trajectories and metrics offer valuable insights into upper limb movement and exercise performance. Furthermore, they were useful to compare the performenace of robotic-assisted exercises with and without gamified MREs, as well as for validating HoloLens performance in comparison to Vicon.

Reach time:

It is the average time taken to reach the target (place a bridge/object correctly) through the symmetrical/asymmetrical flexion of both fully-extended arms holding the dowel rod, starting at rest position.

Path ratio:

It is defined as the average of the distances traveled by each hand divided by the straight-line distances between the initial and final reach positions, and then averaging the resulting values from both hands. It equals 100% straight hand movements and increases with hand-path curvature.

Peak velocity:

It is described as the average of the maximum velocities reached by each hand before reaching the target, and it is expressed in cm/s.

Smoothness:

It is quantified using the dimensionless SPARC (SPectral ARC length) measure (Balasubramanian et al. 2015), which calculates the arc length of the Fourier magnitude spectrum within the frequency range 0 to 20 Hz of a given speed profile v(t). Smoother movements tend to have less intermittencies and thus a higher SPARC measure.

3.5 Statistical analysis

The Shapiro-Wilk test was used to determine whether data follow a normal distribution. Normally distributed data in the UEQ and NASA-TLX results underwent paired-samples t-tests for comparing means between conditions. Non-normally distributed data were analyzed using related-samples Wilcoxon Signed Rank tests. Furthermore, Cohen’s d was used to measure the effect size that describes the difference between conditions.

All data in the objective metrics were found to be normally distributed. The objective metrics obtained in the three robot-assisted shoulder movements, both with and without gamified MREs, underwent a two-way repeated measures ANOVA followed by post hoc pairwise comparisons with Bonferroni adjustments. The same procedure was applied to the objective metrics obtained from the three shoulder movements recorded by both Hololens and Vicon systems. Additionally, a one-way repeated measures ANOVA followed by post hoc pairwise comparisons with Bonferroni adjustments was applied to the reference averaged objective metrics obtained in the three robot-assisted shoulder movements with the gamified MREs. A p value of 0.05 was used to establish statistical significance using R studio software (version 2023.12.0).

Fig. 4
figure 4

Average UEQ factor scores when performing the assisted-robot exercises with and without Gamified MREs. Significant differences are marked with an asterisk (* \(p <0.05\), ** \(p <0.01\), *** \(p <0.001\)). The error bars represent 95% confidence intervals

Full size image

4 Results

4.1 Subjective measures

The average score obtained in the SUS across all participants was 85.9 (SD = 8.2), indicating excellent usability of the Gamified MR robot-assisted training system (Aaron Bangor and Miller 2008). Scores ranged from 73 to 95, suggesting that most users found the system easy to use.

In Fig. 4, the average score for each factor of the UEQ is depicted, while Table 1 provides a summary of the statistical findings. Paired t-tests on the scores revealed a significant difference in all factors between conditions(\(p < 0.05\)), except for the Efficiency score (\(p = 0.3\)). The effect size for the difference in the Attractiveness, Stimulation and Novelty scores between conditions was large (Cohen’s d > 0.8), while the effect size for the difference in Perspicuity and Dependability scores between conditions was medium (0.5 < Cohen’s d < 0.8). Paired t-tests on the average score for pragmatic aspects (perspicuity, efficiency and dependability) and hedonic aspects (stimulation and novelty) revealed a significant difference only in the Hedonic quality aspect (\(p < 0.01\)), but not in the Pragmatic quality aspect between conditions (\(p = 0.12\)). A large effect size was observed for the difference in the Hedonic score between conditions (Cohen’s d > 0.8).

Table 1 Differences in UEQ scores between conditions
Full size table
Fig. 5
figure 5

Average scores of NASA-TLX dimensions when performing the assisted-robot exercises with and without Gamified MREs. Significant differences are marked with an asterisk (* \(p <0.05\), ** \(p <0.01\), *** \(p <0.001\)). The error bars represent 95% confidence intervals

Full size image

The average scores of the six NASA-TLX dimensions are shown in Fig. 5. Statistical results show a significant difference between conditions in mental demand (e.g. thinking, deciding, calculating, remembering, looking; \(p < 0.01\)), temporal demand (how much time pressure participants felt; \(p < 0.05\)), and effort (how hard participants had to work to accomplish their performance; \(p < 0.01\)). These results revealed that mental demand and effort were perceived higher when performing the robot-assisted exercises with than without Gamified MREs; whereas the temporal demand was higher without than with Gamified MREs. The other NASA-TLX dimensions do not show significant differences. The effect size for the difference in mental and temporal demand scores between conditions was medium (0.5 < Cohen’s d < 0.8), while for the effort score was large (Cohen’s d \(=\) 0.81).

4.2 Objective measures

This section presents the average values of the four metrics calculated from Vicon data when performing robotic-assisted exercises with and without gamified MREs (Sect. 4.2.1). Additionally, it includes the average hand trajectories measured using both Hololens and Vicon data specifically for robotic-assisted exercises with gamified MREs (Sect. 4.2.2). Lastly, the section showcases the reference values obtained exclusively from Hololens data for the four objective measures, as well as for paths and velocity profiles (Sect. 4.2.3).

Fig. 6
figure 6

Hand trajectories of one participant during robot-assisted symmetrical (a and b) and asymmetrical movements (c)) with (left side) and without (right side) the gamified MREs. Shaded areas indicate the reach phase of the movement. Dashed lines delineate the necessary hand distance to attain the desired maximum shoulder flexion. Asterisks denote the moment and distance at which the corresponding goal was achieved

Full size image

4.2.1 Performance evaluation of robot-assisted movements with and without gamified MREs

Figure 6 presents the trajectory of a participant’s hands, recorded by Vicon, during robot-assisted bilateral shoulder movements, including both symmetrical and asymmetrical motions, with and without gamified MREs. Additionally, Fig. 7 displays the average objective metrics derived from Vicon measurements for both conditions, with and without gamified MREs.

The graphs in Fig. 6 illustrate how the movement profile of the hands, resulting from bilateral flexion-extension (symmetric and asymmetric) of the shoulders, differs slightly between conditions (with and without gamified MREs). In the graphs representing the use of gamified MREs (left side of Fig. 6), the maximum amplitude of hand movement in each flexion-extension cycle appears more consistent and closely aligned with the reference values (highlighted by red dashed lines). Conversely, in the graphs depicting movements without gamified MREs (right side of Fig. 6), the maximum hand amplitude exceeds the specified limit in most flexion-extension cycles, suggesting that shoulders surpassed the intended flexion angle.

Notably in symmetrical exercises without gamified MREs (first two plots on the right side of Fig. 6), there is a slight discrepancy in the maximum amplitude of movement achieved by each hand. This suggests that the dowel rod was not maintained in a completely horizontal position as instructed, resulting in an asymmetric flexion angle of the shoulders. Furthermore, in asymmetrical exercises (bottom plots in Fig. 6), there is a greater difference in the amplitude of movement between hands than desired, and the maximum amplitude is sustained for a shorter duration. This implies that the shoulders surpassed the intended difference in flexion angle (that is the inclination of the dowel rod exceeded the specified angle) and the maximum asymmetrical shoulder flexion was held for a shorter period. Regarding movement velocity, it is observed that movements were performed at a slower pace with gamified MREs compared to without. These observations were consistent across participants and are reflected in the averaged outcome measures shown in Fig. 7.

Fig. 7
figure 7

Average values of a reach time, b peak velocity, c path ratio, and d smoothness, categorized by type of movement and the presence of gamified MRES

Full size image

The average values of the metrics including reach time, path ratio, peak velocity, and smoothness categorized by type of movement and the presence of gamified MRES, are displayed in 7. Statistical analysis indicates that Reach time and Peak velocity are significantly influenced by the type of movement (\(F(1.5,29.3)=21.5, { p} < 0.001\) and \(F(1.5,29.6)=61.4, { p} < 0.001\), respectively) as well as the presence of gamified MREs (\((1,20)=94.7, { p} < 0.001\) and \(F(1,20)=212, { p} < 0.001\), respectively). Post hoc tests further reveal that Reach time in all movements is significantly prolonged with the presence of gamified MREs compared to without (\(p's < 0.001\)), and that the reach time for asymmetrical movements with gamified MREs is notably the highest (M=6.3, SD=2), followed by symmetrical-level 2 (M=4.3, SD=1.9) and symmetric-level 1 (M=3.3, SD=1.4) movements (\(p's < 0.05\)). Conversely, post hoc analysis shows that Peak velocity is significantly lower with gamified MRE compared to without (\(p's < 0.001\)). Furthermore, Peak velocity during asymmetrical movements is notably the highest in both conditions (with M=38.4, SD=2.5; without M=62.2, SD=6.3), followed by symmetrical-level 2 (with M=31.3, SD=7.4; without M=56.9, SD=9.8) and symmetric-level 1 (with M=29.2, SD=3.2; without M=50.6, SD=5.9) movements (\(p's < 0.01\)).

Fig. 8
figure 8

Right (left side) and left (right side) hand trajectories of one participant during robot-assisted symmetrical (a) and (b) and asymmetrical movements (c)) with gamified MREs. Shaded areas indicate the reach phase of the movement. Dashed lines delineate the necessary hand distance to attain the desired maximum shoulder flexion. Asterisks denote the moment and distance at which the corresponding goal was achieved

Full size image

For the Path ratio, statistical analysis indicates a significant effect of type of movement (\(F(2,40)=9.4, { p} < 0.001\)), but not the presence of gamified MREs (\(F(1,20)=0.9, { p} =0.3\)). Post hoc tests reveal that the Path ratio during asymmetrical movement is significantly higher than in the other movements (\(p's < 0.05\)). Regarding Smoothness, statistical analysis demonstrates that there is no significant effect of the type of movement (\(F(2,40)=2.6, { p} =0.1\)) or the presence of gamified MREs (\(F(1,20)=0.2, { p} =0.7\)).

4.2.2 Comparative analysis of metrics derived from Hololens and Vicon data

The trajectories of both the right and left hands from one participant, captured by the Hololens and Vicon system during robot-assisted movements with gamified MREs are depicted in Fig. 8. While the Hololens recordings closely replicate those from Vicon (considered as the ground truth), they only cover a portion of the complete shoulder movements. Specifically, data capture with Hololens commenced shortly after and stopped shortly before the movement initiation and ending, respectively, thereby missing the initial and final phases of the flexo-extension shoulder movements. This highlights that the Hololens cameras face more difficulty in recording hand movements when the arms are close to the body, outside the camera’s field of view. To obtain a clearer picture of the differences, especially during the reach phase of the movement, the calculation of the Reach time and Path length computed from HoloLens and Vicon data was performed, and average results are presented in Fig. 9.

Statistical analysis on data presented in Fig. 9 reveals significant differences in Reach time (\(F(1,20)=201, p < 0.001\)) and Path length (\(F(1,20)=42.9, p < 0.001\)) depending on the motion capture system used. Post hoc tests indicate that Reach time (\(p's < 0.001\)) and Path length (\(p's < 0.01\)) in all movements were notably greater utilizing data from Vicon compared to Hololens. While the Vicon data yielded higher results, both systems exhibit the same trend: asymmetrical shoulder movements yield the highest values, followed by symmetrical shoulder movements ranging from 0 to 90 (Game A-Level 2) and from 0 to 70 (Game A-Level 1), as illustrated in Fig. 9.

Fig. 9
figure 9

Average a Reach time and b path length, averaged across subjects, trials and hand sides

Full size image

In percentage comparison, it was found that during symmetrical shoulder movements ranging from 0 to 70 (Game A-Level 1), Hololens recorded approximately 70% of the Reach time and 44% of the Path length registered by Vicon. These results improved during symmetrical and asymmetrical shoulder movements ranging from 0 to 90 deg (Game A-Level 2 and B, respectively) where Hololens recorded approximately 74% of the Reach time and 54% of the Path length registered by Vicon.

It’s noteworthy that although Hololens may miss data at the initial and final phases of flexo-extension shoulder movements due to its limited field of view, it does capture movements around the most critical event, which is the target-reaching moment, allowing to know whether the requested maximum shoulder flexion has been achieved. Equally important, the captured interval of the movement provides enough data to obtain metrics and evaluate movement performance.

4.2.3 Obtaining reference values, paths, and velocity profiles

One of the aims of this study was to establish reference values from healthy individuals for the four objective measures described in Sect. 3.4.2, as well as for the path and velocity profiles of hand movements, for each robot-assisted bilateral movement with gamified MREs, all derived exclusively from Hololens data. To qualitatively illustrate similarities or dissimilarities in movement kinematics during the reach phase, the hand paths and velocities in each type of movement were normalized for reach time duration and path length. This normalization ensures that the form of the paths and velocity profiles is clearly depicted, independent of variations in their starting and ending positions and timing.

The results for each type of movement are depicted in Fig. 10. As observed, the path and velocity profiles in all movements exhibits typical kinematic features observed in vertical arm movements (Atkeson and Hollerbach 1985; Gaveau and Papaxanthis 2011). Specifically, the velocity profile increases until reaching a peak, then gradually decreases as the target approaches until it reaches zero. This velocity profile, characterized by a single peak and a bell-shaped curve, is consistent across all movements. The movement profile in the asymmetrical movement differs from the other profiles in that participants required more time to reach the goal. Participants spent roughly half of the time reaching maximal shoulder flexion, and the other half rotating the virtual handles until reaching the goal, all while maintaining maximal shoulder flexion, as seen in Fig. 10c. Upon observing the shaded areas, it becomes evident that the shape profile of hand paths and velocities were consistent and exhibited minimal variability across participants, hand sides, and repetitions, particularly in symmetrical movements from 0 to 70 degrees.

Fig. 10
figure 10

The average normalized path and speed profiles per type of shoulder movement were pooled across participants, hand side, and movement repetitions. The average is depicted in black bold, with the standard deviation represented by the colored shaded region (n=21\(\times\)2\(\times\)10= 420 trials)

Full size image

The average values obtained for the four objective measures described in Sect. 3.4.2 from the robotic-assisted movements with gamified MREs are presented in Fig. 11. Statistical results show a significant effect of the type of movement on all measures: reach time (\(F(1.25,25)=51.2, { p} < 0.001\)), path ratio (\(F(1.2,24)=14.6, { p} < 0.001\)), peak velocity (\(F(2,40)=63.4, { p} < 0.001\)), and smoothness (\(F(1.18,25)=23.5, { p} < 0.001\)). Post hoc analyses revealed statistically significant differences in all pairwise comparisons for average reach time (\(p's<0.001\)) and peak velocity (\(p's<0.001\)) values. The highest values were observed in the asymmetrical movement performed with Game B, followed by the symmetrical movements performed with Game A-Level 2 and Game A-Level 1 (all \(p's<0.001\)), respectively. Post hoc analysis further indicated that there is no significant difference in the path ratio between the two symmetrical movements (\(p = 0.1\)), however it is statistically significantly higher in asymmetrical movements compared to the other symmetrical movements (\(p's<0.001\)). In terms of smoothness, post hoc analysis revealed significant differences in all pairwise comparisons. The lowest value was observed in the asymmetrical movement, followed by the symmetrical movements with Game A-Level 2 and Game A-Level 1 (all \(p's<0.001\)), respectively.

The highest values obtained in all metrics in the asymmetrical movement were expected, as achieving the goal using this movement was more challenging compared to using symmetrical movements. The difference observed in reach time, peak velocity, and smoothness between the two types of symmetrical movements is due to the variance in maximal shoulder flexion, with participants reaching a maximum shoulder flexion of 70 degrees with Game A-Level 1 and 90 degrees with Game A-Level 2.

Fig. 11
figure 11

The reference averaged metrics obtained from Hololens data in the robot-assisted movements with the gamified MREs, averaged across participants, hand sides and repetitions

Full size image

5 General discussions

The subjective evaluation revealed that the gamified MR robot-assisted training system has an excellent usability score (above 85). According to norms from the literature our system obtained an A+ grade (Lewis et al. 2016), meaning it resulted as more usable than 97% of other systems. Furthermore, the user experience evaluation indicated that performing robot-assisted shoulder exercises with gamified MREs is significantly more attractive and offers a superior user experience. In terms of cognitive load, the evaluation revealed that participants experienced greater mental demand and effort when using gamified MREs for robot-assisted shoulder movements, albeit with less temporal demand. These results were expected since interaction with the games, although intuitive, easy, and fun, involves a greater mental process than performing the exercises without games. Additionally, the increased effort can be attributed to the fact that games promote the correct and continuous execution of both symmetrical and asymmetrical shoulder movements, as well as maintaining maximal shoulder flexion for longer duration, thereby eliciting a greater physical exertion. However, the observed low temporal demand suggests that, as anticipated, the games do not impose strict time constraints on users.

The comparative analysis of performing robot-assisted shoulder movements with and without gamified Mixed Reality Environments (MREs) revealed significant differences. In exercises with gamified MREs, hand movement profiles demonstrated consistent amplitudes aligned with reference values, indicating precise execution. Conversely, exercises without gamified MREs often resulted in hand movements exceeding specified limits and shorter durations of maximum shoulder flexion, implying deviations from intended shoulder flexion angles and reduced effort. Additionally, movements were generally slower with gamified MREs compared to without, possibly due to the time required to accomplish virtual tasks within the MREs. In summary, the utilization of gamified MREs enhances the execution of exercises by promoting slower, yet more precise movements.

One significant outcome of the present study was the calculation of average metric values for healthy individuals. The normalized paths and averaged metrics derived from gamified MRES in conjunction with FLOAT offer a robust framework for objective assessment within clinical settings. These datasets offer an objective means to identify and analyze any deviations in movement patterns across rehabilitation sessions. By employing normalized paths and metrics, clinicians gain a detailed perspective on the evolution of patient movements over the course of their rehabilitation journey.

6 Applicability, limitations, and prospective developments

The gamified MREs offer several key technical advantages, notably their seamless integration with the rehabilitation robot FLOAT and their ability to operate independently as standalone components. This versatility ensures that patients receive customized robotic motor support or no support at all when exercising with the gamified MREs, depending on their specific needs and abilities, thereby enhancing the effectiveness of rehabilitation efforts.

It is worth mentioning that the gamified exergames work properly even if the exoskeleton covers a significant portion of the impaired limb, as the interaction with the exergames is based solely on the position of the hands. Unlike many gamified training solutions that lack flexibility in incorporating tangible objects or tools, these gamified MREs provide clinicians with the option to conduct training sessions either without objects or using rehabilitation tools, such as dowel rods or elastic bands, which are ideal for bilateral shoulder exercises. The only restriction is to use objects that, during upper limb movement, do not completely block the HoloLens’ view of the hands. A suggestion would be to use transparent objects that allow the fingers to remain visible, making it easier for the HoloLens to recognize the hands.

When analyzing hand trajectories captured by HoloLens cameras during bilateral shoulder movements, it was noted that HoloLens records the majority of the movement, averaging over 55% coverage, including the full extent of hand paths. However, it does not capture the initial and final phases of movements, where hands are close to the resting position and thus outside the HoloLens field of view. Despite this limitation, as HoloLens records the interval of movement of interest for this application, encompassing the critical event of reaching the target during maximal shoulder flexion, the obtained data remains sufficient for metric calculation and movement performance assessment.

Various improvements can be made to the gamified exergames, such as increasing or diversifying the difficulty levels to ensure they can be used across different stages of rehabilitation, adapting to the patient’s abilities. Additionally, while the dynamics and interaction with the games are straightforward and initially explained by the physiotherapist, this process could be streamlined by incorporating an explanatory video at the beginning of the HoloLens app. This video would demonstrate how to correctly play the game from a third-person perspective, making the process clearer and more intuitive for patients.

7 Conclusions

This study introduces gamified MREs specifically designed for bilateral shoulder movements. These environments offer the option to seamlessly integrate with the rehabilitation robot FLOAT or function independently as a stand-alone system. This versatility enhances rehabilitation effectiveness by allowing tailored robotic motor support or none at all, depending on user needs. Unlike many alternatives, our proposed gamified MREs allow for training with or without additional objects like dowel rods or elastic bands, making them ideal for bilateral shoulder exercises.

Subjective evaluation results indicate that participants found the system highly usable and engaging with gamified MREs, resulting in an improved user experience. On the other hand, objective evaluation findings reveal that the integration of gamified MREs led to slower, more consistent, and precise movements without compromising spatial accuracy. Another significant finding emerged from comparing the data recorded by Hololens with that captured by Vicon systems. Despite challenges in capturing the initial and final phases of movements, Hololens accurately records critical intervals, aligning closely with data from Vicon systems. This indicates that Hololens data collected within the gamified MREs is reliable and suitable for metric calculation and movement performance assessment. The derived normalized paths and averaged metrics offer clinicians objective assessment tools, furnishing valuable insights into patient movement patterns throughout rehabilitation sessions.

In conclusion, the adoption of gamified MREs equipped with Hololens technology not only enhances user engagement and movement effectiveness but also provides reliable data for metric calculation and movement performance assessment, offering valuable insights for clinicians in guiding patient rehabilitation journeys.

Looking ahead, we anticipate that the robust synergy between the exoskeleton and MREs will enable patients to receive upper limb support against gravity while engaging in immersive and tailored MR tasks. This proposed approach promotes cognitive engagement with an external focus, potentially alleviating discomforting body stimuli like pain, joint stiffness, and muscle fatigue, while shifting attention away from the burden of the robotic device. Moreover, it has the potential to foster a sense of presence and self-efficacy, facilitating patient involvement in intensive treatment programs and addressing qualitative movement aspects such as precision and coordination.