Keywords

1 Introduction

Heartbeat feedback is useful for stress reduction. There are two methods in this feedback, true and false. In true heartbeat feedback, heartbeat with the same tempo as the heart rate of the participants is provided through different modalities such as audio, visual, and tactile. Michael et al. provided heartbeat feedback to nonclinical students showing marked claustrophobic fear. They found that heartbeat feedback can relieve the fear of these participants [1]. On the other hand, Kenneth et al. developed a mobile system for controlling feedback of heart rate during outdoor running. They suggested that the participants could exercise more efficiently when the heartbeat feedback was provided [2].

False heartbeat feedback is another variant to provide the modified heartbeat to the participant. It is known to be useful for relieving stress even without instructions, such as asking the participants to take deep breaths during the feedback. Jean et al. designed a wearable device to regulate the anxiety of the users by providing false feedback with a slower tempo than the actual heart rate [3].

Recently, occupational stress in the working environment is becoming a serious problem. At the same time, mixed reality (MR) technology has made remarkable progress, and some companies already introduced it for training and working support in the real working environment.

Our purpose is to reduce the stress of the participants during working unconsciously without instructions. Therefore, in this study, we employed false heartbeat feedback. Although there are several modalities, such as audio, visual, and tactile, for providing the heartbeat feedback, the most effective one is yet to be clarified. In this study, we developed a feedback system using an MR head-mounted display (HMD) and a wrist band heart rate sensor. We also evaluate the effects of false heartbeat feedback on stress reduction among three different feedback modalities, audio, visual, and audio-visual.

2 Measurement and Feedback System

We developed a feedback system using an MR HMD (Microsoft HoloLens [4]) and a wrist band heart rate sensor (Polar OH1 [5]). Figure 1 shows the system setup. In our system, we used a fixed heartbeat, 66 beats per minute, as false feedback through the following different feedback modalities during the stress task to make the participants relieved.

Fig. 1.
figure 1

System setup

Full size image

  • Audio feedback: heartbeats sounds at the 66 bpm was provided

  • Visual feedback: animation of spherical shape changes at the 66 bpm as shown in Fig. 2 was provided

    Fig. 2.
    figure 2

    Shape change of sphere in the visual feedback

    Full size image

  • Audio-visual feedback: audio and visual feedbacks were simultaneously provided

3 Experiment

We performed experiment for 11 participants. In this experiment, we asked them to move small beans from one dish to the other with chopsticks as a stress task assuming manual labor in the factory. Participants wore an MR HMD and performed the task under four different conditions, no feedback provided, and one of the three feedbacks provided (audio, visual or audio-visual feedback) in random order.

We evaluated the effects of feedbacks by questionnaire using the card-sort task load index (CSTLX), salivary amylase test, and HRV analysis using an electrocardiogram (ECG). For this purpose, we measured salivary amylase and ECG at the resting state before the task and ECG of the participants at 256 Hz with a wireless ECG sensor (ZMP ECG2) during the task. After the task completed, we measured salivary amylase again and collected subjective workload assessments with the CSTLX.

4 Results and Discussion

The data from one participant who felt sleepy during the experiment were excluded from the following analysis in this section since he could not concentrate on the task.

4.1 Task Score (Number of Beans)

Figure 3 shows the average number of beans that the participants could move in each condition. There were no significant differences among the condition.

Fig. 3.
figure 3

Task score (number of beans) in each condition.

Full size image

4.2 Results of the CSTLX

Figure 4 shows the results of the CSTLX. We separated the participants into two groups based on the weighted workload (WWL) score under no-feedback condition, high-stress group (WWL > 50), and low-stress group (WWL < 50). This figure demonstrates that all of the three feedback modalities could reduce WWL for the participants in the high-stress group, and single feedback is more effective for reducing WWL than audio-visual feedback. On the other hand, there is no stress reduction by the feedback for the participants in the low-stress group.

Fig. 4.
figure 4

Results of the CSTLX in each condition for high and low stress group.

Full size image

4.3 Results of the Salivary Amylase Test

We measured salivary amylase within 1 min before the task started and after the task completed, and calculated the ratio of the salivary amylase change. Figure 5 shows the results of the salivary amylase test. It is known that the salivary amylase increases as the stress increases. We could not find significant differences in the salivary amylase among task conditions. However, we could speculate that the audio feedback is a better way to provide heartbeat feedback since it has a lower score than the control (no feedback) condition and the other two conditions with visual and audio-visual feedback.

Fig. 5.
figure 5

Results of the salivary amylase test in each condition.

Full size image

4.4 Results of the HRV Analysis

In our system, we employed a photoplethysmogram (PPG) based wrist band heart rate sensor. Since ECG is more reliable than PPG, we also measured ECG before the task and during the task for computing HRV indexes, such as standard deviation of R-R interval (SDNN) and the ratio of low-frequency to high-frequency power (LF/HF).

Figure 6 shows the averages of the heart rate before and during the task for each task condition. There is no significant difference among conditions. Also, the average of the heart rate did not change between before and during the task. This is because the task required low physical activities, the participants performed the task in sitting posture, and only their upper limbs were moving. Figure 7 shows SDNNs of the heart rate before and during the task for each task condition. Same as the average of the heart rate, there is no significant difference among conditions.

Fig. 6.
figure 6

Averages of heart rate before and during the task in each condition.

Full size image
Fig. 7.
figure 7

SDNNs of heart rate before and during the task in each condition.

Full size image

Figure 8 shows LF/HFs before and during the task for each task condition. There is no significant difference among conditions. Only the LF/HF in visual feedback was at a low level. It was also clearly low compared to the resting state. A decrease in LF/HF indicates parasympathetic dominance and is often used as an index of stress reduction. Therefore, our results suggest that visual feedback reduced participants’ stress. However, these results that visual feedback only has a stress-reducing effect are not consistent with the results of CSTLX and salivary amylase test. We might have detected emotional changes that cannot be detected by the salivary amylase test and CSTLX by LF/HF. This may be because only HRV analysis is a real-time measurement. In this HRV analysis, we set the entire task as target sections. For other tasks from the visual feedback task, it may be necessary to examine and set analysis time section carefully for calculating HRV indexes.

Fig. 8.
figure 8

LF/HFs of heart rate before and during the task in each condition.

Full size image

5 Conclusion

In this study, we developed a heartbeat feedback system using an MR HMD and a wrist band heart rate sensor. Using this system, we performed an experiment and evaluated the stress-reducing effect by the feedback in different modalities, audio, visual, and audio-visual. In this experiment, the participants were asked to move small beans from one dish to the other by chopsticks as a stress task, while at the same time, they were provided with a fixed heart rate (66 bpm) as false heartbeat feedback.

The results of the questionnaire showed a stress-reducing effect in audio feedback, while the results of the HRV analysis suggested a stress-reducing effect in visual feedback. In both analyses, audio-visual feedback did not show a stress-reducing effect, suggesting that the combination of multiple modalities has a negative effect on stress reduction.