MagicChem: a MR system based on needs theory for chemical experiments

Maslow’s hierarchical needs (MHN) theory believes that human needs are composed of five layers: physiological needs, security needs, belongingness needs, esteem needs, and self-actualization needs. They are the power to inspire and guide individual behavior McLeod (2007). The human needs theory is also applicable to the field of experimental teaching, so that students can get a better experience and motivation to obtain a good learning effect.

Experiments occupy an important position in chemistry learning Bernath et al. (2005). However, in the real experiments, limited to time, space and resource constraints students have fewer opportunities to repeat experiments. Real chemical experiments may be dangerous, especially with flames, corrosion, toxic gases, etc. Guex et al. (2017), which will be difficult to meet the security needs in MHN theory, causing users to have psychological pressure Luo et al. (2020). In addition, in many schools, due to the lack of teacher resources Ángel (2015), Martin-Villalba et al. (2012), students often encounter problems hard to be solved, which is difficult to meet the needs of self-actualization. Even with teacher’s guidance, students who often make mistakes in operation will be criticized. It will violate esteem needs and frustrate students’ learning motivation.

Guided by needs analysis to design the experiment, the virtual experiment system can make up for the problems of real experiments Duan et al. (2020), Hodges et al. (2018), and improve the interest and efficiency of students in learning chemical experiments Ullah et al. (2016). However, the current researches on virtual experimental systems are mainly focused on the single demand point such as experiment simulation Verkuyl et al. (2017), education and teaching Wang (2018), interactive experience Bozzelli et al. (2019), Luo et al. (2018) and interest de Souza Silva et al. (2017). There are few systematic researches on needs model of human-oriented virtual experiments, especially in MR experiments.

The theory of human needs provides a new perspective for the system design of virtual experiments. The basic purpose of the virtual experiment system is to simulate the real experiment. However, in order for students to obtain better learning effects and enhance the learning interests, it is more important to make the experience of the interactive environment close to the real experiment while having a safe, comfortable, not alone(layers 1-3 in MHN theory) and encouraging (layers 4 and 5 in MHN theory) environment. Compared with VR, MR provides users with a more natural interface that combines the real and virtual world. However, in terms of visual presentation, most MR experimental systems lack the consistency of virtual-real occlusion and illumination, which will cause visual confusion Ellis and Menges (1998), Kalkofen et al. (2007) and make users feel unreal and not easy to immerse the interaction Jacobs and Loscos (2006). In terms of interactive design, most of the MR experimental systems are based on gesture interaction with specific actions or hand-held markers in single camera environments. These systems do not have the operational feeling and tactile feeling as real experiments do. These special shortcomings of the MR environment put forward new requirements for the needs model.

The main contributions of this paper are as follows:

• Based on the theory of human needs and the special needs of virtual experiment system, this paper proposes a new needs model to guide the design of virtual experiment system.

• To effectively meet the needs model, we develop a MR system called MagicChem which supports the consistency of virtual-real occlusion and illumination in terms of vision, and supports 7 multi-modal interaction methods in terms of interaction.

• We develop 3 ablation virtual experimental systems for grouped user study, and explore the effectiveness of MagicChem in terms of usability, user experience, and learning effects.

There are many researches on needs-oriented experimental systems. These experimental systems for education can indeed promote students’ learning Arici et al. (2019), Akçayır and Akçayır (2017), Chang and Hwang (2018).

2.1 Interactive MR systems for experiments

2.1.1 MR experimental systems with mobile environments

The interactive methods of MR on mobile terminals are mainly divided into two versions. One is based on 2D gesture Gan et al. (2018), Boletsis and McCallum (2013), Tuli and Mantri (2015), and the other is on hand-held markers Nachairit and SrisawasdI (2015), Matsutomo et al. (2017). The advantage of this kind of interactive method is that users will be more familiar with the interactive methods of mobile phones. The disadvantage is unnatural to use with one hand holding the mobile phone.

In addition, some researchers develop a method for exploring 3D gesture interaction with touching in the mobile environment Gao (2013), but the results are not ideal. The author analyzes the reasons: lacking touch feeling and without correct relationship of virtual-real occlusion.

2.1.2 MR experimental systems with glasses

A lot of MR applications Müller et al. (2018), Strzys et al. (2017), Zhu et al. (2018), Sapargaliyev (2015) use optical see-through glasses, and achieve good results, but this kind of environments is difficult to achieve correct virtual-real occlusion, since virtual objects are projected on the atomic lens, and the user sees the real scene through the lens, which cause virtual objects cannot completely occlude real objects and show a translucent blending effect of the virtual and real scene in visual. Since it is difficult to judge the distance between the hands and the virtual object, the main interaction mode of this kind of MR system is gesture interaction with specific actions rather than touching virtual objects directly. Mapes Mapes and Moshell (1995) believe that gesture interaction with touching is easier to learn, more natural and reliable than specific actions. Moreover, in terms of tracking algorithms, many MR systems use foreground threshold segmentation, then recognize fingertips and palms based on curvature and distance, finally implement click interaction through fingertips Bai et al. (2013) or gesture interaction with specific actions Hürst and Van Wezel (2013). These methods cannot adapt to complex scenes with tangible objects and flexible gestures.

2.1.3 MR systems with tabletop environments

For virtual experiments, the most appropriate way is based on tabletop, because real experiments are mainly carried out on the tabletop. The main interaction models of the systems are tangible interaction and gesture interaction.

In recent years, many MR experimental applications Benko et al. (2012), Oswald et al. (2014), Roo et al. (2017) use optical projection and tangible interaction. The advantage is that this kind of systems has a natural viewpoint and touch feeling, but the disadvantage is that it requires a projection medium, such as a tangible object’ surface, which usually makes tangible objects not leave the tabletop and lack 6-DoF interactive space. In addition, this kind of systems generally requires a dark lighting environment which is inconvenient for tracking using color information due to the influence of projected light.

Some researchers Maier and Klinker (2013), Seo and Lee (2013) stick the markers on the tangible devices for tangible interaction. However, these systems are based on a single camera. In flexible tangible interaction, MR registration often has errors due to the large angle or distance between the marker and the camera. Controlling the orientation of the marker to face the camera will affect the comfort of the users. Moreover, most of the existed MR environments lack systematic human-oriented research and module design.

2.2 Needs theory of human

Benware and Deci Benware and Deci (1984) find that students who use actively orientated learning have higher internal motivation than those who use passively orientated learning. Ryan and Deci Ryan and Deci (2000) prove the importance of intrinsic motivation based on contemporary research and theories, and it reflects the natural human propensity to learn and assimilate knowledge. The mainstream motivation theory is MHN theory Healy (2016). It believes that everyone has five layers of needs. In terms of motivation, each need layer must be substantively met before the next goal can be activated. At the same time, once the needs of a certain layer are substantially met, it no longer has a stimulating effect, and the next layer will become the dominant need.

MHN theory has been effectively applied in many fields. For example, in management, the research results of Shoura and Singh Shoura and Singh (1999) show that it is desirable to use MHN theory to improve employees’ work consciousness and enhance their personality and sense of mission. In terms of career choice, Goel et al. (2018) find that the motivational factors for students to choose medical study are equivalent to MHN theory. In the medical industry, Benson and Dundis Benson and Dundis (2003) apply MHN model to motivate people to face the growing demands of the healthcare industry. In terms of learning, Kiel (1999) propose through this theory that we need an open and broad structure to better realize ourselves and carry out lifelong learning. In terms of mental health, Lester et al. (1983) summarize and develop a questionnaire to measure people’s satisfaction with the five basic needs described by Maslow, and propose that the level of satisfaction with basic needs is related to the scores of neuroticism and belief in an internal locus of control. In terms of long-term immersive VR systems, Guo et al. (2019) propose the needs based on MHN theory in VE to guide the design of VR office. But they only consider three layers of MHN theory. MHN theory has inherent logic and is universally recognized. We believe that based on the needs theory of human we can effectively design an experiment system that is comfortable, close to the real experiment experience and stimulate users’ learning motivation.

Needs model of virtual experiments

Full size image

The goal of the virtual experiment is to let students gain motivation and interest in a comfortable, safe, and realistic virtual environment in order to obtain good learning results. To achieve this goal, as shown in Fig. 1, this paper proposes a human-oriented and systematic needs model for virtual experiments based on the MHN theory.

3.1 Basic needs

In the first layer of MHN theory, physiological needs are the most basic needs for maintaining human life. But in the virtual experimental environment, food, water and clothing, etc. do not constitute a problem. Therefore, in combination with the application conditions of virtual experiment, we expand the physiological needs to the basic needs, including sensory needs and system needs. Because the sensory experience is the basis of all perceptions of experiments for users, system needs ensure that users use the system effectively.

The real experiment is to fully mobilize the students’ multi-channel sensory experience, so that the students can master and understand the experiment faster. Therefore, we needs to simulate the multi-sensory experience of experiments in the MR experimental environment, including visual experience needs, tactile experience needs, action experience needs, listening and speaking experience needs, temperature experience needs and olfactory experience needs.

In the MR experiment environment, spatial presence needs and the presence of virtual objects are important for visual experience needs. Spatial presence is the cognitive sensation of being in a place Skarbez et al. (2017), with which users can accurately judge the position and distance in the virtual space Balakrishnan and Sundar (2011), Pollock et al. (2012), Luo et al. (2019). For MR experiments, it requires the designed system to have accurate, multilayer virtual-real occlusion functions. If the rendering of the lighting and shadow of a virtual object is incompatible with the real scene, the designed system is visually not conducive to the user’s immersion in the experimental interaction Chamilothori et al. (2019). It requires the system to have illumination consistency function.

System needs include basic equipment needs, experimental content needs, usability needs and saving needs. The basic equipment needs include the experimental devices, hardware and software systems, effective input and output systems. Experimental content needs require the system to have experimental procedures and experimental simulation phenomena conform to objective laws. Usability needs require the system to be complete, robust and easy to learn and use. An important purpose of virtual experiments is to reduce the cost. Since the virtual experiment system generally requires a computer, if it is based on HMD, additional costs are also required. In a virtual experiment environment with tangible interaction, the cost of tangible objects also needs to be taken into consideration. It requires the design of the system to balance saving cost with other needs.

3.2 Security and comfort needs

The second layer of MHN theory is security needs. We extend this layer to security and comfort needs, including safety needs, physical comfort needs and psychological comfort needs. A safe and psychological relaxed virtual environment prevents users from worrying about explosions, corrosion, flames, getting tired, etc., so they can focus more on experimental operations and principles. To meet safety needs and psychological comfort needs, how to design the presenting ways of (virtual/real) objects in MR system is what we need to consider. For physical comfort needs, studies show that compared to not wearing HMD, HMD will cause greater physical stress to the users Chihara and Seo (2018) and more likely to produce simulator disease Sharples et al. (2008), which is more obvious in the VR environment. In addition, in the virtual experiments, the user’s perspective should focus on the tabletop, and there is no need to turn the head to see the surrounding environment. Therefore, our system does not use HMD.

3.3 Belongingness needs

The third layer of MHN theory is the belongingness needs. The core of this needs is to reduce the loneliness of users. In the real experiment classes, the social interaction between teachers and students is an important factor that increases the emotional belongingness of students. In addition, students are accustomed to listening to the teacher’s explanation of the principles during the experiment, and they accept this consistent audio-visual learning method. Some studies Hacker et al. (2009), Machidon et al. (2018) show that virtual humans can make the user interface rich in expressiveness of emotional interaction, and bring users more sense of participation and enjoyable experience. This paper uses technology of virtual human to create a virtual teacher to meet this needs. The design allows students to reduce loneliness and enter the state of learning and experiment faster.

3.4 Esteem needs and self-actualization needs

The high-layer needs (layers 4 and 5) of MHN theory are esteem needs and self-actualization needs. Among them, esteem needs are divided into inner esteem needs and external esteem needs. Self-actualization needs are divided into interest needs and encouragement needs. The purpose of the two layers’ needs is to make users believe in their own power and let users feel the progress they have made step by step through hard learning, so as to achieve the effect of motivation. Therefore, we need to take certain incentive measures in the virtual experiment to stimulate more learning motivation of students. We believe that more positive feedback (praise voice, applause and other encouraging behaviors) and less negative feedback (criticism or blame voice) of teacher helps students to establish their esteem and self-actualization. In this paper, the virtual teacher’s voice and behavior combine interactive method to meet the two needs.

The environment of MagicChem. a System environment. b Virtual match lighting tangible alcohol lamp

Full size image

The environment of MagicChem is shown in Fig. 2a, including three cameras, a smell generator and a temperature generator. Users can freely interact with virtual and real devices. We use one RGB-D camera as the main camera and two RGB cameras as the auxiliary cameras. The main camera is placed in front of the human eye, parallel to the direction of the human eye toward the desktop to simulate the eyes’ angle. We take an interactive scene shown in Fig. 2b as an example to introduce MagicChem. One hand of the user holds the tangible alcohol lamp, and the other hand uses the virtual match to light the alcohol lamp.

We design four modules (user interface module, tracking module, rendering module, interaction module) for MagicChem to meet the needs model. The relationship between the modules is shown in Fig. 3.

Module design of MagicChem

Full size image

4.1 User interface module

This module is used to design the experimental environment of MagicChem, including the presentation form of the user interface, selection of tangible and virtual objects. In order to meet the visual experience needs and physical comfort needs, the user interface we designed is MR environment without HMD. In order to meet the security and comfort needs, the tactile experience needs and the usability needs, we design the tangible and virtual form for experimental devices. For devices that are cheap and suitable in size and weight, as well as easy to attach markers, we directly use real devices and put markers on them for tracking. For experimental devices that are expensive, big or heavy, we substitute these devices with some similar shape and lightweight materials of tangible substitutes (meeting tactile experience needs, saving needs), in order to reduce the physical pressure (meeting physical comfort needs). For devices that are inconvenient to attach markers, we use the form of virtual objects. For dangerous or polluting objects such as flames, corrosive liquids, toxic gases, we use the form of virtual objects (meeting safety needs).

The tracking module. a Error 3D registration of flame in single-camera environment. b Correct 3D registration of flame regardless of the orientation of the marker. c Gesture interaction with touching. d The results of tracking and reconstruction in the corresponding virtual scene

Full size image

4.2 Tracking module

The design of this module is to achieve robust tracking, registration, scene segmentation and illumination estimation to meet the security and comfort needs and paves the way for the functions of rendering module and interaction module.

In the traditional single-camera MR environment, the marker needs to be kept facing to the camera, when the angle or distance between the marker and the camera is too large, 3D registration will fail, as shown in Fig. 4a. So that the user needs to focus on controlling the orientation of the marker, thereby reducing the comfort. Therefore, we make multiple cameras cooperate with each other to solve this problem robustly. In this paper, we have one main camera and two auxiliary cameras to cover the user’s interactive space in multiple directions. In the offline stage, we calibrated the multiple cameras Zhang (2000). In the online phase, multiple cameras track markers’ 6-DoF pose Lepetit et al. (2009) in real time. Finally, the pose parameters are converted to the main camera coordinate system. Figure 4b shows even if the marker on the alcohol lamp is not facing to the main camera, the virtual flame is correctly registered on the tangible alcohol lamp when user uses a virtual match to ignite the tangible alcohol lamp.

The module is also used to segment multiple foreground objects to provide the basis for virtual-real occlusion of the rendering module. We create a dataset of the foreground objects used in the experimental environment, and introduce the Unet network Long et al. (2015) with convolutional block attention module Woo et al. (2018) to conduct the training and the real-time pixels segmentation.

This module is also used for the detection of light sources in real scenes, as a basis for the consistency of illumination of the rendering module. Since this study is in the indoor experimental environment with controlled illumination, we estimate the parameters of multiple light sources in the offline stage. We use a general source model Zhou and Kambhamettu (2008) to estimate multiple illuminants of different types. We use a plastic sphere to probe both diffuse and specular reflections, using specular reflections to estimate geometric parameters and using diffuse reflections to estimate photometric parameters of the illuminants.

The module also tracks the poses of the hand joints and registers them to the virtual scene, providing the basis for the gesture interaction with touching of the interaction module. We tried the popular gesture tracking device Leap Motion, but the robustness was not ideal due to occlusion and confusion with tangible devices. Therefore, we design a quick and robust way to implement gesture interaction in this system, attaching red and blue markers (these colors seldom appear in tangible objects) to the most important thumb and index finger of the hand for tracking, and reconstructing joints into virtual scenes for collision detection. Figure 4c is the MR scene in which the left hand takes the virtual gas cylinder and the right hand takes the virtual dropper. Figure 4d is a virtual scene corresponding to Fig. 4c, where four different colored spheres represent markers on four fingers.

The rendering module. a No virtual-real occlusion. b Our virtual-real mutual occlusion method. c No illumination consistency. d Our illumination consistency method

Full size image

4.3 Rendering module

The design of this module is to meet the visual experience needs. This module is mainly used for rendering the virtual and real fusion of MagicChem, including the realistic rendering of the virtual-real occlusion (spatial presence needs) and illumination (presence needs of virtual objects).

The traditional virtual-real occlusion method is the original depth data of real scene to compare with depth data of virtual scene Yang et al. (2019). Generally, there are be many errors due to low-quality depth data. We correct the depth map of the real scene through the accurate pixel area of each type object segmented by the tracking module. Then, we complete the depth test by sampling the depth map of virtual scene and the corrected depth map of real scene. Finally, opaque objects and semi-transparent objects are rendered according to light sources’ parameters calculated by the tracking module. Figure 5a shows an interactive scene without virtual-real occlusion where the right hand holding the virtual dropper, and Fig. 5b is the MR image rendered by this module. It can be seen that our method can handle multilayer virtual-real occlusion. Figure 5c is an interactive scene without illumination consistency where the left hand holding the virtual stalactite, and Fig. 5d is the MR image rendered by this module. It can be seen that our method can make MR experiments more realistic.

4.4 Interaction module

Multi-modal interaction is more similar to the multi-sensory experience that humans encounter in the physical world Dumas et al. (2009). This module is designed to meet the sensory needs, belongingness needs, esteem needs and self-actualization needs.

4.4.1 Visual interaction

Visual interaction is used to meet the user’s visual experience needs in the experiments. This interactive mode mainly relies on the visual effects of MR to bring users a good visual experience. It depends on the consistency of virtual-real occlusion and illumination provided by the rendering module. Moreover, we simulate experimental phenomena and experimental details to bring users a realistic visual experience. Figure 6a shows the soft deformation of the dropper when the user squeezes the dropper and the ripples caused by the liquid dripping into the water surface.

Multi-modal interaction in MagicChem. a Gesture interaction by touching. b Tangible interaction. c Voice interaction and virtual-human interaction. d Action interaction of virtual teacher. e Temperature Interaction. f Olfactory Interaction

Full size image

4.4.2 Gesture interaction

For the interaction with virtual devices, we use gesture interaction with touching to meet the user’s action experience needs rather than specific actions that do not match the real experimental operation. Since it is easy to judge the positional relationship between the finger and the virtual object, the MR environment with virtual-real occlusion provides the possibility for gestures interaction with touching. For gesture interaction, we use the finger joints registered by the tracking module to detect collisions with virtual devices. In order to prevent accidental pick-up caused by accidental touch, the paradigm of gesture interaction is: when two finger joints of a hand are in contact with two sides of the virtual device at the same time, it is regarded as picking up the virtual device. If the distance between the two finger joints is too large to meet the picking conditions, the virtual device will fall onto the desktop. As shown in Fig. 6a, the real hand interacts with the virtual dropper like real experiment.

4.4.3 Tangible interaction

Tangible interaction is used to meet the user’s tactile experience needs. Based on tracking module, we render virtual objects like drugs, reagents, flame, etc., on tangible devices robustly. Moreover, we call some handmade or 3D printed objects that are similar in shape to the target devices as tangible substitutes. For these tangible substitutes, we render them as target devices as needed in MR images. As shown in Fig. 6b, the left hand holds the tangible alcohol lamp emitting a virtual flame, and the right hand holds the tangible temperature probe for temperature measurement. The high-precision thermometer (rendered by tangible substitute) is placed on the desktop.

4.4.4 Voice interaction

Voice interaction is used to meet the user’s listening and speaking experience needs.

In terms of listening, we use virtual teachers to provide users with guidance on experimental process and principles. Figure 6c shows the virtual teacher explaining the principle of the reaction of sodium and water. It’s worth mentioning that, instead of a machine-generated voice, the voice of the virtual teacher is from a chemistry major student, which is more friendly, persuasive and more enjoyable Stern et al. (2006), Cabral et al. (2017).

In terms of speaking, we calibrate the desktop by placing a marker on the desktop and used it as a “devices warehouse” to generate and destroy virtual devices. The virtual devices are generated on the “devices warehouse” through the user’s voice instructions. Moreover, if the user’s voice is monitored and recognized throughout the entire experiment, the real-time performance and recognition rate of the system will be reduced. We design a virtual microphone (Fig. 6c) as an interaction constraint to overcome this problem. The user needs to use the thumb and index finger of one hand to touch the virtual microphone for starting to monitor user’s voice. When two fingers leave the microphone, the monitoring is ended.

4.4.5 Virtual-human interaction

In order to meet the needs of belongingness, esteem and self-actualization, we set up a virtual teacher to provide users with virtual human’s behavior and voice interaction. The virtual teacher is an optional auxiliary virtual person according to the user’s preferences, which can be called out through the user’s specific voice to assist the user in experiments. When the user completes an experiment step correctly, the virtual teacher will give encouraging behavior and voice feedback. On the contrary, the virtual teacher will give actions of shaking head or crossing hands, and then re-guiding this step. As shown in Fig. 6d, virtual teacher gives behavior feedback for the user’s operation.

Compared with the common monologue teaching, the virtual role dialogue teaching method can enhance the positive emotional factors of the learners Gratch et al. (2016). In addition to the voice interaction mentioned in Sect. 4.4.4, we design a test section with voice interaction to simulate the questions-answers of teachers and students in real classrooms.

The interactive mode of virtual teacher not only allows users to enter the state of learning and experiment faster, reducing their loneliness (meeting the belongingness needs), but also allows users to experience the process of completing the experiment step by step (meeting the self-actualization needs). Even if users make mistakes, they will be encouraged instead of criticized (meeting the esteem needs) to improve users’ learning motivation.

4.4.6 Temperature interaction

We made a temperature generator to meet the user’s temperature experience needs in the experiments. The temperature generator uses a PTC heating body as heat source, and uses a PT100 thermistor sensor to measure temperature, and combines a PID control algorithm to achieve temperature control. For example, when the tangible alcohol lamp in Fig. 6b is ignited to emit a virtual flame, the temperature generating device will emit hot gas to simulate heat release (Fig. 6e).

4.4.7 Olfactory interaction

We made a smell generator to meet the user’s olfactory experience needs in the experimental environment. By high-frequency resonance of ceramic atomizer, the high-frequency oscillation of the electron is used to break the liquid water molecular structure to produce a natural and elegant water mist. For example, when sodium reacts with water to generate hydrogen in Fig. 6c, the olfactory device will spray water vapor into the air to simulate the smell (Fig. 6f).

4.5 Details of system

In this paper, we use Intel Realsense D435i depth camera as the main camera and one Logitech C270i camera and one Gucee HD92 camera as auxiliary cameras. The tracking module and the interaction module are mainly developed through the C\(\sharp\) language on Unity3d. The rendering module is developed through cg/hlsl shader on Unity3d. The FPS of MagicChem is stable at about 25 frames/s.

5.1 The design of user study

In order to study the experience and learn effect of MagicChem, we design three ablation groups of independent variables as shown in Table 1. Among them, only the parts listed in E1, E2, E3 of Table 1 are different from E4(MagicChem). For user experience and learning effects compared with E4(MagicChem), the function of E1 is to test the impact of rendering module and the function of E2 is to test the impact of interactive modules. E3 uses Leap Motion for gesture tracking. We apply our needs model to the E3 to compare with E4 to explore whether a first-person perspective, virtual-real fusion and tangible interaction have an impact on user experience and learning effects. We do not choose real experiments as the ablation group because there may be dangers in the experiment process and difficult to control.

Table 2 illustrates three tasks that each user needs performing. T1 aims to make students learn the operating steps of the experiment “Reaction of sodium and water” and analyze the principle based on the reaction phenomenon (sodium floating on the water, sodium melting into metal balls, sodium swimming around, making a “hissing” sound, and the reaction product turns the color of solution to red). The specific operation steps of T1 include taking a piece of sodium from the bottle, taking a sink filled with water, putting sodium in the sink, and using a dropper to absorb the phenolphthalein reagent from the bottle and drop it into the water tank after the reaction is complete. T2 aims to make students learn to use temperature measuring device and explore the temperature difference between the inner and outer flames. The specific operation steps of T2 include igniting a match with a matchbox, igniting an alcohol lamp with a match, and detecting the temperature of the inner flame and outer flame of the alcohol lamp with a temperature probe. T3 aims to make students learn the operating steps of the experiment “Corrosion of stalactites”, observe the reaction phenomenon of stalactites with different types and different concentrations of acid, and then explore the principles of the phenomenon. The specific operation steps of T3 include putting hydrochloric acid/sulfuric acid/nitric acid in the sink, taking a stalactite from the stalactite samples and putting it into the sink, adjusting the concentration of each type’s acid.

Each participant takes 3 minutes to practice the interactive method of his group and then complete T1, T2, T3 in order. After each user completes all tasks, we use SUS Brooke (1996) to evaluate the degree of satisfaction on system needs, and a subjective questionnaire (Table 3) to evaluate the user experience. Table 3 includes questions of Nasa-TLX(Q1-Q6) Hart (2006) and some other questions (Q7-Q10) based on our needs model. The questionnaire uses a five-point Likert scale (1 to 5, where 1 means lowest and 5 means highest). After completing the questionnaire, users need to make further oral comments on the questionnaires. Finally, we ask users to describe the experimental steps, experimental phenomena and principle of T1 as detailed as possible within 2 minutes to evaluate the learning effect of the experiment. The purpose of this arrangement is to get a long interval between the operation of T1 and the description of T1, so as to better evaluate the effects of learning and memory. On average, the entire user study takes about 45 minutes per participant.

5.2 Experiments preparation

We recruit 56 students (30 males and 26 females) as users of the experiments. These students from the first grade of senior middle school in a local middle school, and they have not studied the experiments of T1. Students are between the ages of 14–16. All the 56 students have experience in doing real chemical experiments. We divide them into 4 groups (E1-E4) equally according to gender, academic performance, etc. Each group has 14 students. 5 of them have VR interface experiences from science and technology museum and VR experience museum, and none of them have experience of MR user interfaces. We briefly introduce the concept of VR/MR, and explain in detail how to use the system of the corresponding group.

5.3 Balance of experiments

We assign a staff member to supervise each experimental environment. All computers in environments are unified. In E2, since there is no guidance and principle explanation of virtual teacher, we use the virtual text-UI method instead. Moreover, E2 has no voice interaction, we set up a 3D-Menu. The users can click the target button on the 3D-Menu to generate virtual device through gesture interaction. Moreover, we ask users to concentrate as much as possible.

Results of the SUS scale, where error bars represent standard deviation. a The average SUS score. b Average score of the three types of indicators

Full size image

6.1 Analysis of system usability

The SUS score reflects the overall availability of each environment. Figure 7a shows the average SUS score of each experimental environment, and the error bar represents the standard deviation. We summarize the ten questions of the usability scale into three indicators, which are functional indicators (the integration of functions, the complexity of the system, the inconsistent of interactive behavior and results, and the cumbersome operation), mastery indicators (easy to learn, easy to use, requiring professional help, and requiring training) and application indicators (use in future and use with confidence). Figure 7b shows the score of three indicators (each question is the converted score Brooke (1996) of 0–4).

Table 4 shows the results of post hoc test for MagicChem and each ablation environment. Among them, E4&E1 represent two sets of users’ data of E4 and E1 for post hoc test, and so on. S represents the significance. Y represents yes, and N represents no. \(\uparrow\)(E4&Ei) represents that the average value of E4 is greater than the Ei(1\(\leqslant\) i\(\leqslant\) 3), and \(\downarrow\) is the opposite. The result shows that the total SUS score of E4 is significantly higher than E1 and E2. The difference in the total SUS score of E4 and E3 is not significant. E4 is significant higher than E1 in terms of all three types of indicators, higher than E2 in function indicators and application indicators, but lower than E2 in mastery indicators, and higher than E3 in application indicators.

Through users’ feedback and our analysis, we obtain the reasons for the significant difference. In E1, where there is no consistency of virtual-real occlusion relationship and the lighting and shadows, virtual objects are “floating” on the real scene to occlude the real scene. This visual confusion and falsehood greatly affects user interaction, especially gesture interaction with touching. Since the position of virtual objects can only be judged by the position of tangible objects in real scene, when there are few tangible devices in the scene, it is hard for user to interact with virtual devices. For example, when a user needs to pick up the virtual dropper, he/she is difficult to use his thumb and forefinger to touch the surface of the dropper accurately. It results in lower scores in the three types of SUS indicators for E1.

E2 has only one interactive method of gestures with touching, and the way to generate virtual devices is to click target button on the 3D-menu by gestures with touching. If the finger accidentally touches other buttons, unexpected virtual device will be generated and the user needs to find and select target button again. E4 generates virtual devices by voice interaction. The difference causes the score of E2’s functional indicators to be lower than that of E4. In addition, E4 has multiple interaction methods, and E2 only has single interaction method, which results in E2’s application indicators being lower than E4. However, in terms of mastery indicators, E4 causes users to spend more effort to learn multi-modal interaction than E2. Finally, we find that E4 has higher application indicators than E3, which proves that users are more willing to use MagicChem.

The mean score of subjective questionnaire

Full size image

6.2 Analysis of user experience

To evaluate the quantitative data collected from these questionnaires, we use ANOVA with post hoc test to analyze the significant differences between groups. Figure 8 shows the mean score of questionnaire and error bars which indicate standard errors of the means. Table 5 provides detailed values in post hoc comparison test.

The results show that E4 is significantly lower than E1 in psychological pressure, physical pressure, time pressure, effort and frustration, and is significantly higher than E1 in visual fidelity, natural operation and interest. The reason is that E1 lacks the realistic 3D feeling visually. Due to the lack of occlusion relationship, the user may need to go through many attempts to correctly take a virtual object, which requires more psychological pressure, physical pressure, effort, time pressure and frustration. The feedback of E1 is generally poor, such as “The virtual object floating on the real, I cannot judge the position of the virtual object, I can only guess and try,” “Very bad experience, it makes my sense of position confusing.” In E4, there is no negative feedback about spatial relationship and interactions. It proves that our rendering module satisfies the need for the presence of virtual objects. It also proves that the visual experience needs in sensory needs are very important to the experience.

E4 is significantly lower than E2 in physical pressure, own performance, effort, and loneliness, and is higher than E2 in natural operation and interest. Compared to E4, E2 lacks interaction in touch, temperature, smell, hearing and speaking, which is not as natural as E4. In addition, we find that the reason for the greater physical pressure and effort in E2 is that the way E2 generates virtual device needs more effort to find and click the target button, especially by accidentally touching. Due to the use of silent text-UI guidance, E2 lacks encouraging voice and action interaction from virtual teachers. The virtual environment is more “cold,” making students feel lonely (not meeting belongingness needs) and less progress (not meeting esteem and self-actualization needs) than E4. Moreover, although all the experimental devices in E2 are virtual, it does not lead to a significant difference in visual fidelity between the E4 and E2.

E4 is significantly lower than E3 in terms of physical pressure. For visual realism, natural operation and interest, E4 is significantly higher than E3. Completely virtual environment is the reason that E3 is lower than E4 in terms of visual realism. In terms of interaction, more operations are tangible in E4 than in E3. The tactile sense of taking objects is important to the user’s sense of interaction, which is the reason that interaction of E3 is not as natural as that of E4. In addition, the reason why the physical pressure of E3 is greater than that of E4 is that wearing HMD causes a certain amount of pressure on users. Many users in the E3 group give the feedbacks such as “HMD is heavy,” “It’s not comfortable wearing HMD.” According to communicate with users, the users in E4 group do not feel that MagicChem’s perspective is unnatural during the experiment. It also proves compared with the first-person VR environment, the design of E4 can better meet the physical comfort needs of users.

6.3 Evaluation of learning effect

We design a scoring standard with a total of 15 points, with the correctness of 5 steps (placing the sink, using a medicine spoon to dig out the sodium in the jar, putting the sodium into the sink, putting the dropper in the jar after reaction, absorbing and dropping phenolphthalein), with the correctness of 5 experimental phenomena (sodium floating on the water, sodium melting into metal balls, sodium swimming around, making a “hissing” sound, and the reaction product turns the color of solution to red), with the correctness of 5 experimental principles corresponding to phenomena (the density of sodium is less than water, the melting point of sodium is low and the reaction will exothermic, the generated hydrogen pushes sodium to move, the gas rushes out of the liquid surface to make the sound, reaction generates NaOH). Record 1 point for each item. Figure 9 shows the mean score of learning effect and error bars indicating standard errors of the means.

The mean score of learning effect

Full size image

The results of ANOVA and post hoc tests show that the learning effect of E4 is significantly better than E1 (p < 0.001) and E2 (p < 0.001), but there is no significant difference with E3 (p = 0.445). Through analysis, we believe that due to the lack of consistency of virtual-real occlusion and illumination in E1, the visual unreality and poor interaction experience affect the user’s focus of experimental phenomena and the principle learning. However, due to the lack of a virtual multi-modal interactive environment, E2 only presents knowledge through text guidance in visual rather than multi-sensory, it is not as impressive as E4. Although E3 is inferior to E4 in some experience indicators, E3 and E4 have no significant difference in the learning effect. It shows that when the virtual experiment environment meets the needs model we proposed, the learning effect of the experiments is affected by the way of presentation and perspective little.

The results of user study show that, in terms of usability, experience, and learning effects, MagicChem (E4) is superior to the two traditional MR environments (E1 and E2) that partially meet needs models. E1 received poor user reviews, which shows that students are very concerned about the visual experience in interactive MR scene. So in the design of the MR experimental system, if the presence needs of space and virtual objects can be met, the system’ practicability, the user experience and user’s learning effect can be greatly improved. Moreover, the more the user’s sensory experience needs are met, the better the experience and learning effect for students. In terms of application indicators, physical pressure, visual fidelity, interactive naturalness and interest, MagicChem is superior to the VR experiment environment (E3) that meets the needs models. However, there is no significant difference in the learning effect between MagicChem (E4) and the VR experiment environment (E3). It shows that when the virtual experiment environment meets the needs model, the learning effect of the experiments will be hardly affected by the way of presentation and perspective. In addition, MagicChem has received good feedback and evaluation from students, which is also reflected in the average score of the scales in terms of system usability, experience, and learning effect. The results of these test indicators also achieve our purpose of designing a human-oriented virtual experiment needs model: to guide the design of a virtual experiment system that is close to real experiments and have multi-sensory, comfortable and stimulating experience.

Based on the proposed needs model and the results of user studies, we propose the following design guidelines of the virtual experimental system:

The bottom-level needs (basic needs, security and comfort needs) of the needs model proposed in the paper are the “material” basis to ensure the stable operation of the experimental system and the good experience of users. First, the needs require the design of a virtual experiment system to be as practical as possible, including good system usability and complete experiment simulation, which can solve some shortcomings of real experiments (danger, pollution, chemical consumption, etc.). Secondly, the system is required to have as many multi-sensory simulations as possible, including vision, tactile, listening/speaking, temperature, olfactory, etc. that exist in real experiments. Among them, the visual needs are the most important, which greatly affects the users’ sense of reality experience. In particular, for the MR environment, visual needs depend on the consistency of illumination and the consistency of occlusion in virtual-real spaces. After satisfying the bottom-level needs, the high-level needs (belongingness needs, esteem needs, and self-actualization needs) are the “spiritual” assistance that improves the pleasure, satisfaction, and learning effect of users who use the experimental system. The needs require the design of the virtual experiment system to consider the social interaction of teachers and students, how to motivate students to learn, etc. In this way, the human-computer interaction environment will become “warm” and students will like to use the system to do experiments.

Although the paper proposes a general needs model for virtual experiments and develops a corresponding system, only a few typical chemical experiment examples have been researched. In the future, we will develop more experiments in different fields (physics, biology, etc.) based on this system framework, and conduct research to improve this needs model and explore how to make experiments in more fields benefit from the needs model. In addition, we will also explore multi-users collaborative virtual experiments (student-student social needs) to expand the belongingness needs of the needs model.

Combining the human needs theory and the special characteristics of virtual experiments, the paper proposes a human-oriented needs model for virtual experiment. In order to verify the effect of the needs model on students, we design an MR experiment system MagicChem based on this needs model and conduct user study. With different aspects of user studies, we find that MagicChem is better than the virtual experimental environment of unsatisfied or incompletely satisfying the needs models. This shows that the needs model we proposed can effectively improve experience and learning effects of the student users.