A Novel Collaborative Virtual Reality Game for Children with ASD to Foster Social Interaction

Autism spectrum disorders (ASD), characterized by impairments in communication and social interaction, represent a group of neurodevelopmental disabilities [1–3]. According to the reports of the Centers for Disease Control and Prevention, the number of children with ASD (6–17 years) has steadily increased from 1.16 % in 2007 to 2.00 % in 2012 [4]. In 2012, 1 in 68 children (8 years) was diagnosed with ASD [5]. Children with ASD often display disabilities in developing competence necessary for appropriately interacting with their typically developing (TD) peers in group tasks. Social challenges are even exacerbated as they get older with limited social skills facing more complex social situations. Various intensive ASD interventions have been employed to improve the deficits of social competence for children with ASD. Recently, collaborative virtual reality technology is applied in some ASD interventions enabling multiple individuals in distributed locations to interact with one another within a safe, flexible and collaborative environment. In the CVE-based interventions, individuals with ASD usually experience less anxiety as face-to-face communications are reduced in CVEs [6]. They get rid of physical distance restrictions and can accept the interventions outside the clinic [7]. CVEs are also capable of creating motivating collaborative tasks that inspire participation of children with ASD in the interactive work so as to develop communicative and social interactive skills.

A growing number of studies have begun to explore the influence and effectiveness of CVEs used in ASD interventions. Weiss et al. created a CVE program called TalkAbout for children to learn and practice social conversation skills with their partner [8]. Cheng et al. developed a collaborative virtual learning environment (CVLE), where students communicated with a virtual teacher to answer questions according to the context of social scenarios so as to learn social techniques (including social cognition and interaction) [9]. Stichter et al. used a distributed 3D CVLE, iSocial, for youth with high functioning autism (HFA) to complete collaborative tasks (e.g., to design and build a restaurant) in the shared virtual environment [10]. Millen et al. introduced a CVE called Island of Ideas that encouraged students with ASD to take part in participatory design activities [11]. Battocchi et al. employed a puzzle game featuring with enforced collaboration on a tabletop to facilitate cooperative behaviors in children with ASD [12].

Our distributed CVE system, called Hand-in-Hand (HIH) CVE system, aims to provide a flexible and effective interaction environment and eventually foster communicative and interactive behaviors among users through collaborative tasks. Since distributed users cannot “read” partners’ intentions through face-to-face observation, they are more likely to talk with one another to share ideas. The nature of collaborative games raise the awareness of the importance of cooperation between users and also requires significant conservations. This system is expected to engage children with ASD in the collaborative tasks implemented with Leap Motion cameras [13], and record performance data as well as conversation audio in real time. In this paper, we present the design of this system and evaluate the efficiency of this system via a usability study. The rest of the paper is organized as follows: Sect. 2 discusses the system structure and game design; Sect. 3 presents the preliminary results of the usability study; and Sect. 4 concludes with a discussion of the contributions and limitations as well as future work.

The major goal of this system is to provide a CVE where distant users can play games in a collaborative manner. We first created the CVE that users can access from distant locations via Internet. This CVE was built using Unity [14] and designed as a 3D game space supporting network connection and simultaneous manipulations from distributed users. Next, we designed a series of collaborative games in this CVE with implicit rules to foster social interaction between users. These games integrated with the Leap Motion offer a chance for players to control virtual game elements in the CVE with their hands. Along with these games, a database was developed to collect gaming data including audio data for offline analysis. Figure 1 shows major components of the HIH CVE system. The user interacts with the CVE Application via the Leap Motion controller. Communication Management enables the distant CVE application connection and real-time communication between the users. All the gaming data and audio data of conversation are recorded by the Data Management component.

Overall architecture of the HIH CVE system

Full size image

2.1 Collaborative Tools

All the games in this CVE are required to play with Leap Motion controller, which is a gesture-based interactive tool. Instead of traditional input devices such as mouse or keyboard, the Leap Motion was chosen in order to provide a more naturalistic way of manipulating virtual game objects and to make users feel more immersed within this system. We programmed the Leap Motion Controller to develop two virtual collaborative tools (Fig. 2) controlled by hand gestures/motion. The user’s actions can be mapped to the behaviors of controlled tools. In the CVE, the user’s hand is represented by a circle. When the user’s hand (the circle) touches the tool, the user can manipulate the tool. For example, when the user clenches his/her hand, the tool is grasped and can be moved to any location. When the user opens his/her hand, the tool is released.

Move Tool (left) and Collection Tool (right)

Full size image

These two collaborative tools are called Move Tool and Collection Tool. They were designed to enforce collaborative activities between users in three ways: (a) they require matching efforts from each player, (b) they only function through collaborative operation, and (c) they display the effect of individual operation. Both tools have two handles, each of which is required to be controlled by one user. These tools become functional only when users coordinate their operations on the handles. When using the Move Tool, each handle is limited to move in one direction (horizontal direction or vertical direction). Thus each user respectively controls one moving direction of the puzzle piece. As for Collection Tool, both handles are prohibited to go beyond a certain distance and users are forced to coordinate their heading directions.

2.2 Game Design

The primary part of the CVE application is a series of collaborative games. When we started designing those games, we focused on creating those that (a) can be easy to learn and play, (b) require players to undertake equal work, and (c) have implicit rules for fostering extensive interactions and communications. Therefore, we chose games in the form of collaborative manipulation of objects from one location of the virtual space to another, sometimes in the presence of obstacles. Finally, we designed three such types of collaborative games, where players virtually hold/move/drop the virtual objects using handles via the Leap Motion controllers. These games are called Puzzle Game, Collection Game and Delivery Game (Fig. 3). We modeled these games using Statechart diagrams (Fig. 4) to support hierarchy and concurrency.

Puzzle Game (left), Collection Game (middle) and Delivery Game (right)

Full size image

Statechart diagrams modeling the Puzzle Game (top), Collection Game (middle) and Delivery Game (bottom).

Full size image

Puzzle Game requires players to put nine puzzle pieces together to match a target picture. In the Statechart model of Fig. 4 (top), the “TIME” mode records the game time and would end the game after 5 min. The “PUZZLES” mode controls the states of nine puzzle pieces, which have the same low-level mode “Puzzle [Num]”. Initially, nine puzzle pieces “Stay” dispersedly in the game space. When both players choose the same piece and grab their handles, the piece would “Move” following the Move Tool. When either player releases the handle, the piece would be put down. If the piece is put down at the correct destination, it means the placement of this piece is “Finished”. Once all the puzzle pieces reach the correct locations, the game would be ended even before the timeout expires. To win this game, two players are required to collaboratively determine the move order of puzzle pieces and the correct destination of each piece.

Similar to Puzzle Game, players bring nine different objects to collection areas where their pictures are shown in the Collection Game. Compared to Puzzle Game, an additional “Target” mode is added into the model of Collection Game. This mode performs the function of randomly displaying one target picture within one of three collection areas every 15 s (defined by “t”) or after one object is successfully collected. We created three Collection Games with some differences in rules. The Statechart diagram of Fig. 4 (middle) is for the first Collection Game. In the first and second Collection Game, every 15 s, only one target picture is shown. But in the first one, the target picture is visible to both players, while it is invisible to one of players in the second game. In the third game, two target pictures are shown and each of them is only visible to one player. Accordingly, operation conflict happens when two players head for different targets. In order to win, two players have to share information and make concerted actions.

The goal of Delivery Game is to deliver three stars to three blue-star areas surrounded by several obstacles. The Statechart diagram in Fig. 4 (bottom) explains this game. The “OBSTACLES” mode manages the states of obstacles according to the predefined logic. Players are supposed to deliver stars one by one and avoid obstacle areas. The star will fall into the obstacle areas and go back to the “InitialLocation” when crossing these areas. Players are rewarded with different points for three successful deliveries. Therefore, to achieve higher scores within limited time, it is important to make a reasonable delivery order and choose safer and faster paths from several available paths. We also modified the layout of obstacle areas to create three Delivery Games. In this type of game, players need to perform collaborative manipulation as well as collaborative path planning.

2.3 Communication Management

The word “Communication” here refers to user communication and game connection. Skype [15] is used in this system to support real-time communication between users. As a popular communication application, Skype is easy to install and use, and provides good video and audio quality [16]. The nature of distributed multiplayer games demands real-time synchronization and consistent maintenance of game states on distant computers. We used server-client architecture to implement data exchange across the network between distant CVE applications. The server runs the major programs. Clients connected to the server constantly receive data from the server in order to carry out their own purpose. When running two CVE applications, any one application can be selected as the server, while the other one connects to the server via a specified IP address and plays the client role. As shown in Fig. 5, the server and the client separately process each player input in the form of hand gestures and perform the logic of handle control locally. Any information regarding local actions would then be sent to the server, which carries out game rules according to the information and synchronizes all actions with the world state so as to update the game states. After that, the server propagates the updated data to the client so as to synchronize the game state on both sides.

Game data exchange and update between two CVE applications

Full size image

2.4 Data Management

The database collects objective process data, which is indexed by time and stored as *.csv files in real time. It also records users’ conversation in the form of audio. Based on these data, we can summarize a series of offline quantitative measures to evaluate the performance and interaction of the users (Table 1).

3.1 Participants

We conducted a usability study to evaluate the HIH CVE system with 12 participants. This study was approved by the Vanderbilt Institutional Review Board (IRB) and supervised by engineers and ASD therapists. All the participants were first paired based on age and gender and divided into three ASD/TD pairs (three male pairs) and three TD/TD pairs (two male pairs and one female pair). Before the experiment, participants’ parents completed the Social Responsiveness Scale, second edition (SRS-2) [17] and Social Communication Questionnaire Lifetime Total Score (SCQ) [18] to quantify the current ASD symptoms of the participants. Participants’ detailed information is shown in Table 2.

3.2 Procedure

This study was conducted in our two separate experimental rooms (Fig. 6). Paired participants were invited to use the system and complete all the sessions lasting about one hour. Before starting the experiment, experimenters explained the experimental procedure, length and devices to participants and taught them how to use the Leap Motion Controller for interaction.

Experimental setup

Full size image

Participants then went to separate rooms to interact with the system. They would wear the headsets during the whole experiment in order to talk with their partners. As shown in Fig. 7, each participant first took an independent practice game that helped them get familiarized with the handle’s operation. The length of this session depended on the abilities of participants. Following the practice session, a brief tutorial of all games informed participants of general game rules and encouraged them to cooperate with one another in the game session. Note that this tutorial did not cover everything; participants still needed to talk with their partners to figure out how to play the games. The game session consists of eight separate games. Experimenters did not provide help for them or intervene during this session. Participants played Puzzle Games in pre-test and post-test. These two games used different target pictures with similar visual complexity. Therefore, from the results of the pre-test and post-test, we could compare the change in participants’ performance and interaction before and after training. After the pre-test, they entered the training part which consisted of three Collection Games and three Delivery Games to foster social understanding and collaborative skills. As mentioned before, all these games were different in terms of game difficulty or game setting. So participants could be kept interested in the games and negotiation with their partners at all times.

Experimental procedure

Full size image

At the end of the experiment, participants filled out a questionnaire and received gift cards as compensation. Twelve questions in the questionnaire obtained participants’ feedbacks regarding their levels of engagement, performance and communication as well as their perception of the game quality. Seven of the questions were scored on a five-point scale based on self-assessment (where 5 = “like very much”, 1 = “not at all”). Four of the questions were answered by choosing one answer from three choices. The last question was optional asking about suggestions for game improvement.

4.1 User Experience

All of the participants completed the entire experiment. They each had prior video game experience and could figure out how to use this system easily. None of them had ever used Leap Motion but most of them could quickly learn how to operate it. Most participants expressed strong curiosity about the Leap Motion device and felt that “the technology is so cool!” Table 3 shows the feedbacks of user experience. Questions 1–7 received mostly positive feedbacks (Mean > 3). Participants perceived the games as interesting and easy to play with their partner. They also thought themselves and their partners did a good job in the games. They even gave highly positive values (Mean > 4.5) regarding the importance of communication and cooperation in the pursuit of success. The only negative value was for “Easy to control handle?” We concluded two likely reasons for the negative feedback: (a) some participants did not place the hand in the appropriate position thus reducing the detection accuracy (e.g. the hand is placed too low, too high or tilted too much), and (b) some participants’ hands were too small for the Leap Motion to correctly detect.

From the group perspective, the TD/TD group had a little more fun with the games compared to the ASD/TD group. However, participants with ASD showed relatively deep interest (Mean = 4) in the games, which indicated these games were engaging for them. Additionally, participants in the TD/TD group thought more highly of their partner (Mean = 4) than of themselves (Mean = 3.33), unlike what was seen in the ASD/TD group. It seems that the participants with ASD were especially very satisfied with their performance (Mean = 4.33), while their partners did not rate their performance as highly (Mean = 3.33). This interesting result might speak to some aspects of social manners, such as modesty and praising others. In fact, we found that some TD participants really played very well, but only chose “Good” instead of “Excellent” in self-evaluation. In the TD/TD group, the maximum value for self-evaluation was 4, compared to 5 for partner-evaluation. As for the fourth question, participants with ASD were “optimistic” about cooperation with their partners; however, participants with ASD gave a relatively low value on the importance of communication. As for the cooperation, both groups had the same attitude that it was necessary and important. This seems appropriate as each game was designed to be a collaborative task.

The number of answers to each choice of questions 8–11 were counted and shown in Table 3. Numbers in the brackets below the “ASD/TD” column represent the answers of participants with ASD (the first value) and TD participants (the second value). Most participants agreed that they talked “very often” in the games, not just “when I needed” and believed “Talking with my partner” was the most useful way to learn how to play the games. Both groups gave different opinions on “Did you play better?” The TD/TD group felt “better” by the end. But TD participants in the ASD/TD group felt “same” while their partner felt “better.” This is reasonable because, from the performance analysis (discussed later in detail), the TD/TD group achieved much more progress than the ASD/TD group. It also suggests that participants with ASD were more pleased with their performance. For the question 11, in accordance with the results for question 7, both groups highlighted the usefulness of “working closely with my partner.”

As for the optional question, except for responses related to game design, some participants felt it was a little difficult to control the handle, which indicated that even though they were curious about the new technology of Leap Motion, they could not adapt to it quickly owning to unfamiliarity and the limitations of device.

4.2 User Performance

The results regarding user performance are shown in Table 4. The significantly decreased mean play time, increased mean collaborative operation efficiency and finished pieces demonstrate that, on the average, participants could spend less time, cooperate more efficiently, achieve higher rewards, and thus performed better collaboratively in the post-test than in the pre-test. Moreover, participants talked more in the post-test, which was an unexpected finding. This may be an indication that participants still needed or wanted to communicate with each other in our CVE system even after they had become familiar with the games.

Comparing both groups, we can see that the TD/TD group played better, especially in terms of “collaborative operation efficiency”. The TD/TD group communicated more frequently in the post-test by increasing about 30 words per minute. Although corresponding growth in the ASD/TD group is weak, it is worth noting that the participants with ASD had a higher level of communication than their TD partner and demonstrated a similar increase trend (Diff. = 12) as that of TD participants in the TD/TD group (Diff. = 15).

In this paper, we present the development and implementation of a novel distributed CVE system capable of providing a flexible interaction platform for children with ASD and their peers, and collecting quantitative and objective metrics regarding collaborative and communicative performance. The system integrated with the Leap Motion employed a series of collaborative games with implicit interaction rules. Based on participants’ feedbacks, they were engaged in these games and motivated to play well, even though they were not well adapted to the use of the Leap Motion device. They also understood the importance of cooperation and communication to win each game and worked hard to achieve success. The real-time communication tool guaranteed flexible and uninterrupted information exchange under the distributed condition. Although current experimental results were limited and we cannot able to generalize the conclusion that children with ASD can improve social competence through our system, the results are promising and demonstrate the potential value of our system in fostering collaboration and communication among children with and without ASD.

In the future, we will refine the system to support better gameplay experience. Considering the limitations of the Leap Motion device, we may adjust its properties based on the children’s characteristics or replace it with another input device. We are investigating haptic devices that provide physical contact [19]. We also want to expand the sample size of the user study to obtain the statistical power of results in verifying the influence of this system on children. Additionally, we found that several participants’ conversations not only involved contents about games, but also referred to other social information. For example, they asked each other’s name, age and the games they played in daily life. Therefore, we plan to analyze the conversation content emerging from the use of this system in order to deeply understand the behaviors of children with ASD.