Which Virtual Piano Keyboard for Children with Autism? A Pilot Study

Literature reports the positive effect of music in treating specific symptoms in autism spectrum disorder, specifically influencing communication, socialization and behavior [7, 8, 13]. Unfortunately, very few studies involve actually teaching music to people with autism. Playing an instrument proficiently requires long training involving many abilities. Individuals with autism may have problems paying attention, decoding notes in the sheet music, interpreting the note values, coordinating hands, and so on. To empower them, it is necessary to exploit their strengths and mitigate their weaknesses.

Autism spectrum disorder (ASD) is defined by the American Psychiatric Association in the “Diagnostic and Statistical Manual of Mental Disorders” (DSM-5), as delays or anomalies emerging in children before age 3 years in at least one of three areas: social interaction, communicative and social use of language, and symbolic or imaginative play. ASD makes it hard to understand and interpret the world and what is happening around one. Impacting on communication and socialization, it compromises the ability of a person to interact with peers. People with autism often need rigid routines and highly predictable events, move comfortably only in familiar and uncrowded environments and they often display behavior inappropriate to the context, especially in new situations. The spectrum of symptoms varies widely from individual to individual, from mild to severe. For this reason the teaching approach needs to be customized according to the individual’s demands and learning pace.

The enormous increase in prevalence of the syndrome, “about 1 in 59, of 8-year-old children,” as estimated by the CDC’s Autism and Developmental Disabilities Monitoring Network [4], underlines the urgency of applying effective rehabilitation therapy to address all functional areas of the subjects. Early behavioral intervention (possibly before age 3 years) can take advantage of the great plasticity of a child’s brain [1].

According to the empathizing-systemizing theory [1], a highly structured intervention can support learning in children with low-functioning ASD. They often need clear and simple interaction patterns, avoiding errors in the initial learning stages to effectively gain skills repeating trials, and facing tasks with a gradual increase in degree of difficulty. A structured learning approach offers a predictable and coherent learning environment. This reduces the subject’s frustration or anxious episodes, and thanks to the rewards provided, attention increases while self-stimulation decreases. Evidence in literature suggests the use of technology within a behavioral and holistic intervention, to support the individual in the learning process.

We are carrying out a multidisciplinary study on effective strategies for teaching music to people with autism using the Suoniamo app, a Web tool for teaching music specifically to these users by exploiting mobile devices (tablet or laptop) [3]. The Suoniamo fluid layout enables smooth interface resize to adapt to device screen dimension. All the UI components, basics or augmentatives (described in the following) are the result of literature analysis together with iterative evaluation of mid-term prototypes conducted during co-design sessions involving professionals with different learning experiences with people with autism. We focused on participative design in order to satisfy accessibility/usability aspects that are relevant when designing for special needs people, also taking into account individual needs. Customization is a key feature when teaching people with special needs, even more so if dealing with the complexity of the autism spectrum disorder.

In this paper, we will describe a pilot test with four teenagers with autism, aimed at evaluating the usability of the Suoniamo app user interfaces for the piano keyboard in three different configurations.

A national project conducted by the National Autism Center in 2009^{Footnote 1} formalized the role of music therapy as an “evidenced-based practice to adopt, exploiting songs and rhythm, when teaching individual skills to subjects with autism” [11]. Several other works in literature report the positive effects of music in treating specific symptoms in autism spectrum disorder. Summarizing a recent review concerning music interventions for children with autism [13], twenty studies focused their attention on music’s influence on communication, socialization and behavior.

Concerning communication, music has been used to improve expressive and receptive skills in different ways, such as using text sung or spoken to music or a rhythm [5], or using improvisational music [5, 14]. The use of music for socialization mainly used engagement as a focus of the research, one study using music to decode emotions [9]. All studies showed an improvement in peer-initiated interaction, but generalization and lasting effects were not shown [14, 15]. Finally, most studies cited in this review focused on using music to increase appropriate behavior and reduce vocal stereotypy. Approaches adopted included listening to music and music combined with social stories [2]. General results, reported in the studies examined, were positive, but not all individuals with autism find listening to music a positive experience [6].

Many studies confirmed that music therapy could improve social behavior and joint attention in children with autism; more evidence is needed to determine the impact of group music therapy [12]. The work of La Gasse [10] is an attempt in this direction, providing some evidence that music therapy group sessions targeting social skills may improve joint attention and eye gaze toward other persons.

Guidelines for teaching music to children with autism mainly focus on methodology (how to set the didactic environment and to schedule activities), omitting principles for the structured learning environment that guarantees an accessible and effective way to learn. The multidisciplinary Suoniamo project aims to design an accessible highly structured learning environment to teach piano to individuals with low- and medium-functioning autism. The target users are adolescents with autism (age 11–15 years) but the app can be exploited by younger children as well as adults.

The app provides three basic learning units that start the training by following note order (starting from C) and the random modules that verify the concepts previously mastered in a generalized and more challenging way:

1. (a)

   Note Discrimination (ND), playing the single notes in sequence (C, C#, D, D#, E, etc.) regardless of duration, in order to identify the position of notes on the piano keyboard. At first, the software proposes notes in ordered sequence to help memorize the scale; afterwards, the student can move to play random notes (program generalization).

2. (b)

   Note Value (NV): requires playing the single notes in sequence for the duration (4/4, 2/4, 1/4) indicated by a visual discriminative stimulus. This program also includes training in the pause concept.

   The training includes three units: it starts from whole notes (4/4) proposing trials on notes in sequence and then the pause (4/4). Once this concept is mastered, the training moves on to half notes (2/4) and next (after having mastered the previous one) to quarter notes (1/4).

3. (c)

   The C Major scale, ascending and descending, to learn the central, left and right scale on the keyboard. This training exercise shows in the pentagram the entire scale to be played, starting from whole notes at the beginning, and once mastered, moves on to half notes and next on to quarter notes

Each learning unit offers a sequence of trials within a predefined workflow. Moreover, colors and visual feedback are used as augmentative stimuli to facilitate task completion without frustration. Errors during task execution are avoided in order to implement the errorless principle of behavioral teaching. After the training, the student should be proficient in decoding pentagram language, playing notes, waiting for pauses and performing simple scores. Another interactive module is provided for supporting the execution of simple pieces of music through graphic cues. Once mastered on the virtual piano keyboard, the program might be generalized on a real piano or via a physical keyboard that can be connected to the app via USB port.

Considering the learning units described above, the student’s work environment should guarantee a logical workflow of tasks and fluid design of the graphic components of the user interfaces (UIs). The most relevant insights gained during ongoing tests mainly concern UI design, since potential improvements in the workflow generally require longer evaluation times.

Since people with autism are often visual learners, each UI is a dynamic entity that exploits the user’s visual channel to convey information. It includes basic elements such as virtual piano-keyboard, pentagram, notes, etc., and elements defined as ‘augmentative’ that implement accessible learning strategies. A preliminary pilot study with three students with autism highlighted several limits, especially in the augmentative components; those limits have been overcome through a new version of the app, the target of the current test discussed herein. Of these elements, the most important is note-color mapping to simplify note identification and positioning on the keyboard. We have implemented the association note-color proposing as default combination the rainbow spectrum and providing exercises for note discrimination using a standard piano keyboard (white and black keys). In an earlier version, only the note being tested was colored. As shown in Fig. 1, the discriminative stimulus requires identifying each note on the keyboard and the target note is colored to avoid errors.

Note discrimination module - old version

Full size image

This simplified the task enormously; in fact, results of previous tests showed the complete success of all participants in the discrimination task. Subsequent observations revealed that such a prompt originated only a basic stimulus-response dynamic since the user reacted only to the color and not to the challenging task showed at the top of the UI. Consequently, the new implementation offers the piano keyboard completely colored to make the learner to pay more attention to the task request, thus reducing the stimulus-response effect (Fig. 2).

Note discrimination module (ND) – new version (1-octave)

Full size image

Another challenge is the concept of note value, in terms of time duration. Learning note value implies the ability to count the flow of time, given a unit of measure, and the ability to inhibit the response to a given stimulus (balance excitation/inhibition). Both have enormous implications in everyday life. In this sense, an active experience of these concepts through music could potentially improve real life behaviors. What could be the best ‘augmentative’ components to convey this concept? As shown in Fig. 3, the UI proposes two main elements: (i) an emoticon that communicates to the user either positive or neutral feedback related to the completion of the task (the figure shows a neutral case); (ii) A progress bar calibrated on the time signature (the default is 4/4). As the student presses the key on the virtual keyboard, each quarter progressively fills in. This element helps the user to count the flow of time. Both components have been identified through successive design refinements and are currently once again being tested.

The Note Value (NV) module (1-octave version)

Full size image

Finally, since the most common songs (proposed to this specific target of users during music lessons at school) expand at least over 3 octaves and considering that the generalization process will succeed on an electronic piano keyboard composed of three octaves, the size of the virtual piano keyboard has to be carefully analyzed. Specifically, we decided to test three configurations: 1-octave, 1-extended-octave (16 notes centered on the central Do scale), and a 3-octave virtual piano keyboard that generalizes the piano keyboard adopted at school (Fig. 4).

The Note discrimination module (ND), 1-extended-octave version

Full size image

However, when expanding the numbers of notes on the virtual piano keyboard, their size is reduced accordingly. Considering motricity difficulties that can be present in users with low-functioning autism, it becomes important to evaluate the usability of the three-octave virtual piano keyboard on the tablet device of about 10.1″ screen size (Fig. 5).

The Note discrimination module (ND), 3-octave version

Full size image

3.1 Customizing the App

Augmentative elements adapt forms and content of the learning program to the individual’s characteristics and to the requirements of the music program. The user interface adds these augmentative elements (acting as visual prompts) as default at the beginning but they can be progressively faded (by the music teacher) when the student’s learning progress reaches a high percentage of success.

The Suoniamo app enables the interface’s customization, to better adapt to the student’s needs, in several features:

Customizing the task flow

• Number of repetitions of each learning module

• Number of proposals of the same trial after an error (attempts)

• Including or not any accidental notes in the learning path

Customizing the piano keyboard

• Anglo-Saxon or Italian musical notation

• Sharp or flat piano keyboard notation

• Number of active octaves (1, 2, 3)

• Colored or black & white piano keyboard

• Piano keys labelled or not

The current form of the Suoniamo prototype is the result of literature analysis together with iterative evaluation of mid-term prototypes conducted during co-design sessions involving professionals experienced in teaching people with autism. Moreover, as previously mentioned, we performed an early pilot study to understand any major usability problems, as well as the app’s pleasantness and the degree of acceptance by the children. Thanks to results of the previous test, the prototypes have been updated to reflect changes suggested by observed difficulties and by participant preferences.

The objective of the current test is twofold:

1. (i)

   To evaluate the improvements applied to the augmentative elements of each new user interface: colors, notation

2. (ii)

   To evaluate potential usability problems arising from the introduction of a simulated three-octave piano keyboard and its variant (1 octave extended) in order to test the feasibility of the future introduction of the additional module, supporting the execution of simple music scores in driven modality. Here we did not discuss the design and implementation of the execution module, which is currently in progress.

For all these reasons, we defined a protocol for the test of the three different versions of the piano keyboard as described in the following.

Who and Where

The test was performed with four adolescents with diagnosed autism, P1, P2, P3 and P4, described in Table 1.

The test was carried out in a natural setting, reassuring for the children in a comfortable and familiar place. We asked participant caregivers to indicate the most comfortable place. Two students performed the test in a lab where they carry out afternoon study sessions twice a week, while two students carried out the test at home. Before starting the test session, a caregiver introduced the researchers and explained the test to each participant. The caregiver was also present during the test, only as an observer, to make the participant more comfortable. The caregiver as well as the researchers verbally reinforced the student in the case of a task performed successfully (also by clapping hands). The user had previously chosen their preferred reinforcement (a piece of music to listen to, a game to play, a piece of candy). After performing each or all tasks of the test (depending on the user’s attention), he/she was reinforced with their chosen reward, and if they needed, they could relax by doing something else for a while.

What

All participants had to interact with two learning modules: Note Discrimination in three different configuration settings (i.e., 1 octave, 1 extended octave and 3 octaves) and the Note Duration (performed on the 1-octave setting). During the test, we only collected observational data without interfering. We proposed the following tasks:

1. (a)

   T1, Task_1: Note Discrimination module, 1 octave

2. (b)

   T2, Task_2: Note Discrimination module, 1 octave extended

3. (c)

   T3, Task_3: Note Discrimination module, 3 octaves

4. (d)

   T4, Task_4: Note Duration module, 1 octave only for notes of 2/4 value.

The initial order was shifted by one while progressing with the test (i.e., the first user started performing T1, the second started performing T2, and so on), in order to minimize bias due to the effect of familiarization with the exercise and the interaction required. We wanted to investigate the usability of three different configurations of the Suoniamo app user interfaces for the piano keyboard. To this aim, we have collected both objective and subjective data during each task execution related to:

T1, T2 and T3:

• Number of errors (touching a target different from the required note (E1))

• Number of additional wrong interactions (attempt to touch a non-interactive UI element (E2))

• Preference on the different configuration settings (i.e., 1 octave, 1 extended octave and 3 octaves), collected directly by users through Smiley-meter rating scale (Fig. 6).

  

  The Smiley-meter rating scale used for the subjective evaluation

  Full size image

T4:

• Number of errors touching a target different from the required note (E1)

• Number of wrong interactions touching a non-interactive UI element (E2)

• Number of errors touching the target note for a duration different from the one requested (E3)

• Pleasantness of the UI (1 octave version) collected directly by users through Smiley-meter rating scale (Fig. 6).

Each task is composed of several sub-tasks, each related to a different note.

How – Materials

• Android Tablet with Wi-Fi connection, screen size 10.1″; all participants carried out the test on the same tablet

• Sheet of paper to collect observational data by the mediator (Errors E1, E2 and E3) and other relevant observations.

• A Smiley-meter rating scale of 5 items (I like it very much (5), I like it (4), neutral (3), I do not like it (2), I do not like it at all (1)) to collect subjective data on preferences from the users. Participants with medium-functioning autism were immediately able to use the scale while the users with autism of severity level 3 were supported by the tutor with additional vocal cues.

Results of the test are shown in Tables 2 and 3.

Regarding observational data, tasks related to the Note Discrimination tasks (T1, T2 and T3) were performed correctly and easily by two of the four users, those who have a lower degree of autism severity. User P2 was confused by the diesis notes, difficult to detect without visual cues for the discrimination.

User P4, a low-functioning girl, was very excited by the presence of the researcher, preferring to interact with her instead of with the app. When it was possible to make her concentrate on the tasks, she interacted very quickly, anticipating the request of the next subtask (e.g., “play F”) as soon as the previous was completed, thus making errors and becoming confused about the lack of the visual positive feedback. For this reason, she touched the emoticon in order to make it smile. We need to implement a new version of the UIs that minimize the loading time of subtasks, making the new one available as soon as the previous one has been completed.

Another problem was again related to the diesis notes. To resolve this issue, we plan to use additional visual cues to help the user, such as writing the note with the two colors of the keys surrounding it, e.g. C# with the C red and the # orange.

Task T4 resulted more complex, as expected, but there were some errors, probably due to a weakness of the UI and not due to a real user fault: users play the note for the required duration and the neutral emoticon changes into a smile, but if the user releases their finger 1/2 s after, the system considers it an error. For users P3 and P4, this feedback was confusing (they said, “The smile says I’m doing well, but the same subtask is proposed instead of the next”) so they evaluated the UI in T4 with the neutral rate. We need to design more tolerant interaction mechanisms to avoid frustration and misunderstanding in users.

Regarding subjective evaluation, all the users found the user interfaces quite pleasant. Considering the ND module, the simple and clear UI with the piano keyboard in 1 octave was the most highly rated, followed by the one with 3 octaves that a user (with severity level 2) liked because it was the most challenging.

The NV module suffers from the previously described issues and it was the least liked.

Music is an important therapeutic approach for users with autism. This study evaluates the usability of virtual piano keyboards for adolescents with different degrees of autism. As confirmed by literature, a high personalization of the user interface is crucial to match the needs of different subjects, so we have designed different user interface settings. This pilot test with only four users will allow us to refine the app, further simplifying both the interfaces and interaction mechanisms. The aim is to offer an accessible and effective app customizable with visual prompts that can be faded over time to help people with autism play music.