Developing Emotion-Aware, Advanced Learning Technologies: A Taxonomy of Approaches and Features

This paper addresses the relationship between student affect and learning and advances a theoretically informed taxonomy that can be used to improve instructional practices and create emotionally supportive learning environments. More specifically, we examine the design of advanced learning technologies (ALTs) and outline approaches and features that can foster emotions that are critical to effective learning experiences. This is a crucial topic because of the potential of emotions to foster or hinder learning. Furthermore, this paper provides an organizing structure to the proliferation of educational technologies and research on the interrelationships between technology, emotions, learning, and individual differences. The organizing structure of this taxonomy is intended to ease the extraordinary challenge (especially for novices to the field) of forging a conceptual map of the aforementioned elements, especially with the design of ALTs.

ALT is a broad term that includes a variety of sophisticated learning environments such as intelligent tutoring systems (AutoTutor: Graesser & D’Mello, 2013; Wayang Outpost: Woolf et al., 2009), multi-agent systems (MetaTutor: Azevedo et al. 2013; Betty’s Brain: Segedy et al. 2013), serious games (Crystal Island: Sabourin and Lester 2014; Newton’s Playground: Shute et al. 2013; MOCAS: Chalfoun and Frasson 2009), mobile-augmented reality (MetaGuide and the MTL Urban Museum; Harley et al. 2016a) and simulations or virtual worlds (BioWorld; Lajoie et al. 2013; Jarrell et al. 2016). Although the above labels for different types of learning environments are helpful for identifying specific features, they are not mutually exclusive. For example, serious games can include multiple agents and intelligent tutoring systems can have different levels of adaptivity. Broadly speaking, ALTs can be differentiated from other educational technologies such as educational websites, Massive Open Online Courses (MOOCs), and courseware (e.g., Blackboard system) as well as hardware (SMART boards) because of their status as emerging technologies that are not typically available in classrooms (Smaldino et al. 2015). Moreover, ALTs have a greater potential to provide learner-adaptive and immersive experiences and have the capacity to function more autonomously from teachers than more typical classroom technologies.

When ALTs include features designed to foster adaptive emotions, emotions that have been empirically linked to better academic achievement such as enjoyment, pride, and curiosity (Harley and Azevedo 2014; Harley et al. 2016b) and/or motivation, they are referred to as affect-aware systems (D’Mello and Graesser 2015). Whereas adaptive emotions are most often positively-valenced, pleasant states such as the above, this is not always the case (see Framing Emotions section below). Nonetheless, the present article uses the term “positive” emotions to describe the emotional states that affect-aware systems aim to support, as research examining the benefits of negatively-valenced emotions such as confusion is nascent and inconsistent in comparison (Pekrun and Perry 2014).

At present, a growing body of work on intelligent tutoring systems, affective computing, and artificial intelligence (A.I.) in education is exploring creative, technology-driven approaches to enhance learners’ experience of adaptive emotions while interacting with ALTs. This work has ranged from applications of psychological theories to system-administered interventions and design features (Calvo et al. 2015; Sottilare et al. 2013;). Despite this proliferation of related research and proposed dichotomies for scaffolding thinking about how these approaches can be used to support learning (D’Mello and Graesser 2015; Shute, 1992), there has been no published work to date that captures this topic’s breadth. Moreover, previous conceptual work has had little in the way of theoretical guidance from educational psychology.

The taxonomy presented in this article takes up this grand challenge. Specifically, we outline a proposed taxonomy that at its highest level of abstraction builds on previous dichotomies and expands the categorization of approaches to drawing on ALT features that can positively affect learners’ emotions. In addition to labeling, describing, and exemplifying high-level approaches that support learner affect, the taxonomy highlights the different sources of information that inform each of the three high-level affect-aware system approaches. The final layer of the taxonomy outlines four broad adaptable features that can be adapted using these approaches, as well as the information that drives each of them. A selective, example-driven approach is used to describe the above elements of affect-aware systems and highlight their impressive variety, as illustrated by the taxonomical figures and discussion of thereof.

It is important to note that the taxonomy extends prior conceptual work that links learning and achievement with emotion. In particular, we refer to the control-value theory of achievement emotions, along with a selective review of the literature, to provide illustrative examples of how the proposed taxonomical elements can influence the creation of emotion-aware systems. As such, this paper is limited to the theories that inform the taxonomy and does not offer a comprehensive review of all empirical literature that overlaps emotion, learning, and AI, which is beyond the scope of this article.

In sum, the taxonomy aims to further thought, discussion, design, and evaluation of the myriad ways that support for students’ emotions can be built into ALTs to enhance learning. The taxonomy also has the potential to inform a broader range of educational technologies in the selection and use of appropriate strategies when different or multiple (including competing) pieces of learner information are available. Accordingly, the taxonomy serves as a resource for educational designers and researchers of learning technologies interested in the varied approaches and features of ALTs for making informed design recommendations. The taxonomy also connects design recommendations with educational theory, in particular, with the control-value theory of achievement emotions.

Framing Emotions with the Control-Value Theory of Achievement Emotions

The control-value theory of achievement emotions (Pekrun 2006, 2011; Pekrun and Perry 2014) was selected to frame our taxonomy’s theoretical assumptions for several reasons: (1) its appropriateness for understanding emotion in educational contexts; (2) its inclusion and hierarchical organization of related psychological phenomena (e.g., motivation, self-regulated learning); and (3) its description of mechanisms that mediate the experience and regulation of emotions (i.e., control and value appraisals). Although other theoretical frameworks exist to further scientific understanding of emotion and how emotion is regulated (e.g., Gross 2015; Gratch and Marsella 2015), the control-value theory is the only theory that situates the elicitation of emotion in an education context and has strong empirical support in this setting. The taxonomy proposed in this paper presents a thorough, high-level categorization of types of emotion-aware approaches and features that presently exist in systems designed for teaching and training thus warranting education-oriented, context-specific theoretical guidelines. Having said this, many of the approaches and features in this taxonomy may be pertinent to the broader affective computing field (Calvo et al. 2015),

Defining Emotion

The terms affect and emotion are often used interchangeably and to refer to different but related psychological processes. In this paper, we define emotion based on Pekrun’s (2006; 2011; Pekrun and Perry 2014) control-value (C-V) theory in which achievement-related emotions are outlined as multi-faceted phenomena consisting of coordinated psychological processes (affective, cognitive, psychological, motivational, and expressive components). As such, we define emotions as responses to situations (e.g., academic achievement situations, such as a test or studying with an ALT) that are perceived as relevant to an individual’s learning goals. We will henceforth refer to affect-aware environments as emotion-aware environments in keeping with our core theoretical perspective that motivation and related psychological processes are components of emotions (as opposed to other hierarchical classifications of emotion and emotion-related processes; e.g., Gross 2010). Relatedly, moods constitute distinctive affective states from emotions due to their tendency to not be directly tied to a specific aspect or objective foci of an academic context (Pekrun 2011; Pekrun and Linnenbrink-Garcia 2014a).

Valence, Arousal, and Objective Focus: Dimensional Components of the C-V Theory

There are three components (i.e., dimensions) to the control-value theory of achievement emotions. The first two, valence and activation (i.e., arousal), are common across most theories and definitions of emotion including the circumplex model of emotion (Russell et al. 1989). Valence refers to the pleasantness (e.g., enjoyment; positive valence) or unpleasantness (e.g., anxiety; negative valence), while activation corresponds to the degree of physiological activation (i.e., arousal) an emotion elicits when triggered. Joy and anxiety are examples of activating (i.e., excitatory) emotions, whereas boredom and relief are examples of deactivating emotions. These examples illustrate a two-dimensional grid (with valence and activation as axes) that different emotions can be situated upon—neutral being situated in the center (i.e., circumplex model; Russell et al. 1989). The C-V theory adds a third dimension to the valence-activation grid with a focus on the temporal nature of an activity: an object focus. Object focus differentiates between a learner or instructor’s attention to an achievement outcome that has yet to happen and one that has already taken place. The former is referred to as prospective outcome object focus, such as experiencing anxiety about an upcoming test. The latter is referred to as retrospective outcome object focus, such as experiencing shame over a poor exam score.

The C-V theory argues that object focus is particularly important in experiencing flow, where attention is focused on an achievement activity rather than outcome (Csikszentmihalyi 2000; Pekrun 2006; Pekrun and Perry 2014). The importance of one’s attention to an outcome or activity is echoed in the process model of emotion regulation (Gross 2015), which argues that an emotion will not be elicited if one fails to attend to an emotion-arousing stimuli. The C-V theory further differentiates between emotions arising from attention to achievement outcomes and from attention to an academic activity itself, such as enjoyment from listening to and participating in an interesting lecture.

Moreover, learners can have habitual, re-occurring emotions that are experienced in a particular achievement context, such as anxiety during tests or boredom while studying algebra (Pekrun 2006, 2011), in addition to the aforementioned state emotions; emotions elicited in response to a specific academic activity or outcome. Research has found that these emotional predispositions influence learning, as well as state emotions (Harley et al. 2016b; Pekrun et al., 2010).

Appraisals of Control and Value: Proximal Antecedents of Emotion

A central component of the control-value theory of achievement emotions is the mediating role that appraisals play in the generation of emotions. Research based on the control-value theory has consistently demonstrated the influence of different appraisal mechanisms on the elicitation of emotions experienced by students in academic achievement situations (e.g., test taking, studying, lectures; for a review see Pekrun and Perry 2014). Appraisals of control and value are recognized in the C-V theory as the most important appraisals in predicting subsequent activity-related emotion, in part, because these constructs integrate appraisal dimensions from other theories.

Pekrun (2006; 2011; Pekrun and Perry 2014) defines subjective control as one’s perceived ability to effectively manage achievement activities and their outcomes. More broadly, subjective control is outlined as one’s beliefs concerning their causal influence (agency appraisal) over their actions and outcomes (controllability), including the subjective likelihood of obtaining said outcome (probability). Pekrun further defines the term subjective value as the perceived importance of an activity and its outcome(s) to oneself (goal relevance) and, more broadly, as the perception that an action or outcome is positive or negative in nature (goal congruence—event supports or hinders goal attainment). The C-V theory also differentiates intrinsic value from extrinsic value, similar to other theories (Bandura 1997; Heckhausen 1991). An example of intrinsic motivation would be an appraisal of physics as important because it is personally interesting. In this case, the value stems from the activity itself. An example of the latter would be an appraisal of physics as important because of its instrumental role in admissions for a general bachelor of science program. In this case the value stems from something external to the activity. These two types of appraisal are well represented in previous work that has examined different approaches to fostering positive emotions in learners while interacting with ALTs (see below).

In summary, learners’ appraisals of an academic achievement activity or outcome will influence (mediate) the expression of an emotion (its valence and arousal) that, in turn, influence learners’ achievement as well as their cognitive resources, motivation, ability to process information, and self-regulation (Pekrun and Perry 2014; Pekrun et al., 2010).

Academic Environments and Regulatory Opportunities

An additional proposition of the C-V theory is the role of the environment in influencing appraisals and emotions through (for example) the provision and types of feedback, the nature of consequences, and the motivational quality of the environment (see Pekrun and Perry 2014; Pekrun 2006 for details). Of relevance to this taxonomy, the regulatory opportunities change as one proceeds from considering the environment → achievement → appraisals → emotions, and in a manner that is complementary with that of Gross’ (2015) process model of emotion regulation. Specifically, regulation at the environmental level concerns the situation, in other words, the design of the environment and its tasks; one must change the environment (e.g., increase autonomy; Tsai et al. 2008) if one is to positively influence the emotions that will be elicited by those in it (similar to Gross’ (2015) situation selection phase). Regulation at the achievement level relates to seeking out training to address any shortcomings, such as attending a tutorial or seeking out a tutor or tutorial software to help clarify a difficult concept. Competence-oriented (achievement level) regulation is best understood in Gross’ (2015) model as an example of situation modification (J. J. Gross and R. Pekrun, personal communication, April 23, 2016). The next level, appraisal, concerns changing the way a learner is thinking about an outcome or activity, such as by emphasizing a boring task’s value (similar to Gross (2015) cognitive change phase). When it comes to the experience of the emotion itself, the learner must steel themselves against negative outcomes or unpleasant activities to sustain a positive mood (see Gross (2015) suppression strategies). Forcing oneself to smile in the face of disappointing news is an example of suppressing an emotion (e.g., sadness).

While the C-V theory outlines emotions, their antecedents (categorized as related to the environment or appraisals), and outcomes, the assumption of reciprocal causation (Pekrun 2006, 2011; Pekrun and Perry 2014) further renders the theory non-linear in nature. Specifically, emotions can reciprocally influence (i.e., change) cognitive appraisals or one’s academic environment. For example, negative emotions, such as hopelessness, can undermine appraisals of control. Moreover, achievement outcomes (i.e., success/failure) can reciprocally influence emotions and appraisals by influencing self-concepts of ability (for example), where success enhances appraisals of ability and failure diminishes it (Pekrun et al. 2014).

Other Emotional States

It is often difficult to divorce one’s thoughts and feelings in academic and non-academic situations alike from one’s performance (achievement). How did Sally manage to come up with such a compelling argument in class debate? What piece of the puzzle are you missing to solve the homework problem? Yet, academic and non-academic situations alike are more than one-dimensional when it comes to the way we think and feel about events. The following section briefly overviews three other types of emotions that can occur in academic settings and overlap with achievement-related emotions.

Epistemic emotions are elicited from a learner attending to the process of cognitively generating knowledge, such as a learner evaluating whether incoming information aligns with their existing knowledge structure and/or whether inconsistencies exist between sources of knowledge (D’Mello et al. 2014a). Unlike achievement emotions, epistemic emotions are more likely to follow a sequence, such as surprise from a discovered inconsistency between one’s knowledge framework and new information, and confusion if the discrepancy is not immediately resolved. While confusion is typically regarded as a negative, activating emotion in an achievement context and hypothesized to be associated with negative learning outcomes, it can also lead to deeper and better learning (Craig et al. 2004; D’Mello et al. 2014b). Confusion can also lead to better learning when it (1) serves as a catalyst for effortful cognitive activities such as problem solving, evaluation and revision of existing mental models to overcome the impasse and (2) is accompanied by situational interest and curiosity if sufficiently challenging. On the other hand, confusion can become unproductive if the impasse is too great (eliciting frustration) or threatens existing cognitive schemas (D’Mello et al. 2012; 2014; Muis et al. 2015). While the object focus of epistemic emotions (cognitive aspects of a learning activity) is somewhat distinct from those of achievement emotions (the achievement aspects of a learning activity), the C-V theory contributes to our understanding of achievement emotions by underlining the importance of proximal antecedents such as appraisals of control and value (Muis et al. 2015).

Social emotions represent another sub category of emotions where the object focus is others, including human or virtual others (Harley et al. 2016b). Empathy, envy, contempt, admiration and other social emotions arise from interactions with others and shared goals, content, and learning outcomes, including achievement outcomes (Pekrun and Linnenbrink-Garcia 2014b). Content emotions, on the other hand, arise from interacting with specific content, such as learning that Pluto was re-classified as a dwarf planet (Pekrun and Linnenbrink-Garcia 2014a; Sinatra et al. 2014). Ultimately, it is helpful to understand that emotions can arise from appraisals directed toward aspects of academic contexts other than achievement. Nevertheless, achievement remains intertwined with, and is often difficult to disambiguate from, epistemic, social, and content-specific academic emotions (Pekrun and Perry 2014). These additional types of emotions are important to consider in addition to achievement emotions as they illustrate additional opportunities for, and sources of, emotions that can arise and must be regulated in relation to the academic environment as well as , achievement-oriented modifications to the environment, appraisals, and the experience of an emotion (Pekrun 2006; 2011; Pekrun and Perry 2014).

Previous Work and the Proposed Taxonomy

Researchers have experimented with a number of approaches to help students regulate their emotions while interacting with ALTs such as empathetic messages (McQuiggan and Lester 2009), attention-eliciting prompts (D’Mello et al. 2012), feedback emphasizing the malleability (e.g., control) of intelligence through hard work and persistence (Arroyo et al. 2013), and the use of narrative to foster affective engagement (Rowe et al. 2011). Researchers have also addressed the importance of emotion regulation in ALTs, describing them as proactive or reactive systems (D’Mello and Graesser 2015) building off of Shute’s (1993) distinction between macro and micro adaptation in an emotion-specific direction. Other emotion-aware ALT strategy distinctions have included value-, context-, and control-oriented strategies (Arroyo et al. 2014), as well as experimentation with emotion regulation strategies outlined in Gross’ (2013, 2015) emotion regulation model (D’Mello et al. 2014b). Conceptual work by du Boulay (2010; 2011) has also proposed a hierarchy of intelligent tutoring systems with respect to their use of metacognitive, motivational, and/or affective data to foster positive emotions and motivation and further prescribe motivational interventions in response to discrete emotional states (e.g., boredom, frustration, and anxiety).

The proposed taxonomy in this paper advances the state of the art by drawing upon existing, high-level descriptive terms (rather than adding to the proliferation of terminology), and contributing sub-classification labels that pull together complex and disparate features from the learning sciences, cognitive and social psychology, as well as affective computing and computer science. The extended set of emotion-aware system labels are best described as approaches to drawing on ALT features that have the capacity to positively affect learners’ emotions while interacting with them. Different approaches draw upon different information to determine which ALT features should be adapted. Examples are provided throughout the paper to illustrate the relationship between emotion-aware approaches—which need not be used in a mutually exclusive manner—the data they draw upon, and how this influences the use and selection of adaptable ALT features. Collectively, the integrated emotion-aware taxonomy should serve as a useful tool for developers and designers to identify ways in which their systems can foster positive emotions during interactions with learners. It also stands to provide readers of emotion-aware system literature with a means to better organize, understand, and evaluate the empirical contributions of studies and the myriad approaches available for leveraging positive emotions in learner-ALT interactions, including identifying future research directions. Given that we are advancing a taxonomy, our main objectives are to describe, and help readers think about available approaches to fostering learning through the inclusion of design features that will help learners experience positive emotions. Accordingly, the taxonomy provides guidance rather than prescribing specific recommendations for the myriad of contexts and systems out there—a goal better suited to a book than a single article.

D’Mello and Graesser (2015) suggest that emotion-aware systems can be effectively differentiated into proactive as opposed to reactive approaches to foster positive emotions during learner-ALT interactions. Proactive system approaches induce or impede emotional states, whereas reactive system approaches are those that respond to states in real time (typically negative states). The proposed integrated emotion-aware taxonomy (summarized in Fig. 1) elaborates upon these two types of approaches by (1) extending their labels, explanations and examples, (2) mapping their different information sources, (3) illustrating how different sources of data influence the use and selection of four general types of adaptable ALT features (user experience, learning material, assessment, and direct system-delivered prompts), (4) advancing a theoretical justification for examining the taxonomical elements as well as (5) providing a theoretically-guided example of their collective use.

Summary of the integrated emotion-aware taxonomy

Full size image

Figure 1 presents the integrated emotion-aware taxonomy in summary form. Specifically, Fig. 1 reveals that we break down proactive approaches into user adaptive and non-user adaptive approaches which are informed by different information: data about individual students (user models) vs. theoretical models and empirical results. In contrast, reactive features are consistently informed by dynamic user models that provide ALTs with guidance regarding how they should adapt relevant features, which we have summarized as user experience, learning material, assessments, and direct, system-delivered prompts. We argue that what makes approaches user-adaptive, non user-adaptive, or reactive is how, and when feature adaptations are executed (i.e., implemented). Accordingly, we begin by providing an overview of what these approaches are, including what information influences them to adapt or be adapted to users the way they are before summarizing the ways that theories, results, and user models can create strategically-designed, emotion-aware ALTs.

Proactive Approaches

Proactive approaches can be classified as either user-adaptive or non user-adaptive, where adaptive refers to whether the ALT uses information it has collected about the user to make any changes to its interaction with the learner.

User-Adaptive Approaches and User Models

What makes these approaches user-adaptive is that they are customized to individual learners based on the information collected about them. Specifically, adaptive proactive approaches involve making changes to features of the ALT based on a user model that is informed by information about the student such as gender, culture, personality traits, or prior knowledge that are associated with variations in the frequency and intensity of emotion as well as varying dispositions toward prompts designed to support adaptive emotions and learning (D’Mello et al. 2010; Frenzel et al. 2007; Goetz et al. 2013; Rosiek 2003).

User models, in this case, are generated from data collected before, rather than during (or after), the learning session begins with the objective of predicting how a specific student is likely to respond emotionally to the ALT. More specifically, this data is used to determine a student profile by identifying unique student characteristics or constellations thereof. More specifically, individual differences can include demographic information (e.g., gender, culture), psychological traits (e.g., personality traits, trait emotions, motivational goal orientations; see Pekrun 2006; Pekrun & Linnenbrink-Garcia, 2012 for review of terms), and prior knowledge of relevant content or skills that comprise the learning objectives of an ALT. As such, adaptive proactive approaches, and the user models they rely upon, require a means of scoring and transmitting relevant data to the ALT. Additionally, the ALT must be able to process the data and effectively implement changes in the user experience, learning material, assessments, or the direct system-delivered prompt features before the session begins.

The complexity of the user model depends on the type and scope of the data collected and analyzed about the student, with the adaptations the ALT makes based on this information having been pre-specified in the system architecture. For example, if one wants an ALT to adapt message content (text boxes or text-to-speech) to a learners’ personality type, different instructional messages must be designed to correspond to each relevant personality type with the system being able to recognize and retrieve the appropriate decision tree in which these messages are mapped out (Harley et al. 2016b). Decision trees are sequential and recursive maps of learner interactions based on different potential learner behaviors.

Recent empirical research indicates that ALTs could be readily adapted to provide optimized content based on learners’ individual differences. For example, Arroyo and colleagues (2013) found that female learners had higher learning gains when they interacted with a female Pedagogical Agent (PA). Learners’ prior knowledge was also found to influence the number of metacognitive strategies they used while interacting with an ALT promoting self-regulation (Taub et al. 2014). Research by Harley and colleagues (2016b) further shows that personality and trait emotions influence students’ emotions toward specific pedagogical agents. Classroom research has also shown that framing content in culturally familiar terms can increase participation and engagement amongst cultural minorities in classrooms (Rosiek 2003). These examples clearly illustrate the potential for user information to be used by ALTs to adapt content to learners even before a learning session has begun. For example, by (1) matching pedagogical agent gender to learner gender, or allowing learners to choose agent gender, (2) increasing metacognitive prompts for low prior knowledge students, and (3) adapting the instructional strategies of pedagogical agents to learners’ psychological characteristics, it is anticipated that learners’ incidence of positive emotions experienced during interactions with ALTs could be improved. Figure 2 summarizes and provides examples of the information user adaptive, non user-adaptive, and reactive approaches rely upon to foster positive emotions.

Information user adaptive, non-user adaptive, and reactive approaches rely upon to foster positive emotions

Full size image

Non-User Adaptive Approaches and Drawing on Theoretically and Empirically-Driven Models

Not all proactive approaches require adaptation to individual learners in order to effectively foster positive emotions. Non user-adaptive approaches are strategically built into the design of an ALT with this goal in mind. These approaches typically focus on eliciting learners’ positive emotions through targeting psychological constructs such as autonomy (i.e., choice), which is related to appraisals of control and thus a proximal antecedent of emotion. Engagement is another psychological state that is often targeted by ALTs, especially with regard to proactive approaches. It is best viewed as a metaconstruct consisting of several components: cognitive, behavioral, motivational, cognitive-behavioral, and social-behavioral (Fredricks et al. 2004; Pekrun & Linnenbrink-Garcia, 2012; see also Azevedo 2015). What binds these different components together is an active, energetic, and approach-oriented involvement with academic tasks which is how engagement is defined for the purposes of the proposed taxonomy (Pekrun & Linnenbrink-Garcia, 2012). This engaged involvement can, however, be expressed and experienced as flow when task-directed attention and sufficient working memory resources are available (Csikszentmihalyi 2000; Pekrun & Linnenbrink-Garcia, 2012).

Non-user adaptive approaches aim to exploit adaptable ALT features such as the use of game-like elements (e.g., narrative, gamification tactics; point systems) and multimedia design principles known to foster positive user experience by enhancing engagement (Poitras et al. in press; Mayer 2015; Shute and Ke 2012; Virk et al. 2015). Non user-adaptive features are most effective when they work for a more general population, as opposed to specific (e.g., neurotic, middle grade males) group of learners—otherwise user-adaptive approaches may be a better design choice.

A recent selective review by Harley and Azevedo (2014) revealed that ALTs with pedagogical agents that provided students effective game-like features (e.g., narrative) as well as choice elicited more positive emotions from students than those that did not; a finding directly in line with the C-V theory of achievement emotions (Pekrun 2006, 2011). Autonomy can be fostered, like engagement, through game-like features in many ways in ALTs. Opportunities range from more constrained choices, such as providing opportunities for learners to explore ALT content and features through hypermedia (e.g., table of contents), to more elaborate, open-ended decisions such as exploring rich 3D worlds, and customizing learner avatars or pedagogical agents’ appearance and dress (Poitras et al. in press; Shute and Ke 2012). One example of an ALT that effectively deploys non-adaptive, proactive features to elicit positive emotions is Crystal Island (Sabourin and Lester 2014; Rowe and Lester 2015; Rowe et al. 2011). This ALT borrows many features from commercial video games (e.g., interface, navigational controls, rewards, character interaction, professional graphics) and immerses students in an interactive 3D world in which they are free to explore and gather clues to determine the nature of a mysterious illness that has stricken the research team.

Reactive Approaches and Dynamic User Models

While proactive features only take advantage of what is known about the learner, specifically or generally, reactive features make use of information collected dynamically in-session about the learner. More specifically, reactive features adapt to learners’ needs as the learning session progresses based on an evolving and dynamically updated user model. Similar to proactive features, however, the type of information that can be drawn upon to customize learner-system interactions is extensive.

Data collected during the session that informs dynamic user models and drives the use and selection of adaptable ALT features include information collected on an ongoing basis about students’ psychological states and learning trajectories. Psychological state information includes how learners are currently feeling at the moment, referred to as learners’ concurrent state activity emotions (Pekrun 2006; Pekrun and Perry 2014). It also includes learners’ object focus directed on current academic activity and their attention to and engagement with an ALT (Pekrun 2006; Pekrun and Perry 2014). Trait information, such as personality type and motivational tendencies, tend to be better aligned with proactive adaptive system features because they tend to be stable and unlikely to change, in contrast to emotions and students’ understanding of learning material that tend to be more variable over time (Alexander 2003; Ekman 1992; Harley et al. 2013; Lajoie 2003).

Collecting data at multiple intervals is therefore critical because of potential changes in learners’ psychological states and learning trajectories. Empirical work has demonstrated that not only do emotions meaningfully change from moment to moment, but that these changes can also reveal patterns that may reflect suboptimal learning such as the deterioration of positive emotions over the course of a learning session or the triggering of persistent negative emotions. Negative emotions can persist as either single states such as boredom, or alternations between different negative states such as boredom and frustration (Baker et al. 2007, 2010; Harley et al. 2013; McQuiggan et al. 2010). As learning happens incrementally rather than all at once, and trajectories toward competence are not necessarily linear, assessments should also be dynamic and ongoing (i.e., formative) rather than cumulative (Lajoie 2003). Lajoie and Lesgold (1992) defined dynamic assessments as moment-to-moment measurements of learning during problem solving that provide opportunities for feedback to be provided within the context of the activity (also see Shute et al. 2016).

Dynamic user models can therefore vary in terms of the information they are built with and are updated as relevant information becomes available. As such, they can be thought of as mapping learning and psychological trajectories for the purpose of affording opportunities and information for interventions if the ALT identifies that the student is going “off course”. Moreover, instructional interventions are more likely to be effective when they are based on the situation and relevant information at hand, rather than more general information about where in the learning session a student might need a word of encouragement or the types of learning content that typically require additional information (Bouchet et al. 2013a; 2016; Bouchet et al. 2013b). Different types of system features that can be adapted in real-time based on dynamic user models are described below and classified according to those providing direct system-delivered prompts, user experience, or assessments and learning material.

Adaptable ALT Features

This section describes and provides additional examples of features of ALT systems that can be adapted to enhance learners’ experience of positive emotions (see Fig. 3 for summary). ALT features do not typically belong exclusively to user adaptive, non-user-adaptive, or reactive feature categories. Rather, it is how and when they are adapted that reflect whether they are user adaptive, non user-adaptive, or reactive in nature.

Adaptable ALT features

Full size image

User Experience

This group of features refers to those related to the subjective experience of the user while they interact with the ALT, such as the degree of engagement and autonomy (i.e., choice) the system supports (see earlier discussion of game-like features and engagement). The choices learners are able to make in ALTs are broad and can include: (1) the order of pages one reads and navigates to in a multi-agent, hypermedia environment (MetaTutor; Trevors et al. 2014); (2) which virtual pedagogical agent one will side with if the two agents disagree (Operation ARIES!; D’Mello et al. 2014a); (3) what content to interact with and investigate in an immersive 3D serious game environment (Crystal Island; Sabourin and Lester 2014); (4) what confirmatory tests to run, if any, to make accurate medical diagnosis in a simulation (BioWorld; Jarrell et al. 2015a; Jarrell et al. 2015b); (5) how to creatively solve a physics problem using a stylus in a 2D serious game (Newton’s Playground; Shute et al. 2013); and (6) how and when to use instructional tools such as a concept map to teach a virtual agent about environmental issues (Betty’s Brain; Segedy et al. 2013) or an annotation tool to evaluate the credibility of a historical document (Poitras and Lajoie 2014). These examples illustrate some of the many different types of choice learners are presented with while interacting with ALTs and their various learner-system interaction parameters. The greater the extent of choice with respect to the number and timing of possible options, as well as possibilities to customize solutions and interactions, the higher the levels of positive and activating emotions (Harley and Azevedo 2014). In other words, learners typically react well to having some freedom to make decisions while interacting with ALTs, though too much choice can be distracting (Harp and Mayer 1998; Lepper 1988).

User experience features such as the affordance of choice can typically be implemented at different points relative to a learners’ interaction with an ALT, though some may be better suited to being adapted to the typical learner or group of learners rather than adapting in-session to individual learners. Two general considerations exist for strategically evaluating how to leverage the potential emotional benefits of user experience (and other adaptable ALT features): cost and value. Developing an ALT is typically an expensive endeavor, and the more features, and more sophisticated those features are, tends to make them more costly in terms of both human and financial resources (programming, visual design, pedagogical content, etc.). The value of implementing user experience features, such as more choice for learners, must therefore be balanced against the cost. Value can be estimated by examining the relationship features have to learning based on theoretical models as well as examining empirical research and reviews that provide evidence for the hypothetical value of implementing various adaptable ALT features (for example see (Virk et al. 2015). Design decisions guided by theory and research are also more likely to maximize the potential value of a feature by modeling their implementation on previous work.

Accordingly, designers of ALTs should first take stock of prior empirical evidence and then identify what user experience features are available and expected to be most effective at fostering positive emotions either directly or as end products of a mediating psychological construct such as autonomy. Multimedia principles of learning (e.g., split-attention, redundancy principles) are an example of design features that have been rigorously researched and empirically demonstrated to foster learning when properly applied (Mayer 2015). As such, they serve as general guidelines for designing multimedia ALTs to maximize learning, often by minimizing cognitive load and side-stepping over-loading learners’ cognitive resources which would retract from engagement given that the amount of information we can hold in our consciousness is limited (Ayres and Kalyuga 2011).

Game-like elements such as narrative are also typically effective for engaging most learners, but can also be costly to develop^{Footnote 1}. Therefore, these features are best implemented (at a high level) generally, for example, without proactive, user adaptive approaches that rely on pre-session user modeling. They may, however, benefit from being (reactively) adapted at finer-grained levels based on learners’ evolving psychological states and learning trajectories, if resources permit. Different narrative paths in a story arc represent an opportunity to adapt an immersive and engaging feature to the user at different points in time (Harley et al. 2015b).

Other examples include making extra educational tools available for students to select if they are needed—or removing them (reactive approach), if they are not being used properly or are being used to game the system (i.e., complete the session quickly without investing effort in learning; Baker et al. 2010). Instances of gaming the system can be identified and addressed based on their nature and timing. For example, the number of potentially contaminated items that can be put under the microscope for examination of contamination in Crystal Island (Sabourin and Lester 2014) is limited to having learners engage in inquiry-oriented learning and consider evidence that supports testing an item rather than simply engaging in trial and error testing. An alternative would be to introduce a requirement based on encountering such a problem (a non user-adaptive approach) to have learners explain to a scientist why they need to use her microscope prior to ordering a battery of tests. Such an approach would fit with the ALTs narrative and more closely approximate a real-life scenario (with the anticipated result of increasing positive emotions), in contrast to relying on the introduction of arbitrary limitations.

Assessments

Assessments can also be adapted to help regulate students’ emotions due to the reciprocal nature of achievement and emotion (Pekrun 2006; Pekrun and Perry 2014). Adapting assessments can be especially critical when students are at risk of experiencing negative emotions like hopelessness due to repeated failure or anxiety about evaluations in general (e.g., trait test anxiety; Pekrun et al., 2002). An example of a non user-adaptive proactive approach would be to make questions easier (or harder) for all students based on prior research (Harley et al. 2014). If multiple items at different levels of difficulty exist for specific sections of content (e.g., a hypermedia page, unit, chapter, etc.), ALT assessments could be tailored to students’ prior knowledge (user adaptive proactive approach). Such an approach could involve providing lower-prior knowledge students with some easier questions in addition to the standard one’s in order to avoid discouraging them but still identify gaps in their knowledge. An ALT with a sophisticated means of assessing learning could go a step further and take a reactive approach characterized by adapting the difficulty and number of questions to reflect students’ learning trajectory and potential differences in their ease of mastery of some vs. other content. This kind of an approach could be done on a quiz-by-quiz or item-by-item basis. In case of an item-by-item basis, assessments might begin with more challenging items that, if answered correctly, could progress the learner through the assessment more quickly than a wrong answer that may provide them with more questions and therefore additional opportunities to evaluate their mastery of the content. It should be noted that most of these examples rely on adaptations to formative assessments (i.e., interim, in-session assessments rather than post-session or summative methods), which are more costly to develop than a single, final post-test. They do, however, provide valuable, additional opportunities to help learners monitor their emerging understanding and mitigate feelings of anxiety that might surround uncertainty regarding their understanding of content. In other words, an increase in the frequency of assessments provides more opportunities to assess one’s evolving understanding that might otherwise only be evaluated once and after time has run out to interact with learning material and address errors or gaps in understanding.

In addition to adapting the difficulty and quantity of assessments (or items therein), ALTs can be reactively adapted to provide additional context (i.e., performance information), like an instructor would, to help point learners in the right direction (e.g., making suggestions about revisiting content that corresponds to questions they got wrong). By providing specific information on learners’ performance, emotions such as frustration and confusion could be mitigated by giving the learner feedback not just on their over-all understanding but what elements they need to revisit, saving them time and effort.

Leighton et al. (2012) also point to the importance of taking affective variables into consideration when students encounter failure during assessments in order to contextualize mistakes as opportunities for identifying gaps in understanding to facilitate subsequent interest, motivation, and performance. Related research by Dweck (2002) also encourages instructors to provide effort-directed praise, which encourages learners to think about intelligence as malleable rather than fixed; a more conducive mindset to face failure in. Although providing prompt, adaptive, emotion-targeted feedback may not always be realistic for instructors, it is possible for emotion-aware systems. The following section on system-directed feedback provides an overview of tutorial messages that these systems can provide to learners in the place of an instructor or human tutor.

Learning Material

Similar to assessments, learning content can be altered with respect to quantity and difficulty, although doing so requires an even greater investment in developing content that may not be used by every learner. Moreover, altering the learning content may make comparisons difficult and undermine learning objectives. Therefore, examining whether learning outcomes and trajectories can be improved by adapting other adaptable ALT features (user experience, assessments, system-delivered prompts) is preferable as well as examining data for outliers and considering demographic issues (e.g., if grade four vs. five students are using an ALT). If adaptions such as providing more learning time, additional instructions, additional learning tools, and / or more information on their performance on assessments have been found to be insufficient to help the learner master the material it may be that learners’ prior knowledge of the content or skill is insufficient to benefit from the material. In other words: The lesson may be outside of their current zone of proximal development (Vygotsky 1987), even with adaptive support at one or more levels from the ALT. Therefore, if more introductory material (intersection of quantity and difficulty in Fig. 2) is available, it may be beneficial to provide the student the opportunity to review and be assessed on the material prior to proceeding to the next module.

Poitras and Lajoie (2014) designed an ALT that provides a highly structured approach to teaching students about historical reasoning and how to interact with the system by having learners progress linearly through several practice modules that prepare them for the primary historical reasoning activity. This approach represents an example of non-adaptive proactive features and suggests that priming modules may be effective in helping learners progress more efficiently through the actual ALT content and perhaps more quickly to more advanced learning modules.

Altering the way the information is presented, such as by adding images to text (carefully attending to redundant information) could also positively improve learners’ learning trajectories by reducing cognitive load through coherence with widely accepted and empirically robust multimedia principles (see Mayer 2015).

Direct System-Delivered Prompts

To date, most of the empirical work on ALTs has examined the utility of messages delivered by the system to foster positive emotional experiences during learning sessions. System-delivered prompts are provided to students through dialogue boxes or speech (using a text-to-speech engine), and often from an animated pedagogical agent. Direct system-delivered prompts can influence students’ emotions by (1) changing the way they are appraising achievement-related information, (2) triggering an emotion through the social expression of one from a virtual or human agent, (3) temporarily re-directing attention away from a source of stress or (4) offering recommendations to enhance learners’ competence, such as procedural clarifications or (5) suggestions for learners to more strategically monitor and regulate their learning. This section provides a brief overview of these types of strategies and their deployment.

The most popular and empirically supported emotional regulation strategies are those associated with cognitive change (i.e., modifying appraisals; Pekrun and Perry 2014), and reappraisal in particular (Butler et al. 2003; Gross 1998; Gross and John 2003; Leroy et al. 2012; McCrae et al. 2012). In an educational context, reappraisal involves construing a potentially emotion-eliciting situation in a manner that will either up- or down^{Footnote 2}-regulate a learner’s emotions (Hall et al. 2006a; Hall et al. 2006b; Hall 2008). For example, a student interacting with an ALT could decide to interpret a poor quiz score as helpful and informative—an indication that it would be useful to re-read the page—rather than suggesting they know too little about the material and should not pursue more advanced studies in this domain. This example illustrates changes in both learners’ appraisals of value (the feedback being useful) and control (having the agency to overcome difficulties with the material) to dampen feelings of disappointment and enhance feelings of pride. Value and control represent the two most critical dimensions of appraisals in an educational context that are involved in construing an emotion-eliciting situation (Pekrun 2006; Pekrun and Perry 2014).

System-delivered prompts leverage the potential benefits of the malleability of learners’ views of academic situations by serving as an external voice that suggests adaptive ways of interpreting information that may not be immediately apparent—a message that is especially important if a negative emotion has been recently elicited (e.g., hopelessness after performance feedback or frustration from not understanding content after viewing hypermedia information for several minutes). System-delivered prompts can also be administered at fixed intervals or before learners begin interacting with the learning content in order to positively influence their appraisal of the learning session with respect to the perceived challenge, failure, and the value of the task; a non-adaptive proactive feature application of direct system-delivered prompts (Chauncey-Strain and D’Mello 2015). Research suggests, however, that such prompts are more effective when administered reactively with recent information about the learners’ emotional state rather than at pre-determined, non user-adaptive or even user adaptive approaches where interventions may be misaligned with current emotional states and hinder rather than improve emotional states (Robison, McQuiggan, & Lester, 2015).

Another type of system-delivered prompt that takes more of a social construction than cognitive appraisal approach to regulating emotions is referred to as parallel empathy (Gross & Barrett, 2011; McGuiggan & Lester, 2007). These types of messages direct learners’ attention to a pedagogical agent who is portrayed as also finding an activity boring or frustrating (Arroyo et al. 2013, 2014; McQuiggan and Lester 2007). The underlying mechanism here is an appeal to the learner to feel a certain way based on another’s emotions, or in this case, the emotions of a virtual peer or tutor. As such, this approach has the potential to affirm learners’ emotional responses as valid, and potentially have these experiences be less upsetting to the learner. These social approaches are particularly novel when a PA’s facial expressions and gestures are deployed. However, the efficacy of these additive features on producing positive emotions is context-dependent and must appear authentic to the learner (Baylor and Kim 2009). Accordingly, such prompts should be reactive rather than proactive.

D’Mello and colleagues (2014) drew parallels between the process model of emotion regulation model by Gross (2013) and strategies employed in ALTs, including attention-related emotion regulation prompts used in GazeTutor. This ALT targeted learners’ attention by asking them to re-direct their gaze to the screen the ALT was on when it detected they were looking away. While this is a good strategy for reminding students to stay on task, and may help enhance their achievement if they are highly distractible we propose a different strategy more in-line with Gross’ concept of attentional deployment: temporarily redirecting their attention elsewhere, potentially off-task (such as a song) to provide a break. Encouraging a learner to focus their attention elsewhere (i.e., off-task) for a short duration (i.e., < 7 minutes) may allow them to return to the task refreshed (Sabourin and Lester 2014) rather than transition from a state of frustration to persistent boredom (Baker et al. 2010). The research literature on mindfulness may be particularly relevant in this regard (Calvo and Peters 2015). While the authors are not aware of any research to-date examining this kind of a reactive and direct-system delivered prompt, ALTs could draw on dynamic information about a learner’s emotional state to identify when they might need to focus elsewhere, such as prolonged states of frustration. Similarly, prompts could be triggered if gaze patterns were observed that are associated with negative emotions such as boredom (Jaques et al. 2014). This type of prompt is most effective with a reactive emotion-aware approach.

Procedural informational prompts are those that provide additional instructional context to learners to help them (1) effectively navigate the ALT, (2) utilize available learning tools, and (3) ensure that they understand their learning objectives. These types of prompts can be thought of in the C-V theory as qualities of the environment that can mediate students’ achievement, and in turn, diminish negative and help foster positive emotions. In the absence of procedural understanding related to any of the above examples, learners’ interactions and opportunities to learn with an ALT will be reduced. This can in turn elicit frustration from prolonged and unresolved confusion about how to use the ALT or sadness or hopelessness from failing to achieve learning goals. While procedural information should be provided at the beginning of the learning session, reminders, especially those adapted reactively to learners straying from objectives or making procedural errors, stand to mitigate feelings of frustration and anxiety and potentially bolster pride through feeling competent interacting with a new learning environment.

Another type of competence-related system-delivered prompts are those that advise learners to engage in more strategic learning behaviors by providing prompts and feedback designed to foster self-regulated learning (SRL). SRL is defined as actively constructing an understanding of a topic or domain such as mathematics (e.g., algebra) by (1) setting sub goals, (2) using learning strategies, (3) monitoring and regulating certain aspects of cognition, behavior, emotions, and motivation, and (4) modifying behavior to achieve one’s goals (Azevedo et al. 2013; Boekaerts et al. 2000 ; Pintrich 2000; Zimmerman and Schunk 2001). For present purposes, we further specify SRL as a concept superordinate to metacognition in-line with Azevedo and colleagues (2013) that incorporates both metacognitive monitoring and metacognitive control, as well as processes related to manipulating contextual conditions and planning for future activities within a learning episode. Ultimately, we hold that SRL is about learners exercising agency by consciously monitoring and intervening in their learning (Azevedo et al. 2013). Examples of system-direct prompts that encourage students to engage in SRL behaviors include prompts to set goals, monitor their progress toward goal completion, and to use strategies such as summarization that afford deeper and more active learning (as opposed to verbatim note-taking). These learning behaviors help students regulate their emotions by enhancing their competence and, in turn, achievement. Moreover, these prompts have the potential to play a role in positively influencing learners’ appraisals of control (Arroyo et al. 2014) and self-efficacy by addressing underlying problems with learning and studying strategies that can otherwise make approaching achievement situation like studying for exams overwhelming.

In general, despite strong evidence that direct, system-delivered prompts, reappraisal messages in particular, work in experimental learning and external learning contexts (Chauncey-Strain and D’Mello 2015; Hall et al. 2006b; Leroy et al. 2012), their effectiveness when deployed in ALTs seems to be mediated by individual differences such as prior knowledge and gender (Arroyo et al. 2013; D’Mello et al. 2010), and has occasionally been found to elicit negative emotions (Robison et al. 2009). Moreover, ALTs often mix different types of system-direct prompts into system architectures under different names such as reactive empathy, which can make empirically evaluating their effectiveness more challenging (McQuiggan and Lester 2007).

Deployment of Direct System-Delivered Prompts

The previous section primarily focused on the content of system-delivered prompts, however, the frequency and timing of these prompts are important considerations (as with assessments). Direct system-delivered prompts can be delivered either based on a reactive approach and dynamic user model that alerts the ALT when an intervention may be needed or based on previously established criteria that may be the same for all users (non user-adaptive, proactive approach) and / or contain variations for different groups of learners (user-adaptive, proactive approach).

Pre-established system-delivered prompts can be administered either before a specific event (event-based prompts) or on a purely time-based schedule (time-based prompts). For example, regardless of the learning task, a time-based text or audio message could be provided encouraging the learner to consider the importance of what they are working on. A pedagogical agent could further express that they were finding the content challenging “as well” but felt confident that they could master it together with a little more effort (regardless of the material being worked on). Time-based prompts could also recommend that learners think about whether the content one was reading is relevant to one’s goals for the learning session. While these prompts are not sensitive to context and have the potential to be disruptive, they can be useful, especially for experimental purposes, for ensuring that different types of messages are delivered (regardless of what happens) and therefore tested. Time-based prompts are also typically less difficult to program into an environment than are learner- or event-specific messages, and therefore may best represent prototypes of increasingly adaptive ALTs. Prompts that are administered before or after events function in a similar fashion, but have the advantage of being less disruptive if administered, for example, between tasks, and can be more targeted (and in doing so stand a better chance of being appropriate). For example, an event-based, system-delivered prompt designed to reduce anxiety may be more effective if administered before a quiz learners know is coming, and may be slightly anxious about, than at a random point during the learning session where they might be curious or frustrated by learning material.

Time and event-related pre-established system-delivered prompts are administered according to when they have been programmed to be. Reactive approaches to system-initiated prompts, on the other hand, have additional parameters to when: whether they should be activated, and if multiple prompts exist, which should be communicated? For example, if direct system-delivered prompts are triggered in response to self-reported emotions (Robison et al. 2009), then they can only occur as often as the emotions are reported (when), and ideally, only if the emotions reported warrant an intervention (whether; e.g., boredom but not enjoyment). If, however, the system is using continuous (online) data that is analyzed and processed in real time, whether can become a more complex question. For example, how often is too often to prompt students? And how long should a student be left in a negative emotional state? Are the answers to these questions different for various negative emotional states (e.g., boredom vs. anger)? The last question is an example of a situation where which prompt comes to be an important question, assuming there are multiple prompts built into the architecture of the ALT to respond to different negative emotional states.

Research by D’Mello et al. (2014) raises interesting questions about the beneficial nature of confusion, typically considered a negative state, given the potential for some learners to achieve deeper learning when left to resolve that confusion themselves. Having multiple prompts available may help optimize reactive systems’ adaptation to learners, especially if different sources of information can be drawn upon to inform decisions. Different sources of information could include data about a learner’s current emotional state from multiple modalities, recent performance, and on-task behavior, amongst others.

Given the aforementioned questions, there is one overarching guideline that can be safely heeded: Do no harm. More specifically, prompts do not need to be delivered if the system classifies a learner as experiencing a positive emotional state. Second, learners might be better off left alone if prompts are poorly calibrated to their present state, either because of an interaction between a particular message and individual difference, the message being poorly designed, or (more likely) because of a failure to accurately detect the learners’ emotional state (Robison et al. 2009). The latter can happen as the result of competing, incongruent emotion information from multiple data channels (Harley et al. 2015c). Prompts designed to mitigate anxiety, for example, may in fact elicit more anxiety if the learner is suddenly made to think that maybe they should be anxious (for a review of meta-emotions see Bartsch et al. 2008). See Calvo and D’Mello (2010), Harley (2015), Mauss and Robinson (2009), and Porayska-Pomsta and colleagues (2013) for a discussion of the state of the art of emotion measurement methods.

Accordingly, detection accuracy, in addition to carefully researched pedagogy, is critical for decisions about when and if to deliver a direct system-delivered prompt as well as what the prompt should say. It is therefore advisable to avoid administering these types of prompts needlessly, such as when neutral states are identified. In these cases, although learning may be maximally bolstered by positive emotional states such as curiosity, it is nonetheless not being hindered by distracting negative emotions such as anxiety or boredom, or by unhelpful information relayed by mis-calibrated messages. These and other considerations are summarized below.

Integrating Proactive and Reactive Features

Up to this point, a broad variety of ALT features have been classified and discussed along with approaches to integrating them into ALTs. The focus of this section is to provide examples of how ALTs can better leverage the strengths and affordances of these different features and approaches to using them, and in doing so, provide emotion-aware environments that are: (1) from the start, designed to foster positive emotions for all learners, and have the capacity for customization based on (2) persistent individual student needs and differences as well as (3) student needs that emerge dynamically during their interaction with the system. After a brief re-review of the approaches and features that designers can use in emotion-aware systems (below), we situate the emotion aware taxonomy within the C-V framework through examples in order to provide additional guidance and context for their integration.

Overview of the Emotion-Aware Taxonomy

Designers of ALTs can achieve the aforementioned objectives by considering each of the approaches in turn as well as the different features that might be used in their execution. The first components of this taxonomy a designer might consider are proactive, non user-adaptive approaches to making the system as emotionally rewarding as possible for students in general, by for example, including narrative elements to foster immersion and enjoyment. Next, proactive user-adaptive approaches can be reviewed and individual differences that stand to be important for the type of ALT or content (e.g., gender for math) can be strategically prioritized in the development of the academic environment and its features. Proactive, user-adaptive approaches rely, however, on data related to individual differences being collected and analyzed/ categorized, typically in self-report form, unless analytics for inferring traits (like personality) from in-system behaviors are available (Cowley and Charles 2016). Finally, decisions about reactive approaches must also be made. This type of approach provides an opportunity to deepen the level of customization of the system to the learner by using a variety of information that can be collected from everything from log files to live feeds from physiological recordings. Across all three of these broad types of approaches, the information that guides them (e.g., profiles, theory/empirical research, or dynamic user models) must be considered as well as opportunities to adapt an ALT’s user experience, assessments, learning material, and direct system-delivered prompts based on the constellation of information and available approaches.

Tying Together the C-V Theory and Emotion-Aware Taxonomy Through Examples

In this section emotion-aware system approaches, the data that inform them, and adaptable ALT features are situated within the control-value theory of achievement emotion’s (Pekrun and Perry 2014) basic propositions related to the antecedents that influence the generation and regulation of emotion. The control-value theory of achievement emotions uses the environment as the starting point for considering distal antecedents of emotion and describes related regulation strategies as situation-oriented to reflect the role that design plays. In the emotion-aware taxonomy the ALT environment is exploited by both adaptive and non-adaptive proactive features (see Fig. 2). Autonomy, an example of a user experience feature, would be categorized in the C-V theory as an environmental factor that stands to influence emotion (Pekrun 2006; Pekrun and Perry 2014). While the level of autonomy that an ALT supports may benefit from being calibrated to individual learners based on their prior experience with related types of software (e.g., games) or prior content knowledge, providing opportunities for choice is a generally effective strategy. The emotion-aware taxonomy would classify calibrating autonomy to specific individuals (e.g., based on their prior knowledge) as a user adaptive proactive approach and the latter as a non user-adaptive approach (see Fig. 2). Providing affordances for users to exert their autonomy in an ALT could take the form of allowing them to explore or navigate though the environment in a non-linear manner.

Competence-oriented emotion regulation strategies aim to support positive emotions through enhancing achievement. In other words, by helping students succeed, competence-oriented emotion regulation strategies elicit positive emotions associated with reaching achievement goals such as pride. An example of this kind of an approach could involve focusing on assessment features of emotion-aware ALTs and changing the difficulty level of a quiz to make it easier over all, which would be classified as a non user-adaptive, proactive approach. Alternatively, novice items could be mixed in with existing challenging items. Struggling students could, in this case, walk away from the quiz with the pride that they got at least some of the answers correct, while recognizing that they have more work to do. This would be classified in the emotion-aware ALT taxonomy as a user-adaptive, proactive approach.

Learners’ cognitive appraisals are an especially central component of the C-V given their role as proximal rather than distal antecedents of emotion. Appraisals can be influenced in a number of ways in emotion-aware ALTs, but the most widely-researched and direct way of influencing appraisals is through system-delivered prompt features (Fig. 3). System-delivered prompts can target appraisals themselves, for example, by highlighting real-world applications of learning content, and in doing so, support learners making higher appraisals of value. High appraisals of value are associated with positive emotions when appraisals of control are medium-to-high (Pekrun 2006). A prompt targeting a learner’s appraisal of task value could be delivered to them using a reactive approach (Fig. 2) if, for example, an update to a students’ dynamic user model revealed that they were (1) bored (psychological state; detected from a sustained drop and plateau in physiological arousal), and/or (2) not deploying effective learning strategies to resolve an impasse in understanding and causing the learner to stall in their learning trajectory. The latter might be detected from log files of the learner clicking repeatedly on a button that triggers the same response, or buttons at random.

Direct system-delivered prompts could also be deployed using a proactive approach (Fig. 2). Delivering prompts designed to help girls overcome tendencies to (inaccurately) rate their competency as lower than boys in mathematics (Goetz et al. 2013) would constitute a user-adaptive proactive approach, where static demographical information about the learner is used tailor the intervention (i.e., boys and girls might receive a different math message in this case). The final type of emotion regulation in Pekrun and Perry’s (2014) four-fold conception of emotion regulation is emotion-oriented regulation (i.e., suppression; Gross 2015). This type of emotion regulation approach involves targeting the emotion itself, and trying to suppress it. Direct-system delivered prompts can be used to request that learners avoid expressing the way they feel or ignore their emotion, although these tend to be ineffective, heighten arousal, and are detrimental to learning and social interactions, the latter having negative implications for social emotions (Butler et al. 2003; Chauncey-Strain and D’Mello 2015; Gross 2015; Gross and Levenson 1993, 1997). As such, we agree with Duckworth et al. (2014) assertion that it is a strategy that is “hardly strategic at all” (pp. 221). We have, accordingly, paid the least amount of attention to it in the examples of approaches and features covered in the taxonomy.

The taxonomy proposed in this manuscript makes a number of contributions to research on emotion-aware systems. At the highest level, it expands upon Graesser and D’Mello’s dichotomization of ALTs as proactive or reactive systems to include user adaptive and non user-adaptive proactive approaches. Distinctions between these three approaches are outlined through a detailed discussion of types of data they are informed by (Fig. 2) and practical and theoretical considerations for implementing them. The operationalization of emotion-aware approaches is also discussed in terms of the strategic selection of key features of ALTs classified as user experience, learning material, assessment, and system-delivered prompts. Examples of each sub feature are provided and discussed in association with different emotion-aware approaches and data considerations. The use of the C-V theory to frame the taxonomy and its assumptions also stands to foster a greater understanding of discrete features and opportunities for their use in fostering emotions that will support learning. Moreover, the proposed taxonomy outlines considerations for the integration of these features in emotion-aware systems using different approaches rather than suggesting that emotion-aware system are proactive or reactive in nature. Finally, the proposed taxonomy provides a means of identifying future directions for developers and researchers, such as the need to take advantage of user adaptive and non user-adaptive proactive, and reactive approaches as well as a broader selection of adaptive ALT features than typically considered in the design and development of emotion-aware ALTs. Indeed, there is no reason (excluding financial resources) that different approaches cannot be combined to collectively leverage features that are (1) generally effective, (2) specific to individual differences, and (3) track participants’ psychological states and learning trajectories and adapt accordingly.

The proposed taxonomy provides an example-driven, structured account of emotion-aware approaches, the data they rely upon, and the ALT features that can be exploited to support positive emotions. Our goal was to represent and visualize the most central considerations behind designing emotion-aware systems. This goal was a different, albeit no less challenging one, than creating a comprehensive review of theories of emotion or emotion-intervention studies (see Quoidbach et al. 2015; Gross and Barret 2011). As such, the taxonomy both benefits and is limited by its breadth. The main limitation is the inability of a journal article to cover all aspects related to the emotion-aware approaches and four types of ALT features that exist. Given that the taxonomy has drawn upon literature related to emotion theory, emotion measurement, affective computing, user experience, different psychological processes and traits, individual differences, and issues related to learning material and assessment, an extension would be needed to expand upon the provided examples. Indeed, future work might elaborate upon some of the examples in Fig. 2 and/or 3 to provide a fuller account of relevant aspects of demographics, psychological traits, psychological states, means of supporting autonomy and engagement in user experience, means of adapting learning and assessments, and different system-delivered types of prompt. The taxonomy identifies the need for future experimental research with emotion-aware ALTs to provide more granular prescriptions and guidelines. At present, the literature is not developed enough to make substantive claims regarding the effectiveness of all techniques. Nor does the current literature support an empirically or theoretically-driven account of when different approaches and features should be used, in which combination, and in which order.

With these limitations and future directions in mind, we would argue that a taxonomy that discusses emotion-aware system design from all of aforementioned perspectives stands to help designers appreciate the scope of the design space they are working in as well as a visual, taxonomical representation for thinking about how to organize and operationalize disparate elements. It should also be noted that although the presented taxonomy is based on research conducted with ALTs, many of the discussed features, especially when designed for use with user adaptive and non user-adaptive approaches, can also be applied to less adaptive educational technologies such as massive open online-courses (MOOCs). Some MOOCS, in fact, already use some of the features mentioned, such as the point system Khan Academy uses to rewards learners with points for progressing through different modules. In this system, a user-selected Pokémon-like avatar (cartoon creature; users can select from a few different types) is shown to benefit from the points by evolving into a different (more advanced-looking) creature; an idea similar to trophy-based gamification systems. The presented taxonomy therefore stands to provide additional directions that MOOCS might attempt based on the flexibility of the system, availability of customizable content, and technological affordances to provide prompts (etc.).

The increasing popularity of MOOCs and other educational technologies, such as mobile learning environments (Harley et al. 2016a), stand to gain significantly from integrating the interdisciplinary approaches that have been under development and evaluation in ALTs for years. This taxonomy thus provides an opportunity to organize and share some of these insights with both the intelligent tutoring system and artificial intelligence in education community as well as the broader educational technology community.