Bridging Conceptual Models and Architectural Interchange for Adaptive Instructional Systems

The Adaptive Instructional Systems core group divides up into three subgroups – the groups of conceptual modeling, interoperability, and evaluation. These groups are listed in order of the developmental order – defining the space, defining the parts, and evaluating the compliance and utility. The conceptual group has recently published a draft document standard for classification of adaptive instructional systems. The next steps for such a document, as it gels into standards documents, are to evaluate the component parts for the establishment of interoperability components. This paper performs this function.

The first major division that the AIS conceptual modeling paper makes is the division between the components of (1) the individual learner model, (2) adaptation model, (3), domain model, and (4) interface model. These divisions roughly align to the model put forth by Sottilare and Brawner (Sottilare and Brawner 2018), and (Sottilare et al. 2018), but also align to the Design Recommendations for Adaptive Instructional Systems book series (Sottilare et al. 2013; Sottilare et al. 2014; Sottilare et al. 2015; Sottilare et al. 2016), to the AutoTutor models (student expectation model, feedback model, content model, and tutor interface), the Yixue model (learner profile, adaptive engine, learning/content map, interface), and others. The author commends the conceptual modeling group for coming to a conclusion which encompasses many, if not all, of the existing systems; this eases the process of determining interchange and later evaluation.

These divisions then define what is contained within each of these models. A learner model shall have a structured representation of the learner’s knowledge, abilities, difficulties, etc. constructed of both raw and processed data. An adaptation model receives information from the learner and domain models to select/generate adaptions or recommendations, and may be performance-based (gating, looping), model-based (dynamic), or intelligent-based (self-improving), with various levels of adaptivity assigned to the various areas. A domain model contains the knowledge of what to teach (skills, knowledge), and the information about how to teach it (knowledge map, strategies/tactics). The interface model defines what is interchanged in between the student and the other system components, which is a relatively broad category intended to encompass both reading e-textbooks and flying hours in a physiologically monitored flight simulator.

This conceptual model of an adaptive instructional system defines where the interactivity points are defined – at the levels of [student->interface], [interface->domain], [domain->learner], [learner->adaptation], and [adaptation->domain||interface]. Further, the draft document produced by the conceptual modeling group assists in defining the area where data about the various items should be stored and accessed. As an example, a pass/fail grading for a student should be part of a learner model. A record of instructional actions taken across a population of students is a requisite component for an adaptivity model. A game-based environment, regardless of its complexity, is an interface to the rest of the adaptive instructional system.

As the IEEE group continues to work to develop standards – this division into finite components by conceptual modeling is important, as it defines the boundaries of the respective systems. As an example, SquirrelAI Engine Ontology Layer has multiple components – a Content Map, Learning Map, Mistake Ontology, and Learning Goals. Under the definition put forth in the Conceptual Modeling subcommittee – these are all a domain model. A domain model of a competing product in the same domain may or may not have these components, as well as either a larger or smaller number of components, such as the addition of a Feedback Generation model, or the removal of the Mistake Ontology.

The IEEE AIS subgroup on interoperability, however, is charged with dealing with the exchange of information between the components defined in the conceptual modeling group. As such, there isn’t, and more importantly, shouldn’t, be a standard for the interchange of information between a Learning Map and a Content Map in the Squirrel AI system. In the Generalized Intelligent Framework for Tutoring (GIFT) system, there shouldn’t be a standard for the interchange of information between the Assessment Model and the Domain Knowledge Map. In the AutoTutor system, there shouldn’t be a standard for defining the difference between the expected student dialogue and entered student dialogue. These types of distinctions are left to the individual systems. Simultaneously, there should be a way to communicate the Squirrel AI progress on learning objectives, GIFT progress on learning objectives, and AutoTutor progress on learning objectives, as these parts of information are needed by the learner model portion of each of the systems – or the system is not compliant with the written standard as defined later by the evaluation AIS subgroup.

The AIS conceptual group, as part of P2247.1 Draft Version 19, has defined the following system components as follows:

• Individual Learner Model (IEEE-ILM)– states and traits that describe the critical attitudes, behaviors and cognition of an individual and serve as a basis for adaptation decisions by the AIS

• Adaptive Model (IEEE-AM) – logic that assesses the states and traits of the learner/team and the knowledge from the domain model in order to select content, generate feedback or generate recommendations tailored to that learner; the adaptive model combines features from the instructional model and the recommender to make instructional decisions

• Domain Model (IEEE-DM) – includes the knowledge model, learning objectives, content, feedback, question banks, and other information relevant to the subject or topical area.

• Interface Model (IEEE-IM) – technology (tools or methods) that support learner/team interaction with the other AIS functions

The above definitions are placed alongside a diagram which shows the Learner (learner states/traits) and Domain Model (learning objectives, knowledge) feeding information to the Adaptive Model (instructional decisions), and the Adaptive Model feeding information to the Interface Model (visual presentation, speech). The IEEE-ILM and IEEE-DM connect to the IEEE-AM which connects to the IEEE-IM. The below section considers the implications of this distinction between the Generalized Intelligent Framework for Tutoring (GIFT), AutoTutor (AT), Squirrel AI (SAI), Watson Tutor (WT), Traditional Model Tracing Tutor (TMT), and Adding a Tutorial Model (ATM) tutoring systems as examples of interchangeable systems, dialogue systems, model tracing systems, cognitive-based systems, service-API driven systems, and service collections. This maps to the following example systems in the following manners:

• Generalized Intelligent Framework for Tutoring (GIFT) has the following major components:

  • Domain Module (GIFT-DM), Learner Module (GIFT-LM), Sensor Module (GIFT-SM), Pedagogical Module (GIFT-PM), and Gateway Module (GIFT-GM).

  • These map approximately as follows:

    • IEEE-ILM = GIFT-SM + GIFT-LM

    • IEEE-AM = GIFT-PM

    • IEEE-DM = GIFT-DM

    • IEEE-IM = GIFT-GM

• AutoTutor has the following major components (Cai et al. 2019)

  • Conversation Module (AutoTutor-CM), Tutoring Interface (AT-TI), and Conversation Engine (CE). The Conversation Module is composed of a Main Question (CM-MQ), Ideal Answer (CM-IA), Expectations (CM-E), Misconceptions (CM-M), Hints (CM0H), and Agent Assignments (CM-AA)

  • These map approximately as follows:

    • IEEE-ILM = AT-CM

      • AT-CM assessment against IEEE-CM below

    • IEEE-AM = AT-CE

    • IEEE-DM = AT-CM

      • IEEE-DM = AT-CM (MQ, IA, E, M, H)

    • IEEE-IM = AT-TI

• Squirrel AI Learning Engine has the following major components: Mistake Reasoning Ontology (SAI-MRO), Learning Map (SAI-LM), Content Map (SAI-CM), Goal Engine (SAI-GI), LRS (SAI-LRS), Profile (SAI-P), Recommendation Engine (SAI-RE), User State Evaluation Engine (SAI-USE), Diagnostic Engine (SAI-D), an outside system interface (SAI-I), and Realtime Classifier and Predictor (SAI-RCP) which has multiple subcomponents.

  • These map approximately as follows:

    • IEEE-ILM = SAI-LM + SAI-GI + SAI-LRS + SAI-P + SAI-USE

    • IEEE-AM = SAI-RE + SAI-D + SAI-RCP

    • IEEE-DM = SAI-CM

    • IEEE-IM = SAI-I

• Watson Tutor (WT) from IBM has the following major components, implemented as separate API services (Mukhi et al. 2017):

  • Tutor User Interface (WT-TUI), Next Best Action (WT-NBA), Response Analyzer (WT-RA), Hint Generator (WT-HG), Leaner Model (WT-LM), Feedback (WT-F), Question Recommender (WT-QR), and Conversation Engine (WT-CE)

  • These map approximately as follows:

    • IEEE-ILM = WT-RA + WT-LM

    • IEEE-AM = WT-NGA + WT-QR + WT-HG + WT-F

    • IEEE-DM = WT-CE

    • IEEE-IM = WT-TUI

• Traditional Model Tracing Architecture, as presented elsewhere (Heffernan et al. 2008) has the following major components:

  • Student Model (TMT-SM), Diagnosis (TMT-D), Feedback (TMT-F; buggy messages or hints), and a Tutor Response (TMT-TR) module where the student can place input.

  • These map approximately as follows:

    • IEEE-ILM = TMT-SM + TMT-D

    • IEEE-AM = TMT-F

    • IEEE-DM = TMT-D

      • note – TMT-D contains both the content and the diagnosis of the student

    • IEEE-IM = TMT-TR

• The Adding a Tutorial Model (ATM) Architecture grows the baseline TMT architecture to include more advanced components. The full list of the components is a Student Model (ATM-SM), a Tutorial Module (ATM-TM), and a Tutor Response (ATM-TR) module where the student can place input. The ATM-TM encompasses the TMT-D (ATM-D) as well as modules on Selection (ATM-S), Feedback (ATM-F), Reasoning (TMT-R), and Strategies (ATM-Strat)

  • These map approximately as follows:

    • IEEE-ILM = ATM-SM + ATM-D

    • IEEE-AM = ATM-T = ATM-S + ATM-F + ATM-R + ATM-Strat

    • IEEE-DM = ATM-D

      • note – ATM-D contains both the content and the diagnosis of the student

    • IEEE-IM = ATM-TR

Somewhat fundamentally, the author could continue to list systems and the mapping of systems against the conceptual model assembled in draft by the Adaptive Instructional Systems Conceptual Modeling group. However, the author believes that it is safe to say that the conceptual model components from the community map approximately against the models used in contemporary systems. It is relatively common for a system, based on models of dialogue (AT), cognition (TMT, ATM), architectural interchange (GIFT), service APIs (IBM), or service collections (SAI) to have the fundamental components defined in the modeling group. Among these systems there is only one point of less-than-perfect mapping – in the injection of content into architectural components.

At this point the author must declare his bias towards systems, such as GIFT and SAI, which can train many different domains without domain-specific configuration. However, there should be nothing in a standard which prohibits the creation of system-level components which are specific to the domain. Any standard should be sufficient to support both tutor-anything systems and tutor-something systems. A future vendor should be able to provide an instructional model for training a specific task, with future sales and marketing teams describing how their component was created and customized towards this task.

The conceptual model AIS group, at the time of writing, has written that “An adaptive instructional system (AIS) shall have a learner model.”, and that the data within the IEEE-ILM is relatively free to vary. This data may include physiology, competency assertions, raw data, habits, misconceptions, learner attributes, history of actions, and many other forms of data – the conceptual model describes the learner model as an imperfect digital twin of the learner.

Conceptually – each of the GIFT, AT, SAI, WT, TMT, and ATM have a model of the learner. Many of these systems have multiple modules for input to the IEEE-ILM, such as GIFT, which has a Sensor Module and a Domain Module for the separation of physiological data (eye tracking, face movements) and domain-dependent data (performance, history).

In the IEEE-ILM, however, this information is communicated to the IEEE-AM. In GIFT, this information is communicated as a table of states, traits, and learner performance (at/above/under performance). In AT, this information is communicated as a list of misconceptions and conversational assessments - information as it relates to ideal answers. In SAI, this information is communicated as a “user state evaluation” fed from multiple sub-processes. In CT, this information is communicated as analyzed responses couple with learner model. In both TMT and ATM, this information is communicated as a student model coupled with domain assessment.

As such, the author proposes that the standard adopted contain information on:

• Learner States (unknown values possible)

• Learner Traits (unknown values possible)

• Leaner Performance on Learning Objects (unknown values possible)

Given that unknown values are possible, there is the ability for multiple systems/components to run multiple assessments and for configuration to trust certain systems more for certain items. In this way, an LRS-based Learner Trait information module can complement a physiological Learner State information module and a conversational assessment Learner Assessment module to be packaged into a single IEEE-ILM conformant with a standard. Given that the states, traits, and performance are calculated in very different manners by the differing systems, there is reason to believe that there isn’t significant potential for standardization at the sub-component level, but the three values listed above and in the conceptual model align significantly to the various systems at the module level.

The conceptual model AIS group, at the time of writing, has written that “An adaptive instructional system (AIS) shall have an adaptive model.”, and that the data within the IEEE-ILM is relatively free to vary. Adaptations may include decisions about the learning experience (e.g., feedback, support, selection of different content), learner options, recommendations (e.g., next steps or future learning experiences), interaction with the current lesson, learning objectives, instructional resources, and methods of assessment (e.g., algorithms, gating, machine learning models, etc.). The draft standard expands on this by indicating that the IEEE-AM may be performance-based, model-based, multi-dimensional, learning, self-improving, or other types of implementation. What is not left to system designers is that the model must input information from the IEEE-ILM and output information to the IEEE-DM.

Conceptually, each of GIFT, AT, SAI, WT, TMT, ATM have an adaptive model. In GIFT, this consists of domain-general recommendations for scenario changes, feedback, and content presentation. In AT, this is the Tutoring Interface which presents hint/prompt/pump information. In SAI, these are recommendations, diagnostics, and groupings of students. In WT, these are generated hints, suggested next-actions, question recommendations, and feedback. In TMT, this is feedback. In AMT, this is expanded to include feedback as well as question selection, reasoning, and strategy selection.

As such, the author proposes that the standard adopted contain information on:

• Feedback (with levels)

• Next Steps (within problem/scenario)

• Next Steps (outside of problem/scenario)

Each system can find a mapping between their activities and the above actions for the system to take, while not every system performs each of these functions.

The conceptual model AIS group, at the time of writing, has written that “An adaptive instructional system (AIS) shall have a domain model”, and that the data within the IEEE-DM is relatively free to vary. That said, the data within the domain model is suggested to include, conceptually, the descriptions of good performance, descriptions of poor performance, content that the student should be able to know, and the maneuver space (feedback, changes) that the IEEE-AM can select from. In each of the systems, this maps in the manner described next.

GIFT has a Domain Module with a model of all of the content that can be presented within the content, assessments against the content, and feedback. AT has a description of the ideal answers, misconceptions, hints, and agent assignment. SAI has both content and classifier/predictor systems for goal management. WT has a conversational manager which has a model of content and assessment against it. Both TMT and ATM have models of tutor diagnosis and response which map roughly to the model of the domain.

As such, the authors suggest that, for the purposes of standardization, the stand should include:

• Content and delivery, provided via the tutor interface

• Assessment of Content, provided to the IEEE-ILM

• Deliverable Feedback, as requested from the IEEE-AM.

The conceptual model AIS group, at the time of writing, has written that “An adaptive instructional system (AIS) shall have an interface model.”. Somewhat fundamentally, however, the draft standard does not include any restrictions or requirements on the interface module. GIFT, AT, SAI, WT, TMT, and ATM all use dramatically different interfaces. Aside from the data which is interchanged as part as part of the IEEE-AM and the IEEE-DM, the IEEE-IM serves as an input function to the other core modules. GIFT has no standard interfaces, as it interfaces with dozens of systems, many of which are proprietary. AutoTutor operates similarly – interfacing with several base game engines. TMT/ATM systems are most frequently created through the use of the Cognitive Tutor Authoring Tools (CTAT) (Aleven et al. 2006) series of tools which creates interfaces at the same time as building systems (Tables 1, 2 and 3).

In 2018 the author was involved in a paper trying to establish initial sets of compliance messages for module-level interoperability between systems (Brawner and Sottilare 2018). The table which was the primary output from that work is below, which reflects the thinking from 24 months ago.

Based on the information presented in the conceptual modeling paper, stripping the conceptual modeling paper of its optional fields, and stripping the above table of its optional fields, this would be updated to reflect the following tables, with changes tracked via strikeout and bold, and presented in summary. The below table in red marks the changes in the 2018 proposed standard to be updated with the current thinking of the AIS conceptual modeling group. The final table within this work proposes standards for communicating information in accordance with the conceptual modeling information presented and the information presented and mapped to various systems within this work. This work serves as a contribution to document the current thinking of the AIS groups – but also as a starting point for creating module-level standards of interchange.