Auto-Adaptive Interactive Systems for Active and Assisted Living Applications

There is an emerging trend focused in auto-adaptable and self-reconfiguring ambient intelligence systems in order to support smarter habitats. The associated technological challenges include active perception features, mobility in unstructured environments, understanding human actions, detect human behaviors and predict human intentions, access to large repositories of personal and social related data, adapt to changing context. In the case of social artificial agents, systems must incorporate features that allow an agent to be capable of delivering a sociable experience with the user. These can be classified as cyber-physical systems, as they are designed as a network of interacting elements with physical input and output instead of as standalone devices. These features are paramount for applications in the field of Active and Assisted Living (AAL). In AAL, the primary goal is to provide solutions that help people through ageing, by promoting active and healthy living. Part of being active and healthy includes socializing. In a vast number of cases, this activity is done in care centers or nursing homes. In such scenarios, the demand for social stimulation as part of care service aggravates the need for scarce qualified human resources. Thus, technological solutions are seen as a benefice but the challenge of adoption remains associated to user acceptance. However, most of interactive approaches rely on explicit information (e.g. direct commands) to achieve expected behaviors from the system. How can we take into account implicit information to achieve human-like interaction skills in cyber-physical systems? Our hypothesis is that, we can overcome limitations of current interaction functionalities by integrating contextual information (i.e. implicit) to improve algorithms accuracy when performing under very different conditions, to select most adequate algorithms to provide a given functionality, and to adapt interfaces and interaction patterns according user intentions and emotional states. In Sect. 2 we present our contribution to the field of cyber-physical systems focusing in applications for the AAL domain. Section 3 describes a survey of relevant literature. In Sects. 4 and 5 we present our approach and initial results. Sections 6 and 7 conclude the paper summarizing the major findings and stating further work.

Our contribution aims to develop automatically adaptive cyber-physical agents that will evolve according to the end-user needs and intentions. The overall goal is improving acceptance of interactive technologies, with special focus to robotics and cyber-physical systems, by users of assistive technologies, developed to address challenges from the Active and Assisted Living (AAL) domain.

From experience, it is common that the misunderstanding of functionalities and handling of new technology often leads to rejection or even to fear if the system is not behaving as expected. Thus, the challenge of user acceptance is of vital importance for future systems and still one of the major reasons for reluctance to deploy or introduce cyber-physical agents in AAL applications. The analysis of the results of relevant research projects allows us to identify common needs, which are repeatedly requested by end-users. In summary, users expect intuitive interaction with social agents. Therefore, Natural Interaction features (e.g. gestures, speech, etc.) and adaptation to the user’s general profile, specific needs and intentions, were identified as high priority requirements. In the AAL domain, these features must address the needs of users with cognitive or physical skills degradation (e.g. elderly or impaired people) and aim to compensate these limitations, hence enriching user experience and accessibility.

Taking into account this list, we conclude that interaction aspects are a central point for end-users. Therefore, we can summarize the major features according Fig. 1.

Scientific and technological challenges related with HMI related features, as summarized from end-user needs

Full size image

To refine reasoning and to integrate context information about a person within his/her home environment in the broadest sense possible, namely the identification of activities of daily life (ADLs) in order to provide the correct services to attain end-user goals, models must be developed that are capable (1) of using a priori knowledge, either hardcoded or from experience, and (2) of evolving in time. However, there is not available yet a satisfactory implementation that is capable of reason and learn based in of context information in a distributed system, which involves sharing contextual information between different components of the system [1].

The current state of the art in the domain of user interaction with cyber-physical agents is still mainly based on graphical user interfaces implemented on stationary (often wall-mounted) touch panels (or terminals), portable devices (like touch tablets or mobile phones) or by using TV-sets (often in combination with set-top-boxes) and their remote controls as front end for the elderly. The graphical user interfaces are often text or icon based. Speech input is becoming more and more stable but is still in the research and development phase since surrounding noise and sounds, speaker localization and optimum input with ambient microphones (by avoiding wearable microphones) are still a challenge and are addressed by several research projects (e.g. Companionable etc.). Recent approaches as well apply avatar technology to enhance (increase acceptance and entertainment value) and personalize user interaction [2]. Human-machine interaction (HMI) technologies and corresponding cognitive capabilities of agent systems have seen many developments in the last few decades – see [3] for an extensive survey. Solutions for multisensory active perception and attention allocation have greatly evolved [4], as has multimodal human emotion and dialogue analysis and human-like emotion and dialogue synthesis [5] and also human behavior analysis [6, 7]. In summary, we identified that existing cyber-physical systems that have been developed recently in order to provide assistance in domestic, professional and public environments are based on closed architectures, thus being operational only in specific contexts of usage, equipment and data. Furthermore, one drawback of existing implementations is commonly that perceptual models neglect the context and the spatial relation during the perception process. Our conclusions from the literature survey point us in the direction of proposing a human-machine interaction framework that considers integration as keystone to overcome existing challenges. This framework goes beyond the state-of-the-art approaches by taking into account contextual information in the interaction process.

Our main contribution aims to create a reference framework that targets improving the current state of the art of auto-adaptive interactive systems for Active Assisted Living applications. The innovative aspect of this framework is the use of contextual information to orchestrate the behaviors of the social agent (i.e. perception and actuation features involved during interaction). This paper extends the concepts and design presented from our previous work in [8].

4.1 Context-Based Human-Machine Interaction (CB-HMI) Framework

We consider the integration of contextual information a vital aspect in HMI approaches. In our approach we consider context recognition to be the process that leads to auto-adaption of the system. We call this process the Contextual adaptation loop. As depicted in Fig. 2, this is a periodic process that operates in the background of the system, while the user is interacting with it. It plays the role of detecting changes in the context and provides this information to the agent’s main execution loop (e.g. perception-action).

The execution of this process in mainly related with the Interaction inputs, Environment analysis and Interaction control components depicted in Fig. 3, which invokes the adaption of the agent’s interaction behavior (i.e. changes the orchestration of Interaction outputs components in Fig. 3). This process must take into consideration restrictions or rules that may apply for different situations. This information corresponds to Context models that are learnt and stored, in the Context recognition component, for future reference. The CB-HMI framework is designed to be modular, loosely coupled and to take advantage of context models to orchestrate different NI modules and user interfaces. The goal is to provide features allowing the agent to be able to adapt its behavior according to the user’s expectations and intentions, as illustrated in Fig. 3.

Context verification process

Full size image

Framework component diagram

Full size image

The components are grouped into four main categories. Interaction inputs include the components related with user interaction inputs (i.e. Speech Recognition, Gaze recognition, Gesture recognition, Touch/Haptics recognition and Psyco-physiological recognition). These modules contribute to the overall system by perceiving the user’s explicit state (i.e. intentions and emotions). Environment analysis is also related with perception mechanisms, but it contributes with implicit states. It refers mainly to what can be recognized and identified from the environment and from user habits profile (i.e. Object recognition, User identification, ADLs recognition, User behavior analysis, Time, Location). Interaction control components receive inputs from the Interaction inputs components and handle the data fusion and fission in order to update the agent’s internal state and determine appropriate response (i.e. Multimodal data fusion, Multimodal data fusion, Context recognition, Behavior synthesis and Dialogue management). Interaction outputs are directly linked to the synthesis components generating the appropriate verbal and non-verbal behavior of the companion and communicate relevant information to the user via the graphical user interface (i.e. Expression synthesis, Movement synthesis) and vocal outputs (i.e. Sound synthesis).

To validate our approach we started the implementation of a cyber-physical social agent that took the form of a virtual assistant. Our objective at this stage was to create a working prototype in line with the state of the art to establish a baseline for future work. In this first version of the prototype we addressed the Main execution loop depicted in Fig. 2. The prototype included an initial implementation of some of the components listed in Fig. 3, namely: Speech recognition, Dialogue Management, Expression synthesis, Sound synthesis and Movement synthesis. The system operates on an all-in-one computer (Lenovo ThinkCentre Edge 93z All-in-One) and supports various interaction modalities. According to the preferences of the user and depending on the distance between the user and the hardware, interaction can be done either by speech (2–3 m) or via the Graphical User Interface (GUI) on the computer’s touchscreen (arm length). The virtual assistant interface (Fig. 4a) closely simulates human conversational behavior through the use of synthesized voice and synchronized non-verbal behavior such as head nods, posture shifts, facial expressions and hand gestures. The interface is built on top of the Behavioral Markup Language (BML) to describe the physical realization of behaviors, such as speech and gesture and the synchronization constraints between these behaviors.

Implemented prototype (a) and user testing (b)

Full size image

5.1 Experimental Validation

The current implementation of the system was validated in three different sites. Test trials were conducted in Switzerland, in the Netherlands and in Portugal. The criteria for selecting the participants stated that the target group should be 65+ years of age, living alone, and independent and or with light impairments related with ageing. In Table 1 we present detailed information regarding participant characteristics.

Evaluation Process:

In all test sites, the common approach to introduce the system to the elderly was by gathering small groups of 2 to 3 and then the elderly interact with the system individually (Fig. 4b). These sessions worked as focus group. The Think aloud method and the System Usability Score (SUS) were used as tools to assess the overall impressions and usability of the system.

5.2 Results

System Usability Scores have been calculated based on the 10 questions of the validated questionnaire for SUS. In this tool, a score between 50-70 is positive, where above 68 is considered as ‘above average’. Besides, a total SUS score below 70 is problematic since this value seems to be the threshold for a “good” usability. The overall results from the trials are depicted in Fig. 5.

SUS scores results

Full size image

In general all participants found the system very interesting and showed a lot of motivation in participation. The SUS scores, for the Swiss and Dutch users, were all above 68 and especially below 70 which means that the system is as “above average” and below the threshold for a “good usability”. It seems to summarize what the participants reported and the results shown: the idea behind the system was very good but the system needs to be improved, to be more intuitive in its use. In Portugal, the overall of SUS scores were 67, which is still considered as positive but falls bellow “above average”. This may be an indication that, for those users, the system requires a deeper adaptation. Evidence from the study revealed that the (cognitively impaired) older adults found it difficult to gain insights into the system’s possibilities and cope with the limitations. Age related cognitive degradation, specifically age related memory changes and their effects on learning, make older adults feel mentally stressed when interacting with new technologies.

The paper presented a framework that is being developed to improve the human-machine interaction process within cyber-physical systems. We identified HMI as a vital aspect in the adoption of Active and Assisted Living solutions. Moreover, we identified key HMI features that are commonly referred by end-users as a must, which are still imposing relevant scientific and technical challenges. To address these challenges, we implemented a virtual agent that aims to serve as artificial and interactive companion. The current version of the system was tested in three end-users organizations, in three different European countries. The results showed that the current version of the system can reach positive scores, but it is still considered to be in the borderline of usability. Future improvement will require further development of the proposed framework.