Multimodal Affective State Prediction in Mobile Settings

Download

Full text (PDF, 16.43Mb)

Open access

Author

Wampfler, Rafael

Date

2021

Type

Doctoral Thesis

ETH Bibliography

yes

Altmetrics

Download

Full text (PDF, 16.43Mb)

Rights / license

In Copyright - Non-Commercial Use Permitted

Abstract

Gaining awareness of affective states enables leveraging emotional information as additional context in order to design emotionally sentient systems. Applications of such systems are manifold. For example, the learning gain can be increased in educational settings by incorporating targeted interventions that are capable of adjusting to affective states of students. Another application consists of enabling smartphones to support enriched interactions that are sensitive to the user’s contexts. To accomplish the prediction of affective states in different contexts, multimodal data tailored to the domain need to be collected and adequately modeled. Research on such affective models mainly focused on expensive and stationary lab devices that are not well suited for everyday use, but recently, lightweight data collection in mobile settings gained interest. In this thesis, we present data-driven models for the prediction of affective states. We focus on models relying on lightweight data collection tailored to mobile settings. We further discuss the protection of privacy and the usability in real-world environments of the different data modalities. First, we propose a pipeline for affective state prediction based on front camera recordings (i.e., action units, eye gaze, eye blinks, and head movement) during math-solving tasks (active) and emotional stimuli from pictures (passive) shown on a tablet. Using data from a study with 88 participants, we demonstrate that our setup provides comparable performance for affective state prediction to recordings taken with an external and more obtrusive GoPro camera. In addition, we present a neural inpainting pipeline and techniques for image reconstruction of partially occluded and skewed faces. In combination with our novel hardware setup consisting of a cheap and unobtrusive mirror construction, the neural inpainting pipeline improves the visibility of the face in recordings of built-in cameras of mobile devices. Second, we present an automated pipeline capable of accurately predicting (AUC up to 0.86) the affective states of users solving tablet-based math tasks using signals from low-cost mobile biosensors. In addition, we show that we can achieve a similar classification performance (AUC up to 0.84) by only using handwriting data recorded from a stylus while users solved the math tasks. Given the emerging digitization of classrooms and increased reliance on tablets as teaching tools, we demonstrate that stylus data may be a viable alternative to biosensors for the prediction of affective states in educational settings. Third, we propose a system that analyzes the user’s text typing behavior on smartphones using a semi-supervised deep learning pipeline for predicting affective states. Using a data collection study in a laboratory setting with 70 participants on text conversations designed to trigger different affective responses, we developed a variational autoencoder to learn efficient feature embeddings of two-dimensional heat maps generated from touch data while participants engaged in these conversations. Using the learned embedding in a cross-validated analysis, our system predicts affective states with an AUC of up to 0.84. We demonstrate the feasibility of our approach to accurately predict affective states based only on touch data collected on smartphones. Fourth, we present an approach to expand affective state prediction to smartphone usage in the wild. We developed two-dimensional heat maps generated from keystroke and smartphone sensor data. Using data collected in the wild from 82 participants over 10 weeks, we demonstrate that by using a convolutional neural network we can achieve an AUC of up to 0.85 for the prediction of affective states. We also show that using less privacy-invasive sensor data alone, a similar performance (AUC up to 0.83) can be achieved. In addition, by personalizing the model to the user, the performance can be increased by up to 0.07 AUC. We exemplify the usability of our model for the prediction of affective states in real-world environments based on readily available smartphone data. Finally, we describe two widgets for a compact and unobtrusive visualization of users' affective states on mobile devices. We test the widgets on intuitiveness and understandability based on a user study with 644 participants. We conclude with a discussion of the advantages and limitations of our methods and possible future work. As we believe that the knowledge of affective states will become crucial for a variety of systems and in different domains in the next decade, we hope that our work provides an important contribution in such a direction. Show more

Permanent link

https://doi.org/10.3929/ethz-b-000516399

Publication status

published

External links

Search print copy at ETH Library

Contributors

Examiner: Gross, Markus
Examiner: Holz, Christian

Examiner: Hilliges, Otmar

Examiner: Iqbal, Shamsi

Publisher

ETH Zurich