Towards a Multi-Modal Methodology for User-Centred Evaluation of Adaptive Systems

This paper presents an approach and methodology for user-centred evaluation of adaptive systems. In contrast to layered evaluation approaches that decompose adaptation into its constituents, our approach conceptualises the quality and benefit for the user into separate evaluation qualities for a comprehensive and multifaceted evaluation. Instruments of different modalities are used to measure these qualities from the user perspective. A service is presented that takes up this approach and enables time- and cost-efficient evaluation by defining and re-using evaluation qualities and instruments, as well as collecting and analysing data based on these definitions. This approach allows to compare different adaptive systems by using the same qualities and adapting the instruments to the specific characteristics of the particular adaptive system.