Extending the Concept of User Satisfaction in E-Learning Systems from ISO/IEC 25010

Evaluating user satisfaction in applications dedicated to learning is a complex activity. This situation is due in large part to the increase in trends in learning styles, the diversity of students, technological advances, etc. [1]. These characteristics become important challenges for the definition of methods that allow the evaluation of designs of e-learning systems in a more precise and objective way with respect to the traditional methods.

The main task for a user when faced with an e-learning system is to learn. This is a rather abstract process. As such, professionals in the assessment of User Experience (UX) should significantly increase their efforts when dealing with such environments. Authors such as Squires point out the need to incorporate ease of use into learning in computer tools, and also point out the lack of reciprocity between the areas of Human Computer Interaction (HCI) and educational computing [2]. An educational application can be useful, but not in the pedagogical sense and vice versa [3,4,5].

In this sense, it is necessary to make use of a model that is consistent with the new challenges implied in UX evaluation, in particular evaluation of user satisfaction in e-learning environments. For this purpose it has been found that ISO/IEC 25010 provides a definition that contemplates different nuances of general user experience that can be fully adapted to the human-computer system, including computer systems in use and software products in use [6]. Within this standard, satisfaction is divided into four sub-characteristics intended to identify user needs when using a system in a specific use context [6]: utility, confidence, pleasure and comfort. Although a product is satisfactory in one given use context, it may not be in another with different users, tasks or environments [7]. Therefore, it is considered important to focus more on the set of factors that determine the nature of an e-learning system, since these are the main components for adapting the satisfaction sub-characteristics established in ISO/IEC 25010 within the e-learning context. The adaptation of these sub-features will allow the defining of the parameters necessary to evaluating user satisfaction in e-learning environments.

2.1 Satisfaction in UX

Due to the current boom in technology use in society, it is logical that the satisfaction of the end user becomes the primary conditioner for the success or failure of any interactive system. In the event that the user does not achieve their objectives or the software does not meet their needs, they will simply abandon it in search of an alternative, from the competition [8]. User satisfaction is a complex concept, difficult to delimit, but of paramount importance since it will model the UX, making possible or preventing the achievement of its objectives [9]. In this context, several efforts have been made by the HCI community to establish the factors influencing satisfaction, to be able to manage them well in designing interfaces [10]. One of the widespread concepts that aid this aim is usability, which also seeks to provide elements for measuring the degree of satisfaction as well as the efficiency and effectiveness with which specific users can achieve specific objectives, in specific contexts of use [11, 12]. Some authors extend this concept and establish two dimensions of usability: objective or inherent usability and subjective or apparent usability ([9, 13, 14] cited in [8]). The first refers to the functional or dynamic part of the interface and focuses on how to make the product easy to understand and learn (efficiency and effectiveness) [8, 15]. Subjective usability, on the other hand, is more related to the visual impression of the interface, which the user perceives by means of design, aesthetics and interaction with the interface (user satisfaction) [8, 14, 15].

However, recent research has indicated that the usability of a product may not be the only, or even the main, determining factor in user satisfaction [16, 17], due to the fact that in recent years there has been evolution in interactions between user and different interactive systems, in which this has gone from being a purely functional interaction, determined by the efficiency and effectiveness of the use of the product, toward being a sensory vision projected through pleasure, the subjective nature of the experience, the perception of a product and the emotional responses [16,17,18]. Thus, user satisfaction cannot be analyzed as an attribute of usability; instead, usability must be understood as a factor in the consolidation of elements that tend to satisfy user satisfaction [8]. UX then becomes a holistic view of interaction with a product [18], as it comprises the full set of effects brought about by the use of a product, including aesthetic experience, sense experience, emotional experience, and other aspects that involve user satisfaction [17].

2.2 Satisfaction in UX with E-Learning

Evolution in the concept of user satisfaction and in the understanding toward UX, also extends to teaching and learning systems. However, while virtual education is becoming one of the most representative approaches in the Internet [19], most studies conducted on evaluations of e-learning systems have provided minimal participation in UX aspects [20], reflected in educational applications having higher dropout rates compared to traditional courses led by an instructor. Many reasons may explain these high dropout rates, such as the relevance of the content, the level of comfort with the technology, the availability of technical support, etc., but one important factor is the lack of cognitive and emotional characteristics such as guidelines in the development of learning spaces with valid designs [1, 21]. Consequently, evaluation studies of UX in e-learning are scarce [21].

Within such a scope, UX comprises an essential element in the ability of the student to acquire knowledge and competences in a satisfactory way. The evaluation of UX also therefore becomes a means of support so that learning and teaching processes are productive, since its objective is to design systems that are intuitive from which users can interact easily with e-learning systems and focus on acquiring the knowledge and skills provided in their training [22]. The less effort devoted to understanding and learning the functionality of the system, the more the student can devote to learning [23]. It should also be noted that one of the relevant aspects considered in UX - and in the particular case of e-learning have a substantial role - are emotions [24]. Several studies point out that emotions have a decisive influence on the motivation, attention span and performance of the student [25,26,27]. Therefore, in evaluating user satisfaction of e-learning systems, it is important to treat the affective qualities involved as an inherent component of UX, since these compromise the functional and non-functional attributes of a system, functional ones being understood as those related to usability, utility and accessibility, while in the case of the non-functional ones, these refer to aesthetic, symbolic, and motivational qualities, among others [25].

User satisfaction assessment methods in e-learning are complex both in practice and in research [28] and are usually focused from a pragmatic perspective, in which assessment of the efficiency and effectiveness of an interactive system take precedence; or instead, they are oriented toward assessing the quality of the teaching, in the same way as happens in a traditional classroom [28, 29]. This constant fluctuation between preferences in the way in which user satisfaction is assessed in e-learning causes fundamental aspects to be neglected that are part of the interaction with these systems and that require to be assessed. Following from this then is a compilation of the literature from two approaches:

3.1 Evaluation of User Satisfaction in Interactive Systems

User satisfaction can be considered as the main parameter in the use of interactive systems [30]. From this point of view, several studies implement different types of work related to usability and user satisfaction [30]. However, many of these do not provide details on the questionnaires used for assessing satisfaction [9], several even rethinking the constructs and measures of user satisfaction, omitting validated and readily available questionnaires [9, 30]. Among some of the questionnaires widely used in industry and academic environments are found the following described in Table 1.

Although these measures are widely used, their diagnostic value is diminished on comparison with more specialized measurement instruments, i.e. those based on particular aspects of a given context [43]. Although many authors support this premise, very little has been done to examine critically and in-depth the implications and specificities of assessing UX in e-learning (particularly user satisfaction) [44, 45]. For this reason, there is a constant need in UX professionals to base their research on virtual educational contexts, on instructional models, and on learning styles, among other aspects that determine the quality of learning in e-learning systems. These fundamentals are one of the main reasons why it is necessary either to define new UX evaluation techniques in e-learning [44] or to complement existing ones. In addition to the elements of context, emotions are another critical conditioner in interaction with e-learning systems, given their value in cognitive processes. It is therefore crucial to consider the emotional states of users within the assessment parameters [44].

3.2 Evaluation of User Satisfaction in E-Learning

In the literature, a great number of studies measure various user satisfaction factors in e-learning systems. They usually focus on the quality of teaching and learning. However, these measures are not appropriate for an e-learning context, since the role of an “e-learner” is different from that of a traditional learner [28]. This special group of users (e-learners) has a unique view regarding satisfaction [29]. Table 2 shows some important research that has focused its efforts toward the definition of factors that affect user satisfaction in e-learning systems.

In response to the different problems and gaps that exist in the evaluation of user satisfaction in e-learning systems, the need arises to define the attributes that ought to be considered for evaluation. This approach is described below:

4.1 Attributes for Assessing User Satisfaction in E-Learning

To define the assessment attributes, a contextualization of each of the sub-characteristics of user satisfaction defined in ISO/IEC 25010 is carried out. For this purpose, the following treatment has been carried out on each sub-characteristic of satisfaction:

Utility. This corresponds to the degree to which a user is satisfied with the perceived achievement of pragmatic goals, including the results and the consequences of the use [6].

According to this concept, it is necessary to establish what will be considered as pragmatic goals. Hassenzahl [18], states that a product is perceived as pragmatic if it provides an efficient and effective means for achieving the objectives. In this sense, a student could have a large number of objectives when interacting with an e-learning system. However, in general terms the main objective of the student is to learn [52]. Consequently, it is considered that the pragmatic objectives will be directed to favor the learning process in such a way that this is achieved in a simple, fast way, using the least possible amount of resources.

To find a match for the above, studies are referred to from which the pragmatic goals of the student can be obtained and the usefulness thus determined of interacting with an online learning environment. An example of work suited to this purpose is the TAM Model [53], which emphasizes perceived utility, a concept that has been adapted by different authors and brought to the context of online learning environments [54,55,56,57]. Pragmatic objectives can be abstracted from these studies, such as performing learning tasks easily and quickly, facilitating the learning of course content, and improving learning performance, among others no less important.

Trust. Trust is understood as the degree to which a user has confidence that a product or system will behave as intended. Nevertheless, given that confidence is part of user satisfaction, which compromises a more generalized concept that is subject to the specific context of use [6], there is a clear need to consider elements from this context (in this case e-learning) that define a construct that supports confidence.

Consequently, the components that would become part of this purpose are obtained from instructional design, particularly the work of Keller [58], who has made several contributions to the subject and is considered one of the most representative authors in this area [59]. Keller proposes a model of four categories that make up the motivation of a student. These correspond to Attention, Relevance, Confidence and Satisfaction (ARCS) [58]. In this model, confidence is defined as strategies that help develop positive expectations for the achievement of student goals in e-learning, in such a way that the apprentices experience their successes as relating to their efforts and not to external factors such as luck or the difficulty of the task [58].

Moreover, from the perspective of interaction between a user and a software system, trust is positively affected by ease of navigation within the environment [60, 61] and good use of visual design elements [4]. The model of confidence treated in [62] states that when an interface is consistent in terms of visual appearance, elements of interaction (buttons, menus, etc.), navigation, and terminology, this increases user confidence. Likewise, it has been identified that aspects such as lack of control by the user [62] and curt, non-constructive error messages ([63] cited in [62]) have a negative impact on user confidence.

Pleasure. The degree in which a user gets pleasure from fulfilling their personal needs [6]. Such needs may include needs to acquire new knowledge and skills, communicate personal identity and provoke pleasant memories [6].

Hassenzahl, one of the most influential researchers in the area of UX, compiles a list of the 8 psychological needs of humans, based on the needs proposed by Sheldon et al. [64]. In addition there are widely recognized studies, arising from outwith the psychology that support them: [65,66,67,68]. These needs involve the dimension of pleasure as the confirmation of the possibilities of a desirable event [64].

According to Hassenzahl’s studies, stimulation, identification and evocation are considered important needs in the context of interactive technologies, and in turn correspond to the hedonic attributes that underlie pleasure [18]: influence - popularity, pleasure - stimulation, and sense - self-realization.

Comfort. This is related to the degree to which the user is satisfied with the physical comfort [6]. Jordan [69] and Tiger [67] establish that physical comfort is determined by the pleasures derived directly from such the senses, such as touch, taste and smell. This sub-characteristic of satisfaction will not actually be taken into account because of its apparent weak connection with conventional interaction styles in e-learning and its inability to emerge as a clear need for the present study.

Based on the above, the suggested approach allows a combination of interface design, the motivation construct in learning, and user experience. Thus, by including these structures within the UX evaluation process of e-learning systems, a more objective and accurate approach is obtained of both the pragmatic and hedonic objectives implicit in student satisfaction.

The absence of robust and reliable mechanisms for translating user needs into design features is a factor that considerably limits the assessment process of a system, resulting in a high degree of uncertainty in the results of the evaluator. Generally, this is due to the fact that the process is based on the experience of the evaluator and on his abstraction capacity, since it is based on the intuition and subjective criteria of the evaluator.

The way in which the different characteristics and sub-characteristics set out in ISO/IEC 25010 are defined (in particular satisfaction and its corresponding sub-characteristics) highlights the versatility of the standard to adapt it to a given context. This quality also facilitates the understanding of the elements of the context required to define the assessment parameters.

The need to redefine processes related to the assessment of user satisfaction in interactive systems, in particular e-learning systems, is evident, since traditional user studies focus on objective parameters, associated with measurable and verifiable aspects in an interactive product, neglecting hedonic components that are part of user satisfaction and are directly related to the degree of involvement and motivation that a person shows when using an e-learning system.

The majority of research related to the evaluation of user satisfaction in e-learning are oriented to considering satisfaction in a subjective way. In the present study we propose the integration of different structures that complement each other, in order to deal with the particularities implicit in the design of e-learning systems: learner-centered design, instructional design, and UX. This aspect allows a holistic view regarding the components that are part of the evaluation of user satisfaction in e-learning systems.

Current practice in the evaluation of UX suggests that the choice of UX measurement instruments is difficult and that the conclusions of some usability studies are weakened by the elements that they evaluate and by the way in which they use UX evaluation measures to provide support to the quality in use of software products. Suggestions on how to respond to identified challenges can provide tools that facilitate UX evaluation from an emotional perspective, and in turn establish more valid and complete UX measures regarding the perception of a student when faced with using an e-learning environments.