ISO 9241-11 Revised: What Have We Learnt About Usability Since 1998?

What is usability?^{Footnote 1} In the English language, usability is typically defined as the “capability of being used”, implicitly the capability of an entity to be used. In the 1980s and 1990s a considerable amount of material was published describing the attributes that would make a product usable (e.g. Smith and Mosier [20], and the early parts of the ISO 9241 series [9]). From this perspective, usability could be designed into the product, and evaluated by assessing consistency with these design guidelines, or by heuristic evaluation. This was the perspective taken in ISO 9126:1992: “Software engineering—Product quality”, which defined usability as “a set of attributes of software which bear on the effort needed for use and on the individual assessment of such use by a stated or implied set of users”.

However an alternative approach recognised that the same product could have significantly different levels of usability depending on who was using it with what goals. In order to achieve usability for the intended users it would be necessary to address the actual outcomes of their use of the product. This was the approach advocated by Whiteside, Bennett and Holzblatt in 1988, based on an operational view of usability in terms of user performance and satisfaction [22].

The approach taken to usability in ISO 9241-11 (1998) was similar, defining usability as: “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.”

This approach has the benefit that it directly relates to user and business requirements: effectiveness means success in achieving goals, efficiency means not wasting time and satisfaction means willingness to use the system.

ISO 9241-11:1998 explains that in order to specify or measure usability it is necessary to identify the goals and to decompose effectiveness, efficiency and satisfaction and the components of the context of use into sub-components with measurable and verifiable attributes. The standard identifies the benefits of this approach:

• The framework can be used to identify the usability measures and the components of the context of use to be taken into account when specifying, designing or evaluating the usability of a product.

• The performance (effectiveness and efficiency) and satisfaction of the users can be used to measure the extent to which a product is usable in a particular context.

• Measures of the performance and satisfaction of the users can provide a basis for the comparison of the relative usability of products with different technical characteristics which are used in the same context

• The level of usability needed for a product can be defined, documented and verified (e.g. as part of a quality plan).

ISO 9241-11:1998 has been highly successful in providing an internationally accepted basis for understanding and applying usability. Its definition of usability is widely referenced in research, industry, and other international standards. This widespread use has been accompanied by a greater level of understanding of usability than was available when this first version of ISO 9241-11 was developed. As a result, work was started in 2011 to revise this standard to provide enhanced guidance based on over a decade of experience with the concepts that it introduces. This revision is intended to retain the basic concept of usability and to provide users of the concept with further levels of understanding about usability taking account of what we have learnt about usability since 1998, including the issues identified below.

2.1 It Is Important to Understand the User’s Experience

In much early work in industry, usability was operationalised primarily in terms of the user’s performance (effectiveness and efficiency), which was regarded as the prime issue given the many problems that were experienced by users of commercial systems. But as use of complex consumer products and of the World Wide Web became widespread, there was an increasing awareness of the importance of the user’s subjective reactions and emotional experience. This has led some authors to regard usability as being restricted to “ease-of-use”, and relegate it to the role of a “hygiene factor”, e.g. [5] usability “…is a thermometer that sets the ‘hygiene’ level of a product. Users today take the ‘ease of use’ part of product concepts for granted and will not praise the fact that a product or service has good usability.” Similarly, Hassenzahl et al. [4] differentiate between ease of use as a “pragmatic quality being a ‘hygiene factor’, enabling the fulfilment of needs through removing barriers but not being a source of positive experience in itself”, and “hedonic quality being a ‘motivator’, capturing the product’s perceived ability to create positive experiences through need fulfilment”. ISO 9241-210:2010 “Human-centred design for interactive systems” defines user experience as a “person’s perceptions and responses resulting from the use and/or anticipated use of a product, system or service”. User experience focuses on the experience of an individual in contrast with the view of effectiveness, efficiency and satisfaction as representing the collective responses of a group of users.

One objective of the revision of ISO 9241-11 is to clarify that the satisfaction component of usability includes aspects of user experience.

2.2 There Is More to Usability Evaluation Than Usability Measurement

ISO 9241-11:1998 focussed on the evaluation of usability by user based measurement of effectiveness, efficiency and satisfaction, as this was a convincing way of demonstrating the existence of usability problems to system developers. ISO/IEC DIS 25066:2015 “Common Industry Format (CIF) for usability: Evaluation report” provides a broader view explaining that usability evaluation can be based on inspection to identify potential usability problems, in addition to observation of user behaviour and the collection of user-reported data. The revision of ISO 9241-11 needs to make it clear that effectiveness, efficiency and satisfaction represent the intended outcomes of interaction, but that their measurement does not represent the only way of evaluating usability.

Another issue is that many authors have emphasised the importance of more specific aspects of usability. For example Nielsen’s definition of usability [19] includes, in addition to efficiency in normal use and satisfaction with use:

• Learnability in early use.

• Memorability after a period of non-use.

• That errors during use can be corrected, and do not lead to undesirable consequences.

2.3 Avoidance of Negative Outcomes

In the existing ISO 9241-11 effectiveness focuses on the accuracy and completeness with which goals are achieved. But an unintended outcome can have significant undesirable negative consequences for the individual or the organisation (such as inconvenience, wasted time, or financial loss). The ISO/IEC 25010:2010 “System and software quality models” standard takes account of both positive and negative outcomes by defining “quality in use” as a combination of the positive outcomes of usability in the existing ISO 9241-11 combined with freedom from risk of negative outcomes. The revised ISO 9241-11 also needs to explain how to take account of the risk of any negative outcomes that could arise from inadequate usability.

3.1 Requests for Changes and Feedback

The revision of ISO 9241-11 was preceded by consultation with the countries that participate in the ISO TC159/SC4 Ergonomics standards committee and the related ISO working groups. The feedback (which identified many of the issues above) provided a starting point for the changes listed below that have been made in the first draft that has been circulated for national voting and comment. The text will be revised based on these comments, and will be circulated for further voting and comment later in 2015.

3.2 Changes Made in the New Draft

Systems, Products, Services and Environments. ISO 9241-11 currently applies to the usability of “products”. In line with changes in standards such as ISO 9241-210 this has been extended in the new draft to “products, systems and services”, as the concept of usability applies equally well to all these categories. The new draft also explains that although environments are considered as part of the context of use, user interactions with a specific environment or component of the environment can be considered in terms of the usability of an environment (e.g. the smoothness of a path used by a wheelchair).

Goals. The current standard only mentions goals that achieve well-defined outputs. In reality, people may have other reasons for interacting with a product, system or service, so the new draft takes account of a much wider range of goals that include aspects of user experience:

1. (a)

   output related outcome(s) that could either be assigned (in an organisational context) and/or personally chosen;

2. (b)

   personal outcomes such as entertainment or personal development;

3. (c)

   usability outcomes in terms of levels of specific (sub)dimensions of usability, such as the desired level of accuracy;

4. (d)

   other outcomes (e.g. related to safety, security or privacy) to be satisfied in the course of achieving output-related or personal outcomes.

Effectiveness. The effectiveness with which a goal is achieved was previously defined in terms of accuracy and completeness. Appropriateness has been added as an additional consideration that can include: a) the form and needed degree of precision of the output (e.g. is information displayed appropriately on a web page?) and b) avoidance of errors and minimization of the risk of any unacceptable consequences that could arise from lack of accuracy and completeness. Adding appropriateness to the other components of effectiveness goes some way towards taking account of both the potential positive outcomes (accuracy and completeness) and the risk of negative outcomes.

Efficiency. Efficiency was previously defined as the ratio of effectiveness divided by the resources consumed. While this is scientifically correct as a productivity measure, it is only meaningful for continuous output, which is not a common situation. So, efficiency has been redefined in the revised standard as the resources (time, human effort, costs and material resources) that are expended when achieving a specific goal (e.g. the time to complete a specific task).

Satisfaction. In the current ISO 9241-11, satisfaction is defined as “freedom from discomfort, and positive attitudes towards the use of the product”. The new draft identifies a much wider range of personal responses, including those that have been highlighted in research on user experience: “the extent to which attitudes related to the use of a system, product or service and the emotional and physiological effects arising from use are positive or negative”.

Context of use. The current ISO 9241-11 does not say anything about how wide a range of users, tasks and environments should be included in the context of use. The revised standard explains that usability can be related to: (a) All potentially relevant contexts of use (when considering overall usability), (b) Specified contexts of use (the users, goals and environments of particular interest), (c) A single instance of the context of use, or (d) The context of use for a single individual (for individual usability when considering a user’s experience).

The current ISO 9241-11 provides detailed information on how to specify the context of use. This has been removed from the new standard as it is now included in ISO/IEC 25063:2014 “Common Industry Format (CIF) for usability: Context of use description”.

Scope of “Usability”. The revision explains the relationships between the approach to usability in ISO 9241-11, and other interpretations. By defining usability in terms of the measurable outcomes of effectiveness, efficiency and satisfaction, ISO 9241-11 takes an approach to usability that:

1. (a)

   Focuses on the outcomes of interaction rather than the user interface design activities and resulting product attributes that make a product usable.

The new draft explains that the term “usability” can also be used as a qualifier to refer to design related activities and product attributes. Thus it may be used refer to the knowledge, competencies, activities and design attributes that contribute to usability (such as usability expertise, usability professional, usability issue, usability method, usability evaluation, usability problem, and usability guidance).

1. (b)

   Defines usability a high-level concept rather than referring to normal use of a product in contrast to learning to use or reuse a product.

The new draft makes it clear that usability applies to all aspects of use, including:

• Learnability, to enable new users to be effective, efficient and satisfied when learning to use a new system.

• Regular use, to enable users to achieve their goals effectively, efficiently and with satisfaction.

• Error protection, to minimise the possibility that users can make errors that could lead to undesirable consequences.

• Accessibility, so that the system if effective, efficient and satisfying for users with the widest range of capabilities.

• Maintainability, to enable maintenance tasks to be completed effectively, efficiently and with satisfaction.

Usability Measures. The current ISO 9241-11 focuses on specification and measurement of usability, and includes detailed information on usability measures and specification of usability. This information has not been included in the new draft. It is possible that it might be included in a new related standard on usability measurement.

3.3 Additional Proposed Changes

Some of the other aspects of usability that have been proposed as relevant, but that have not yet been included are explained below.

System and User Outcomes. One of the issues raised in the feedback was that it is sometimes necessary to take account of both the objective system outcome (e.g. whether an ecommerce transaction has been completed) and the subjective user outcome (e.g. whether the user believes that the transaction is complete). Both types of goals may need to be achieved for a successful outcome that can be regarded as effective.

An additional distinction could be made in the standard between goals for system outcomes resulting from interaction (such as information provided to the user, completing a purchase or casting a vote), and the outcomes for the user such as acquiring knowledge or making a decision based on the output of a system, product or service.

Evolving Goals. The user’s goals can evolve during interaction, particularly when the user is exploring use of a product. So a product may have low usability for the initially intended outcomes, but be quite usable for the final outcomes achieved. Conversely, with new goals the user may find that the level of usability is lower for these goals. Usability can be considered in relation to the user’s goals at any stage during the time that the user interacts with the product, system or service.

Social Responsibility. One critique of ISO 9241-11 is that it ignores social responsibility (for which there is now a standard: ISO 26000). A clear distinction needs to be made between considering usability for the user’s intended outcomes and for another stakeholder’s intended outcomes. Taking account of the user’s goals satisfies fundamental human needs and produces designs that respects human dignity.

The new draft defines accessibility as usability for people with the widest range of capabilities. To respect social responsibility, systems, products and services should be designed to be usable by people with the widest range of capabilities who could potentially use the system, product or service.

3.4 Related Concepts

In addition to elaborating on the understanding of usability, the revised version of ISO 9241-11 provides an explanation of the relationship between usability and some associated concepts including:

Human-Centred Quality. The objectives of human-centred design are identified in ISO CD 9241-220:2015 “Processes for enabling, executing and assessing human-centred design within organizations” as achievement of human-centred quality, which is composed of usability, accessibility, user experience and risk reduction. While usability as described in the revised ISO 9241-11 takes account of aspects of accessibility, user experience and the risks that could arise from poor usability, use of the concept human-centred quality makes each of these explicit and independent objectives.

Accessibility. ISO 26800 emphasises the importance of widening the target population to take account of the range and diversity of human characteristics, thus making products, systems, services, environments and facilities more accessible to more people. ISO 9241-11 interprets accessibility as usability for people with the widest range of capabilities, which is applied in the same way as usability. This provides a basis for specifying and evaluating accessibility in terms of effectiveness, efficiency and satisfaction for a wider range of user capabilities.

User Experience. User experience focuses on the user’s preferences, perceptions, emotions and physical and psychological responses that occur before, during and after use, rather than the observed effectiveness and efficiency. While usability typically deals with goals shared by a user group, user experience is concerned with individual goals, which can include personal motivations including needs to acquire new knowledge and skills, to communicate personal identity and to provoke pleasant memories. User experience also puts emphasis on how the experience changes with repeated use.

One source of potential confusion is the increasingly widespread use of the term user experience to refer to an overall view of all aspects of the user’s interaction with a system, product or service, rather than the original meaning that emphasized the importance of emotional experience. This use of the term user experience is closer to the concept of usability in the revised version of 9241-11, which explicitly includes the personal factors for individuals.

If you would like to contribute to the development of ISO 9241-11, or to comment on drafts, you can either do this via your national standards body [6], or if you are a member of one of the ISO TC159/SC4 liaison organisations [7] such as UXPA [21] you can participate through the liaison organisation.