A Set of Usability Heuristics for Mobile Applications

The market for mobile devices has grown year after year, and over the years there has also been hardware evolution of these devices, the complexity of their operating systems and applications [1]. Another type of change that has been happening since the evolution of cell phones to smartphones is in the form of their use, which was previously basically limited to telephone calls, currently comprises a much wider range of uses by the user, such as listening music, make bank transfers, make online purchases, among others [2].

With the change in the use of the mobile phone by the user, new challenges and peculiarities appeared and such factors should be considered and studied for the development of a mobile application, such as software usability in this context [3].

Usability can be defined as being “a broad concept that basically refers to how easy it is for users to learn a system, how efficient they can be once they have already learned it, and how enjoyable it is to use it” [4], i.e., usability is understood to be the ability to use a product with effectiveness, efficiency and satisfaction in a specific context of use [5].

Usability is a very important requirement of design, being of utmost importance to users in the decision to purchase a product [6]. The “IS Success Model” shows the point of view of the product, how a system is related to user satisfaction. It can be inferred why it is an important factor to be considered a process of development of a product and/or software, so it is important to know how to apply it is to evaluate it. Through the evaluation by usability heuristics it is possible to identify usability problems and thus to evaluate a software as to its usability [6].

Usability heuristic evaluations describe design/usability principles that serve to evaluate a particular software, called heuristics, and this evaluation is performed largely by usability experts or by ordinary users, but the latter being less indicated [7, 8]. The evaluation of usability by heuristics has been widely studied and is one of the most used methods to evaluate the quality of a software, being considered in the literature as a traditional evaluation of software usability [9, 10].

The set of the ten heuristics proposed by Nielsen [11] are classic of the literature and reveal principles for the construction of a software interface in order to have a good usability. In the context of mobile applications, new factors to be considered have arisen in relation to human-computer interaction and should be taken into account in the design and development of a software application that aims to have a good usability, as well as a new set of usability heuristics take such changes into account.

This work carried out a Systematic Literature Review (SLR) with the objective of identifying the usability heuristics for the mobile context proposed in the literature and also identified the main metrics used in heuristic evaluations of a mobile application. Based on the results of the SLR, the scope of this work consists of the proposal of a new set of usability heuristics specific to the context of applications for touch-screen smartphone that takes into account the user, the context and the tasks as usability factors [12] and Cognitive Load, as an important usability attribute, so that each heuristic contains a detailed description to facilitate its understanding.

This work did not propose specific heuristics for a given context of mobile application use, such as the one proposed by Ajibola and Goosen [13] focused in the context of e-commerce, but will focus on general heuristics in the context of mobile applications for smartphones touch-screen, according to the work developed by Salvucci [14], since more general heuristics for evaluating interfaces generally become easier to understand and apply [15].

The general objective of this work is to propose a set of usability heuristics, focused on the context of mobile applications, detailing them and also identifying the main metrics used during heuristic evaluations. The main contribution of this work will be to propose a model that will contain a set of specific usability heuristics for mobile applications, in the context of touch-screen smartphones, which consider the user, context and tasks to be fulfilled in the application as usability factors and Cognitive Load as an important usability attribute.

This paper is organized as follows. The Sect. 2 presents the theoretical basis necessary for the understanding of this work. Furthermore, related work is presented. Section 3 presents the systematic literature review and the results obtained from it. The Sect. 4 presents the set of proposed heuristics to carry out the evaluation of Mobile Applications. The Sect. 5 presents the considerations of this work, learning and future work.

According to the literature the term usability has several definitions. The International Organization for Standardization (ISO) together with the International Electrotechnical Commission (IEC) define usability in ISO/IEC 9241-11 [5], such as: “the extent to which a product can be used by specific users to achieve (the accuracy and completeness with which users achieve the specified goals), efficiency (the resources expended in relation to the accuracy and completeness with which users reach goals), and satisfaction (comfort and acceptability of use) in a specified context of use” [5, 16].

There are standards that define what is important to be considered in terms of usability when the goal is software quality during your development process. ISO/IEC 9126-1 [17] describes six categories of software quality that are relevant in the software development process, among which is usability basically defined as ease of use [17]. However, ISO/IEC 14598 [18] provides a framework for the use of the ISO/IEC 9126-1 model as a way to evaluate software products [18].

ISO/IEC 25000 [19] is a series of standards that came to replace and extend ISO/IEC 9126 [17] and ISO/IEC 14598 [18], with the main objective of organizing, improve and unify concepts related to two major software development processes: specification of software quality requirements and software quality assessment, which is performed in conjunction with the software quality measurement process [19]. Usability is considered in every standard and is specifically mentioned also in DTR 25060 (Common Industry Format (CIF) for Usability) and ISO 25062: 2006 [20] (Common Industry Format (CIF) for usability testing reports) [9].

2.1 Related Work

In the work presented by Miranda [21] a set of 16 focused heuristics for mobile devices is described. Furthermore to the proposed set of heuristics, the work performs some heuristic evaluations in different mobile applications, which are characterized by being of different categories and because they contain different functionalities for validation of the proposal: CNN, Amazon, TripAdvisor, Ebook Reader, Calendar, QR Code Scanner, Dropbox, Dictionary and Skype. The applications were tested on different mobile devices so that the greatest number of possible errors were discovered, to cover more than one mobile platform, so the study used the following devices: Smartphone; Samsung Galaxy S4: running the Android system; iPad: running the iOS system; HTC Titan: running Windows Phone OS.

Miranda [21] concludes that with the popularization of mobile devices, such as smartphones, good usability in a mobile software application is a feature that distinguishes a successful software solution from others and that seek a usability of excellence in an application should be something to consider during development. The work presented by Miranda [21] reinforces that heuristic evaluation is an adequate method to evaluate the usability of mobile applications, and that the set of heuristics proposed in his research can be improved so that more usability errors can be through this method of evaluation.

In the paper presented by Harrison et al. [12] a usability model called PACMAD (People at the Center of Mobile Application Development) is proposed which addresses the limitations that the author believes exist in other usability models when applied to mobile devices, thus PACMAD brings together important attributes of other usability models and is characterized by being more comprehensive.

Harrison et al. [12] compare their model with the usability models proposed by ISO 9241 [22] and Nielsen [4]. PACMAD incorporates the attributes of both models and adds Cognitive Load as a usability attribute for mobile applications. Furthermore, PACMAD proposes three usability factors, User, Task and Context, which the author argues are important when developing a mobile application, as it may impact the final interface of the system.

The related works identified and reported in this paper present usability guidelines and heuristics for mobile applications, focusing on the user and the tasks that the user will perform when using a particular application. However, according to Harrison et al. [12], there are few usability works that consider context as a usability factor and the author argues that there may be a gap in the literature about this subject. Therefore, the present work seeks to propose a set of usability heuristics that consider the user, the task and the context, as usability factors. Furthermore, it seeks to contribute by minimizing the existence of this gap on the subject in the literature.

In this work the Systematic Literature Review (SLR) is used. The SLR is a framework that aims to provide a way to identify, analyze and interpret relevant research for a particular research question, area of knowledge or phenomenon of interest [23]. The studies that contribute to answer the research questions of a systematic review are called primary studies [23]. During the SLR, the Planning, Conduct and Publication of Results phases were followed, as defined in the work presented by Kitchenham [23], together with the Manual Search and the Snowballing. Table 1 presents the SLR Research Questions:

The search strategy involved the use of Automatic Search, which consists of the search through a search string in digital databases [24], followed by Manual Search, through which searches for papers in Conference proceedings, Journals or specific Magazines [25]. Furthermore, Snowballing was applied [26].

The Automatic Search was performed in 5 databases, selected for having a considerable volume of papers published in periodicals and conferences of the area of knowledge in usability, the focus of this SLR, being:

• Biblioteca Digital ACM;

• IEEEXplore;

• Science Direct;

• Scopus;

• Springer.

The Manual Search was carried out by analyzing the titles and abstracts (if necessary) of studies published in Conferences Annals and Journals, dealing with Human-Computer Iteration. The studies considered potentially relevant were added to the set of selected papers.

The papers selected by the search string in the databases can present results with some limitations, either by the lack of keywords or synonyms in the String, or by the non-selection of a database that could return important works of the area in question or even the way the String was defined can affect the results obtained in the conduction of an SLR [27]. In order to minimize the loss of important works, it was decided to use Snowballing’s set of instructions, proposed by Wohlin and Prikladniki [27], which basically consists of reviewing the bibliographic references of the selected articles, automatic and manual search, with the objective of selecting more works related to the research area.

Selection criteria were defined to include and exclude a primary study in our study object, according to the adopted research strategy. Thus, the inclusion and exclusion criteria were defined to select the most relevant papers in relation to the research questions to be answered.

3.1 SLR Results

When applying the automatic search strategy adopted in the selected databases, from a total of 31 papers returned from the Search String, after reading the title, abstract and keywords were selected 15 papers and excluded 16 (Fig. 1). Subsequently, the following steps of the adopted research protocol were carried out, resulting in the selection of 6 papers to answer the research questions and the exclusion of 9 papers. Thus, all steps of the adopted research protocol were performed and resulted in the complete reading of the 6 primary studies. After the selection of the papers by the automatic search strategy, 1 paper was selected from the manual search and 1 through Snowballing, totalizing 8 primary studies for the extraction of data through the systematic review of literature (Fig. 1).

Result of the papers selection.

Full size image

The extraction of the information to compose the SLR result occurred through the complete reading of the 8 selected papers. From the complete reading of the primary studies it was possible to elaborate the answers to the research questions defined in this study.

RQ1 - Which Heuristics Are Used in the Context of Mobile Applications to Evaluate the Quality of the Product? The paper presented by Neto and Pimentel [28], proposes a set of eleven heuristics of usability focused specifically for the mobile context, presents a comparison with the ten heuristics of Nielsen [11]. This comparison is a common thing to do since Nielsen’s work is a benchmark in the area of usability in general. The objective of this paper is to compare the proposed heuristics with those of Nielsen in a practical study where the evaluators use the two models for future comparison of the final number of usability errors coming from both models. As a result, the model proposed by Neto and Pimentel [28] enabled the evaluators to find more interface usability errors than the Nielsen model [29].

Inostroza et al. [9] and [30] propose a set of twelve general heuristics for touchscreen-based devices. The set of proposed heuristics were refined from an evaluation with usability specialists divided into two groups, one group used the set of Nielsen heuristics and the other group used the one proposed by the author in the evaluation of some applications. In the end it was concluded that the model proposed by Inostroza et al. [9, 30] captured more usability problems compared to the model proposed by Nielsen [29].

Humayoun et al. [31] proposes a set of 15 heuristics focused on mobile applications that use multi-touch gestures. Based on the heuristic evaluation conducted by the author, he concluded that through the proposed set of heuristics, the evaluators were able to find more usability problems than other heuristic proposals also focused on mobile applications, such as in the paper of Joyce and Lilley [32], which also proposes a set of heuristics focused on the mobile context.

The work proposed by Billi et al. [33], presents a set of eight general usability heuristics focused on the mobile context. The author states that traditional heuristics, such as Nielsen [11], do not deal with context switching and therefore new heuristics are required for better results in a heuristic evaluation for mobile applications.

There are some works that defend a set of heuristics for the mobile context more focused in some specific domain, according to the work presented by Ajibola and Goosen [13]. This paper presents a proposal of eleven heuristics, based on the heuristics of Nielsen [11], but containing some more focused on the context of m-commerce. The paper presents a revised proposal of heuristics as they have been re-evaluated with domain experts to improve and validate their proposal.

The work presented by Silva et al. [34], proposes a set of thirty-three usability heuristics (evolution of the work [35] of the author) for the mobile context, focused on elderly users (senior citizens). The study presents the heuristics grouped as follows: 1. Perception: these are heuristics related to the limitations of perception that the older user tends to suffer, such as: visual and auditory alterations; 2. Cognition: these are heuristics that refer to the cognitive changes that can occur with advancing age, such as the difficulty of maintaining attention or managing a large number of items through working memory; 3. Skill: these are heuristics related to the difficulty in accomplishing tasks due to the limitations of the user’s motor skills; 4. Navigation: these are heuristics directed to the understanding of the structure of the application and of how the user can use that application based on this structure; 5. Content: these are heuristics related to the information and language used in the application; 6. Visual Design: these are heuristics that address design details, for example, formatting details and visual representations.

RQ2 - What Are the Usability Heuristics Used, in the Context of Mobile Applications, that Consider Usability Factors: User (its Characteristics), Task (user Goal to Be Achieved in the Application Use) and Application Usage Context? In general, all the works that propose heuristics of usability seek to highlight the usability problems of a software application and, based on this, to determine if such application has good usability and is easy to use for users in general or for those with specific characteristics. Thus, there are papers that propose heuristics that evidence the usability factors proposed by Harrison et al. [12].

The first usability factor (user) is evidenced in all works that propose usability heuristics, since their purpose is to represent general principles of usability to be applied in a software interface, and based on that, the interface will be easy and intuitive for the largest number of users with the most diverse characteristics. When the target audience of an application has more unique characteristics, such as some physical or mental limitation, the work is conducted with a focus on these more specific characteristics of the users. The paper presented by Silva et al. [34] suggests a set of focused heuristics for elderly users that usually have certain special characteristics that, according to the author, may be psychosocial changes and functional disorders that affect vision, hearing, movement, cognition and their relationship to themselves and others around them. Thus, the heuristics that evidence the usability factor “user” in the author’s work are: 1. Heuristics 2 and 3: Older users tend to be slower at performing tasks overall; 2. Heuristic 10: The characteristics of the target audience should be taken into account in the language used in the application; 3. Heuristics 13, 19, 20, 23, 24, 25, 26 and 27: Older users tend to have vision problems; 4. Heuristic 22: Older users tend to have hearing problems.

The second factor of usability (task) is evidenced in all works that propose a set of usability heuristics, since the main goal of the heuristics is to present general principles of usability, when applied in a particular software, if the final result is the best the user is able to perform their tasks and achieve their goals in an easy and intuitive way when using software. This statement comes in line with the definition of usability given by Shackel and Richardson [36]. Thus, all the heuristics presented by Inostroza et al. [9], for example, evidence the usability factor (task) proposed by Harrison [12], due to the fact that all heuristics are geared towards maximizing the ease to achieve their goals in the most intuitive way possible.

The third factor of usability (context) is exactly the factor that Harrison [12] mentions as being a gap in the literature of works related to software usability. Thus, during the execution of this systematic literature review, no work was found containing context-oriented heuristics, reinforcing Harrison’s assertion [12].

RQ3 - What Are the Metrics Used in a Heuristic Evaluation in the Context of Mobile Applications? Gómez et al. [37] used a metric to prioritize the relevance of heuristic items to the specific interface evaluated. In this way, the experts prioritized heuristics from 1 to 4, based on the application of evaluated software, being: 1 - for completed heuristic items, 2 - for those corresponding to usability gaps, 3 - for heuristic items that were not evaluated in the current phase of the software life cycle and 4 - for issues not applicable to the interface.

Inostroza et al. [30] conducted a study for the evaluation of their proposed heuristics, comparing them with the Nielsen heuristics [11], causing two distinct groups of evaluators to evaluate a mobile application under egalitarian conditions. Inostroza et al. [30] used a metric that consisted in evaluating the severity of usability problems related to a given heuristic using a severity scale from 0 (low) to 4 (high).

Billi et al. [33] carried out in their work a heuristic evaluation divided into three stages: pre-evaluation, individual evaluation and consolidation of individual findings. In the pre-evaluation phase the evaluators sign a consent form and a demographic questionnaire is given for the heuristic evaluation, as well as the instructions necessary for the evaluators to familiarize themselves with the set of mobile heuristics proposed by the author. In the individual evaluation phase, the evaluators sought to identify and prioritize usability problems based on the proposed heuristics. In the consolidation phase of the individual findings, the evaluators after completing the previous phase met to discuss the findings with the other evaluators. In the heuristic evaluation conducted, a metric proposed by Nielsen [38] was used to prioritize usability problems, which consists in evaluating the usability problem found for its severity on a scale of 0 to 4, being: 0 for no problems encountered, 1 for aesthetic problems found, 2 for minor usability problems found, 3 for found usability problems that need to be fixed with a high priority and 4 for extremely urgent usability problems, and must be repaired before the product is released to end users.

Humayoun et al. [31], conducted a heuristic evaluation with five expert evaluators in the field of computer science. The evaluation was conducted so that the evaluators were given a small training of 30 to 60 min to become familiar with the method. Thus, the evaluation scenarios were given to the evaluators and later the actual heuristic evaluation was performed, the author describes that the Likert scale metric was used [39] (Ajibola and Goosen [13] also use this the same metric in his work) to classify the heuristics from 1 to 5 as to their usefulness during the evaluation, being: 1 - strongly disagree, 2 - disagree, 3 - neutral, 4 - agree and 5 - strongly agree.

According to the results obtained with the SLR conducted it is proposed a set of heuristics for the evaluation of the usability of mobile applications. The Fig. 2 presents an overview of the proposed set. Each heuristic is structured with its respective ID; Name; Definition; Explanation; Primary Studies that justify its use; Benefits associated with the use of heuristics and; Problems associated with misinterpretation.

Disposition of usability factors by usability heuristics.

Full size image

Usability Heuristics Set found are:

• UH1 - Visibility of System Status. The application should keep the user informed about all processes and state changes within a reasonable period of time.

• UH2 - Correspondence between the Application and the Real World. The application must speak the language of the users and not in technical terms of the system. The application must follow the conventions of the real world and display the information in a logical and natural order.

• UH3 - User Control and Freedom. The application should allow the user to undo and redo their actions for clear navigation and should provide the user with an option to exit undesirable system states.

• UH4 - Consistency and Standards. The application must follow the established conventions, allowing the user to perform their tasks in familiar, standardized and consistent manner.

• UH5 - Error Prevention. Eliminate error prone conditions and give the user a confirmation option with additional information before committing to the action.

• UH6 - Minimize the User’s Memory Load. The application should provide visible objects, actions, and options to prevent users from having to memorize information from one interface to another.

• UH7 - Customization and Shortcuts. The application should provide basic and advanced settings for setting and customizing shortcuts for frequent actions.

• UH8 - Efficiency of Use and Performance. The device must be able to load and display information in a reasonable amount of time and minimize the steps required to perform a task (number of steps to be taken by the user to reach a goal). Animations and transitions should display smoothly and smoothly.

• UH9 - Aesthetic and Minimalist Design. The application should avoid displaying unwanted information that overwhelms the screen.

• UH10 - Help Users Recognize, Diagnose, and Recover from Errors. The application should display error messages in a language familiar to the user, accurately indicating the problem and suggesting a constructive solution.

• UH11 - Help and Documentation. The application should provide easy-to-find documentation and help centering on the user’s current task and indicating concrete steps to follow.

• UH12 - Pleasant and Respectful Interaction with the User. The device should provide a nice iteration with the user so that the user does not feel uncomfortable while using the application.

• UH13 - Privacy. The application must protect the user’s sensitive data.

Due to the growth of the production of smartphones and associated with its evolution, usability is a key factor of differentiation for products and mobile applications and also a fundamental attribute for the quality of the product. Usability is a factor that facilitates the use of the software by the user, which can help in the user’s loyalty and also in his satisfaction in the use of a software application that presents a good usability.

As the usability heuristics proposed by Nielsen [11] were not developed with the focus of encompassing the mobile applications [28], it became necessary to identify and propose a new set of heuristics that focused on applications based on the mobile context, for example, the work proposed by Dourado and Canedo [26]. Thus, the present work proposes a set of usability heuristics that consider the user, the task and the context as usability factors. The proposal has a basis in the works identified through the systematic literature review (SLR) carried out in this work. Furthermore, the SLR allowed to answer the research questions that were proposed.

The main contribution of the present work is to propose a set of heuristics for the context of mobile applications. Furthermore, to highlight the usability factors proposed by Harrison et al. [12] and to include Cognitive Load as an important attribute of usability. During the conduction of the SLR the metrics that were used by the academy during an evaluation of usability heuristics were identified.

For future work, it is still necessary to validate the set of heuristics with specialists in the area of usability to refine and validate the proposal, as well as to perform an empirical validation of the present work by means of a heuristic evaluation of one or more mobile applications and by to carry out a full evaluation of the results.