More Web Proxy on the site http://driver.im/

research-article

A survey of software learnability: metrics, methodologies and guidelines

Authors:

George Fitzmaurice,

Ramtin AttarAuthors Info & Claims

CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 649 - 658

https://doi.org/10.1145/1518701.1518803

Published: 04 April 2009 Publication History

Abstract

It is well-accepted that learnability is an important aspect of usability, yet there is little agreement as to how learnability should be defined, measured, and evaluated. In this paper, we present a survey of the previous definitions, metrics, and evaluation methodologies which have been used for software learnability. Our survey of evaluation methodologies leads us to a new question-suggestion protocol, which, in a user study, was shown to expose a significantly higher number of learnability issues in comparison to a more traditional think-aloud protocol. Based on the issues identified in our study, we present a classification system of learnability issues, and demonstrate how these categories can lead to guidelines for addressing the associated challenges.

References

[1]

Abran, A., Khelifi, A., Suryn, W. and Seffah, A. (2003). Usability Meanings and Interpretations in ISO standards. Software Quality Journal. 11:325--338.

Digital Library

[2]

Baecker, R., Booth, K., Jovicic, S., McGrenere, J. and Moore, G. (2000). Reducing the gap between what users know and what they need to know. Proceedings on the 2000 conference on Universal Usability. 17--23.

Digital Library

[3]

Bevan, N. and Macleod, M. (1994). Usability measurement in context. Behaviour and Information Technology. 13:132--145.

[4]

Butler, K. A. (1985). Connecting theory and practice: a case study of achieving usability goals. ACM CHI. 85--88.

Digital Library

[5]

Carroll, J., Mack, R., Lewis, C., Grischkowsky, N. and Robertson, S. (1985). Exploring exploring a word processor. Human-Computer Inter. 1:283--307.

Digital Library

[6]

Carroll, J. M. (1990). The Nurnberg Funnel. MIT Press.

[7]

Carroll, J. M. and Carrithers, C. (1984). Training wheels in a user interface. Comm. ACM. 27(8):800--806.

Digital Library

[8]

Carroll, J. M. and Rosson, M. B. (1987). Paradox of the active user. Interfacing thought: cognitive aspects of human-computer interaction. MIT Press. 80--111.

Digital Library

[9]

Davis, S. and Wiedenbeck, S. (1998). The effect of interaction style and training method on end user learning of software packages. Interacting with Computers. 11(2):147--172.

[10]

Dix, A., Finlay, J. E., Abowd, G. D. and Beale, R. (2003). Human Computer Interaction. Prentice-Hall.

Digital Library

[11]

Dumas, J. F. and Redish, J. C. (1993). A Practical Guide to Usability Testing. Greenwood Publishing Group Inc.

Digital Library

[12]

Elliott, G. J., Jones, E. and Barker, P. (2002). A grounded theory approach to modeling learnability of hypermedia authoring tools. Interacting with Computers. 14(5):547--574.

[13]

Ericsson, K. A. and Simon, H. A. (1984). Protocol Analysis: Verbal Reports as Data. MIT Press.

[14]

Franzke, M. (1995). Turning research into practice: characteristics of display-based interaction. ACM CHI. 421--428.

Digital Library

[15]

Gilb, T. (1996). Level 6: Why we can't get there from here. IEEE Software. 13(1):97--98, 103.

Digital Library

[16]

Grossman, T., Dragicevic, P. and Balakrishnan, R. (2007). Strategies for accelerating on-line learning of hotkeys. ACM CHI. 1591--1600.

Digital Library

[17]

Haramundanis, K. (2001). Learnability in information design. ACM SIGDOC. 7--11.

Digital Library

[18]

Holzinger, A. (2005). Usability engineering methods for software developers. Comm. of ACM. 48(1):71--74.

Digital Library

[19]

Howes, A. and Young, R. M. (1991). Predicting the learnability of task-action mappings. ACM CHI. 1204--1209.

Digital Library

[20]

Hurst, A., Hudson, S. E. and Mankoff, J. (2007). Dynamic detection of novice vs. skilled use without a task model. ACM CHI. 271--280.

Digital Library

[21]

Jeffries, R., Miller, J. R., Wharton, C. and Uyeda, K. (1991). User interface evaluation in the real world: a comparison of four techniques. ACM CHI. 119--124.

Digital Library

[22]

Kato, T. (1986). What question-asking protocols can say about the user interface. Int. J. Man-Mach. Stud. 25(6):659--673.

Digital Library

[23]

Kelleher, C. and Pausch, R. (2005). Stencils-based tutorials: design and evaluation. ACM CHI. 541--550.

Digital Library

[24]

Kieras, D. E. and Polson, P. G. (1985). An approach to the formal analysis of user complexity. International Journal of Man-Machine Studies. 22:365--394.

[25]

Lazar, J., Jones, A. and Shneiderman, B. (2006). Workplace user frustration with computers: An exploratory investigation of the causes and severity. Behaviour and Info. Technology. 25(3):239--251.

[26]

Licklider, J. C. R. (1976). User-oriented interactive computer graphics. ACM UODIGS. 89--96.

Digital Library

[27]

Lin, H. X., Choong, Y. and Salvendy, G. (1997). A proposed index of usability: a method for comparing the relative usability of different software systems. Behaviour&Information Technology. 16:267--278.

[28]

Linja-aho, M. (2005). Evaluating and Improving the Learnability of a Building Modeling System. Helsinki University of Technology.

[29]

Mack, R. and Robinson, J. B. (1992). When novices elicit knowledge: question asking in designing, evaluating, and learning to use software. The psychology of expertise: cognitive research and empirical AI. Springer-Verlag, Inc. p. 245--268.

Digital Library

[30]

Mack, R. L., Lewis, C. H. and Carroll, J. M. (1983). Learning to use word processors: problems and prospects. ACM Trans. Inf. Syst. 1(3):254--271.

Digital Library

[31]

Mackay, W. E. (1991). Triggers and barriers to customizing software. ACM CHI. 153--160.

Digital Library

[32]

Michelsen, C. D., Dominick, W. D. and Urban, J. E. (1980). A methodology for the objective evaluation of the user/system interfaces of the MADAM system using software engineering principles. ACM Southeast Regional Conference. 103--109.

Digital Library

[33]

Nielsen, J. (1994). Usability Engineering. Morgan Kaufmann.

Digital Library

[34]

Palmiter, S. and Elkerton, J. (1991). An evaluation of animated demonstrations of learning computer-based tasks. ACM CHI. 257--263.

Digital Library

[35]

Paymans, T. F., Lindenberg, J. and Neerincx, M. (2004). Usability trade-offs for adaptive user interfaces: ease of use and learnability. International Conference on Intelligent User Interfaces. 301--303.

Digital Library

[36]

Rieman, J. (1996). A field study of exploratory learning strategies. ACM TOCHI. 3(3):189--218.

Digital Library

[37]

Santos, P. J. and Badre, A. N. (1994). Automatic chunk detection in human-computer interaction. AVI. 69--77.

Digital Library

[38]

Santos, P. J. and Badre, A. N. (1995). Discount learnability evaluation. GVU Technical Report GIT-GVU-95-30. Georgia Institute of Technology.

[39]

Shneiderman, B. (1983). Direct Manipulation: A Step Beyond Programming Languages. Comp. 16(8):57--69.

Digital Library

[40]

Shneiderman, B. (1997). Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison-Wesley Longman Publishing Co., Inc.

Digital Library

[41]

Stickel, C., Fink, J. and Holzinger, A. (2007). Enhancing Universal Access-EEG Based Learnability Assessment. Lecture Notes in Comp. Sci. 813--822.

[42]

Twidale, M. B. (2005). Over the Shoulder Learning: Supporting Brief Informal Learning. CSCW. 14(6):505--547.

Digital Library

[43]

Wixon, D. (2003). Evaluating usability methods: why the current literature fails the practitioner. Interactions. 10(4):28--34.

Digital Library

[44]

Xiao, J., Stasko, J. and Catrambone, R. (2004). An empirical study of the effect of agent competence on user performance and perception. Autonomous Agents and Multiagent Systems - Volume 1. 178--185.

Digital Library

Cited By

L’Yi Svan den Brandt AAdams ENguyen HGehlenborg N(2025)Learnable and Expressive Visualization Authoring Through Blended InterfacesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345659831:1(459-469)Online publication date: Jan-2025
https://doi.org/10.1109/TVCG.2024.3456598
Caro-Alvaro SGarcia-Lopez EBrun-Guajardo AGarcia-Cabot AMavri A(2024)Gesture-Based Interactions: Integrating Accelerometer and Gyroscope Sensors in the Use of Mobile AppsSensors10.3390/s2403100424:3(1004)Online publication date: 4-Feb-2024
https://doi.org/10.3390/s24031004
Deshmukh AChalmeta R(2024)User Experience and Usability of Voice User Interfaces: A Systematic Literature ReviewInformation10.3390/info1509057915:9(579)Online publication date: 19-Sep-2024
https://doi.org/10.3390/info15090579
Show More Cited By

Index Terms

A survey of software learnability: metrics, methodologies and guidelines
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

SLC: a visual cohesion metric to predict the usability of graphical user interfaces
SAC '15: Proceedings of the 30th Annual ACM Symposium on Applied Computing

User interfaces are evaluated through techniques that require end-users and usability experts. These techniques are costly and time consuming. Therefore, quantitative techniques such as predictive metrics can be considered an indispensable alternative ...
Improving the learnability of machine learning APIs by semi-automated API wrapping
ICSE-NIER '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results

A major hurdle for students and professional software developers who want to enter the world of machine learning (ML), is mastering not just the scientific background but also the available ML APIs. Therefore, we address the challenge of creating APIs ...
Learnability Assessment of Speech-Based Intelligent Personal Assistants by Older Adults
Human Aspects of IT for the Aged Population
Abstract
Older adults often have problems and difficulties with adopting technologies but the popularity of technology usage among older adults had increased significantly in recent years. Separately, Speech-Based Intelligent Personal Assistants (sIPA) or ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

April 2009

2426 pages

ISBN:9781605582467

DOI:10.1145/1518701

General Chairs:
Dan R. Olsen
Brigham Young University
,
Richard B. Arthur
Brigham Young University
,
Program Chairs:
Ken Hinckley
Microsoft Research
,
Meredith Ringel Morris
Microsoft Research
,
Scott Hudson
Carnegie Mellon University
,
Saul Greenberg
University of Calgary

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 April 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '09

Sponsor:

CHI '09: CHI Conference on Human Factors in Computing Systems

April 4 - 9, 2009

MA, Boston, USA

Acceptance Rates

CHI '09 Paper Acceptance Rate 277 of 1,130 submissions, 25%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

185
Total Citations
View Citations
4,588
Total Downloads

Downloads (Last 12 months)306
Downloads (Last 6 weeks)25

Reflects downloads up to 07 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

L’Yi Svan den Brandt AAdams ENguyen HGehlenborg N(2025)Learnable and Expressive Visualization Authoring Through Blended InterfacesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345659831:1(459-469)Online publication date: Jan-2025
https://doi.org/10.1109/TVCG.2024.3456598
Caro-Alvaro SGarcia-Lopez EBrun-Guajardo AGarcia-Cabot AMavri A(2024)Gesture-Based Interactions: Integrating Accelerometer and Gyroscope Sensors in the Use of Mobile AppsSensors10.3390/s2403100424:3(1004)Online publication date: 4-Feb-2024
https://doi.org/10.3390/s24031004
Deshmukh AChalmeta R(2024)User Experience and Usability of Voice User Interfaces: A Systematic Literature ReviewInformation10.3390/info1509057915:9(579)Online publication date: 19-Sep-2024
https://doi.org/10.3390/info15090579
Khurana AChilana PKostakos VKay JHoang T(2024)Understanding Novice Users' Mental Models of Gesture Discoverability and Designing Effective OnboardingCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678370(290-295)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678370
Perraud RTabard AMalacria S(2024)Tutorial mismatches: investigating the frictions due to interface differences when following software video tutorialsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661511(1942-1955)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661511
Khurana ASubramonyam HChilana P(2024)Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-SeekingProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645200(288-303)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645200
Baghestani ALatulipe CBunt A(2024)Older Adults' Collaborative Learning Dynamics When Exploring Feature-Rich SoftwareProceedings of the ACM on Human-Computer Interaction10.1145/36373788:CSCW1(1-27)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637378
Khurana AGlueck MChilana P(2024)Do I Just Tap My Headset?Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314517:4(1-28)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631451
Batbold TSoro ASchroeter R(2024)Mentorable Interfaces for Automated Vehicles: A New Paradigm for Designing Learnable Technology for Older AdultsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642390(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642390
Murphy-Hill EElizondo AMurillo AHarbach MVasilescu BCarlson DDessloch FRoychoudhury APaiva AAbreu RStorey M(2024)GenderMag Improves Discoverability in the Field, Especially for WomenProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639097(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639097
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents