More Web Proxy on the site http://driver.im/

Article

Free access

A mathematical model of the finding of usability problems

Authors:

Thomas K. LandauerAuthors Info & Claims

CHI '93: Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems

Pages 206 - 213

https://doi.org/10.1145/169059.169166

Published: 01 May 1993 Publication History

Abstract

For 11 studies, we find that the detection of usability problems as a function of number of users tested or heuristic evaluators employed is well modeled as a Poisson process. The model can be used to plan the amount of evaluation required to achieve desired levels of thoroughness or benefits. Results of early tests can provide estimates of the number of problems left to be found and the number of additional evaluations needed to find a given fraction. With quantitative evaluation costs and detection values, the model can estimate the numbers of evaluations at which optimal cost/benefit ratios are obtained and at which marginal utility vanishes. For a “medium” example, we estimate that 16 evaluations would be worth their cost, with maximum benefit/cost ratio at four.

References

[1]

Dalal, S.R., and Mallows, C.L. (1988). When should one stop testing software? J. American Statistical Association 83, 403 (September), 872-879.

[2]

Dalai, S.R., and Mallows, C.L. (1990). Some graphical aids for deciding when to stop testing software. IEEE J. Selected Areas in Communication 8, 2 (February), 169- 175.

[3]

Erhan, S. (1975). Introduction to Stochastic Processes. Prentice Hall, Englewood Cliffs, NJ. p. 87.

[4]

Gray, W.D., John, B.E., and Atwood, M.E. (1992). The precis of project Ernestine, or, an overview of a validation of GOMS. Proc. ACM CHI'92 (Monterey, CA, 3-7 May), 307-312.

Digital Library

[5]

Jeffries, R., Miller, J.R., Wharton, C., and Uyeda, K.M. (1991 ). User interface evaluation in the real world: A comparison of four techniques. Proc. ACM CHI'91 (New Orleans, LA, 27 April-2 May), 119-124.

Digital Library

[6]

Jq~rgensen, A.H. (1989). Using the thinking-aloud method in system development. In Salvendy, G., and Smith, M.J. (Eds.), Designing and Using Human-Computer Interfaces and Knowledge Based Systems. Amsterdam: Elsevier Science Publishers, 743-750.

Digital Library

[7]

Karat, C., Campbell, R., Fiegel, T. (1992). Comparisons of empirical testing and walkthrough methods in user interface evaluation. Proc. ACM CHI'92 (Monterey, CA, 3-7 May), 397-404.

Digital Library

[8]

Lewis, C. (1982). Using the 'thinking-aloud' method in cognitive interface design. Research Report RC-9265, IBM T.J. Watson Research Center, Yorktown Heights, NY.

[9]

Lewis, J.R., Henry, S.C., and Mack, R.L. (I 990). Integrated office software benchmarks: A case study. Proc. INTERACT'90 3rd IFIP Conf. Human-Computer Interaction (Cambridge, U.K., 27-31 August 1990), 337-343.

Digital Library

[10]

Mantei, M.M., and Teorey, T.J. (1988). Cost/benefit analysis for incorporating human factors in the software lifecycle. Communications of the ACM 31, 4 (April), 428-439.

Digital Library

[11]

Molich, R., and Nielsen, J. (1990). Improving a humancomputer dialogue. Communications of the A CM 33, 3 (March), 338-348.

Digital Library

[12]

Nielsen, J. (1989). Usability engineering at a discount. In Salvendy, G., and Smith, M.J. (Eds.), Designing and Using Human-Computer Interfaces and Knowledge Based Systems, Elsevier Science Publishers, Amsterdam. 394-401.

Digital Library

[13]

Nielsen, J. (1990). Big paybacks from 'discount' usability engineering. IEEE Software 7, 3 (May), 107-108.

Digital Library

[14]

Nielsen, J. (1992). Evaluating the thinking aloud technique for use by computer scientists. In Hartson, H.R., and Hix, D. (Eds.), Advances in Human--Computer Interaction Vol. 3, Ablex. 69-82.

Digital Library

[15]

Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Proc. ACM CHI'92 (Monterey, CA, 3-7 May), 373-380.

Digital Library

[16]

Nielsen, J. (1992). The usability engineering lifecycle. IEEE Computer 25, 3 (March), 12-22.

Digital Library

[17]

Nielsen, J. (1993). Estimating the number of subjects needed for a thinking aloud test. Intl. J. Man-Machine Studies in press.

[18]

Nielsen, J. (1993). Usability Engineering, Academic Press, San Diego, CA.

[19]

Nielsen, J. (1993). Iterative design of user interfaces. IEEE Computer 26 (to appear, probably in the July issue).

Digital Library

[20]

Nielsen, J. (1993). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods. Book in preparation.

Digital Library

[21]

Nielsen, J., and Molich, R. (1990). Heuristic evaluation of user interfaces. Proc. ACM CHI'90 (Seattle, WA, 1-5 April), 249-256. "

Digital Library

[22]

Virzi, R.A. (1990). Streamlining the design process: Running fewer subjects. Proceedings of the Human Factors Society 34th Annual Meeting (Orlando, FL, 8-12 October), 291-294.

[23]

Virzi, R.A. (1992). Refining the test phase of usability evaluation: How many subjects are enough? Human Factors 34, 4 (August), 457-468.

Digital Library

[24]

Wright, P.C., and Monk, A.F. (1991). A cost-effective evaluation method for use by designers. Intl. J. Man- Machine Studies 35, 6 (December), 891-912.

Digital Library

Cited By

Kitromili SHargood CSkains R(2025)Interactive digital narratives for mental resilience: Understanding the player experience of betwixtEntertainment Computing10.1016/j.entcom.2024.10091652(100916)Online publication date: Jan-2025
https://doi.org/10.1016/j.entcom.2024.100916
Barz NBenick MDörrenbächer-Ulrich LPerels F(2025)The evaluation of an educational game to promote pre-service teachers’ self-regulated learningEntertainment Computing10.1016/j.entcom.2024.10083652(100836)Online publication date: Jan-2025
https://doi.org/10.1016/j.entcom.2024.100836
Kreshanti PKekalih ARahyussalim ASupriadi SPriosoeryanto BNoviana DOley MSukasah C(2024)Usability testing of a novel interlocking three-dimensional miniplate for mandibular angle fracturesArchives of Craniofacial Surgery10.7181/acfs.2024.0029025:4(171-178)Online publication date: 20-Aug-2024
https://doi.org/10.7181/acfs.2024.00290
Show More Cited By

Index Terms

A mathematical model of the finding of usability problems
1. Computing methodologies
  1. Modeling and simulation
    1. Simulation types and techniques
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Estimating the relative usability of two interfaces: heuristic, formal, and empirical methods compared
CHI '93: Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems

Two alternative user interface designs were subjected to user testing to measure user performance in a database query task. User performance was also estimated heuristically in three different ways and by use of formal GOMS modelling. The estimated ...
Finding usability problems through heuristic evaluation
CHI '92: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Usability specialists were better than non-specialists at performing heuristic evaluation, and “double experts” with specific expertise in the kind of interface being evaluated performed even better. Major usability problems have a higher probability ...
Usability in practice: formative usability evaluations - evolution and revolution
CHI EA '02: CHI '02 Extended Abstracts on Human Factors in Computing Systems
Formative evaluation is a collection of "find-and-fix" usability engineering methods, focused on identifying usability problems before a product is completed. In this forum, four experienced usability professionals will address different aspects of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '93: Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems

May 1993

547 pages

ISBN:0897915755

DOI:10.1145/169059

Chairmen:
Bert Arnold
Delft Univ. of Technology, The Netherlands
,
Gerrit van der Veer
Univ. of Twente and Vrije Univ., The Netherlands
,
Ted White
Univ. of Twente and Univ. of Ultrecht, Utrecht, The Netherlands

Copyright © 1993 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCAPH: ACM SIGCAPH Computers and the Physically Handicapped
Austrian Comp Soc: Austrian Computer Society
SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques
Human Factors Soc: Human Factors Society
SIGCHI: ACM Special Interest Group on Computer-Human Interaction
SIGGROUP: ACM Special Interest Group on Supporting Group Work
IEEE-CS: Computer Society
IFIP: International Federation for Information Processing
NGI: Dutch Computer Soc - Nederlands Genoostschapvoor Informatica

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 May 1993

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

INTERCHI93

Sponsor:

SIGCAPH
Austrian Comp Soc
SIGGRAPH
Human Factors Soc
SIGCHI
SIGGROUP
IEEE-CS
IFIP
NGI

INTERCHI93: Conference on Human Factors in Computing

April 24 - 29, 1993

Amsterdam, The Netherlands

Acceptance Rates

CHI '93 Paper Acceptance Rate 62 of 330 submissions, 19%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

898
Total Citations
View Citations
22,228
Total Downloads

Downloads (Last 12 months)4,293
Downloads (Last 6 weeks)566

Reflects downloads up to 20 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kitromili SHargood CSkains R(2025)Interactive digital narratives for mental resilience: Understanding the player experience of betwixtEntertainment Computing10.1016/j.entcom.2024.10091652(100916)Online publication date: Jan-2025
https://doi.org/10.1016/j.entcom.2024.100916
Barz NBenick MDörrenbächer-Ulrich LPerels F(2025)The evaluation of an educational game to promote pre-service teachers’ self-regulated learningEntertainment Computing10.1016/j.entcom.2024.10083652(100836)Online publication date: Jan-2025
https://doi.org/10.1016/j.entcom.2024.100836
Kreshanti PKekalih ARahyussalim ASupriadi SPriosoeryanto BNoviana DOley MSukasah C(2024)Usability testing of a novel interlocking three-dimensional miniplate for mandibular angle fracturesArchives of Craniofacial Surgery10.7181/acfs.2024.0029025:4(171-178)Online publication date: 20-Aug-2024
https://doi.org/10.7181/acfs.2024.00290
Oliveira RLins FCorreia Neto J(2024)Avaliação de Usabilidade de Jogos Educacionais para pessoas com a Síndrome de Williams-BeurenRevista Brasileira de Informática na Educação10.5753/rbie.2024.245232(181-212)Online publication date: 7-Apr-2024
https://doi.org/10.5753/rbie.2024.2452
Schuh ACabral ADewes M(2024)Avaliação da inovação em projetos de pesquisa: proposta do artefato Gerenciador de Inovação em Projetos de Pesquisa para Instituições de Ciência e Tecnologia (GIPPICT)International Journal of Innovation10.5585/2024.2609312:4(e26093)Online publication date: 17-Dec-2024
https://doi.org/10.5585/2024.26093
Şekerli TTuker C(2024)Grafik Tasarım Pratiğinde Üretken Yapay Zekânın Etkinliği Üzerine Nitel Bir AraştırmaJournal of Computational Design10.53710/jcode.14201465:1(105-138)Online publication date: 31-Mar-2024
https://doi.org/10.53710/jcode.1420146
Seesukong SAngskun TKeandoungchun NThippongtorn AAngskun J(2024)CARIAInternational Journal of Information and Communication Technology Education10.4018/IJICTE.35649920:1(1-24)Online publication date: 7-Nov-2024
https://dl.acm.org/doi/10.4018/IJICTE.356499
Marin-Alvarez ECarcelen-Garcia SGalmes-Cerezo M(2024)A Holistic and Multidimensional Methodology Proposal for a Persona with Total Visual Impairment Archetype on the WebSocieties10.3390/soc1407010214:7(102)Online publication date: 27-Jun-2024
https://doi.org/10.3390/soc14070102
Delnevo GGhini VFiumana EMirri S(2024)A Support Tool for Emergency Management in Smart Campuses: Reference Architecture and Enhanced Web User InterfacesSensors10.3390/s2418588724:18(5887)Online publication date: 11-Sep-2024
https://doi.org/10.3390/s24185887
Shen YWang SShen YTan SDong YQin WZhuang Y(2024)Evaluating the Usability of mHealth Apps: An Evaluation Model Based on Task Analysis Methods and Eye Movement DataHealthcare10.3390/healthcare1213131012:13(1310)Online publication date: 30-Jun-2024
https://doi.org/10.3390/healthcare12131310
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents