More Web Proxy on the site http://driver.im/

Article

Free access

A mathematical model of the finding of usability problems

Authors:

Thomas K. LandauerAuthors Info & Claims

CHI '93: Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems

Pages 206 - 213

https://doi.org/10.1145/169059.169166

Published: 01 May 1993 Publication History

Abstract

For 11 studies, we find that the detection of usability problems as a function of number of users tested or heuristic evaluators employed is well modeled as a Poisson process. The model can be used to plan the amount of evaluation required to achieve desired levels of thoroughness or benefits. Results of early tests can provide estimates of the number of problems left to be found and the number of additional evaluations needed to find a given fraction. With quantitative evaluation costs and detection values, the model can estimate the numbers of evaluations at which optimal cost/benefit ratios are obtained and at which marginal utility vanishes. For a “medium” example, we estimate that 16 evaluations would be worth their cost, with maximum benefit/cost ratio at four.

References

[1]

Dalal, S.R., and Mallows, C.L. (1988). When should one stop testing software? J. American Statistical Association 83, 403 (September), 872-879.

[2]

Dalai, S.R., and Mallows, C.L. (1990). Some graphical aids for deciding when to stop testing software. IEEE J. Selected Areas in Communication 8, 2 (February), 169- 175.

[3]

Erhan, S. (1975). Introduction to Stochastic Processes. Prentice Hall, Englewood Cliffs, NJ. p. 87.

[4]

Gray, W.D., John, B.E., and Atwood, M.E. (1992). The precis of project Ernestine, or, an overview of a validation of GOMS. Proc. ACM CHI'92 (Monterey, CA, 3-7 May), 307-312.

Digital Library

[5]

Jeffries, R., Miller, J.R., Wharton, C., and Uyeda, K.M. (1991 ). User interface evaluation in the real world: A comparison of four techniques. Proc. ACM CHI'91 (New Orleans, LA, 27 April-2 May), 119-124.

Digital Library

[6]

Jq~rgensen, A.H. (1989). Using the thinking-aloud method in system development. In Salvendy, G., and Smith, M.J. (Eds.), Designing and Using Human-Computer Interfaces and Knowledge Based Systems. Amsterdam: Elsevier Science Publishers, 743-750.

Digital Library

[7]

Karat, C., Campbell, R., Fiegel, T. (1992). Comparisons of empirical testing and walkthrough methods in user interface evaluation. Proc. ACM CHI'92 (Monterey, CA, 3-7 May), 397-404.

Digital Library

[8]

Lewis, C. (1982). Using the 'thinking-aloud' method in cognitive interface design. Research Report RC-9265, IBM T.J. Watson Research Center, Yorktown Heights, NY.

[9]

Lewis, J.R., Henry, S.C., and Mack, R.L. (I 990). Integrated office software benchmarks: A case study. Proc. INTERACT'90 3rd IFIP Conf. Human-Computer Interaction (Cambridge, U.K., 27-31 August 1990), 337-343.

Digital Library

[10]

Mantei, M.M., and Teorey, T.J. (1988). Cost/benefit analysis for incorporating human factors in the software lifecycle. Communications of the ACM 31, 4 (April), 428-439.

Digital Library

[11]

Molich, R., and Nielsen, J. (1990). Improving a humancomputer dialogue. Communications of the A CM 33, 3 (March), 338-348.

Digital Library

[12]

Nielsen, J. (1989). Usability engineering at a discount. In Salvendy, G., and Smith, M.J. (Eds.), Designing and Using Human-Computer Interfaces and Knowledge Based Systems, Elsevier Science Publishers, Amsterdam. 394-401.

Digital Library

[13]

Nielsen, J. (1990). Big paybacks from 'discount' usability engineering. IEEE Software 7, 3 (May), 107-108.

Digital Library

[14]

Nielsen, J. (1992). Evaluating the thinking aloud technique for use by computer scientists. In Hartson, H.R., and Hix, D. (Eds.), Advances in Human--Computer Interaction Vol. 3, Ablex. 69-82.

Digital Library

[15]

Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Proc. ACM CHI'92 (Monterey, CA, 3-7 May), 373-380.

Digital Library

[16]

Nielsen, J. (1992). The usability engineering lifecycle. IEEE Computer 25, 3 (March), 12-22.

Digital Library

[17]

Nielsen, J. (1993). Estimating the number of subjects needed for a thinking aloud test. Intl. J. Man-Machine Studies in press.

[18]

Nielsen, J. (1993). Usability Engineering, Academic Press, San Diego, CA.

[19]

Nielsen, J. (1993). Iterative design of user interfaces. IEEE Computer 26 (to appear, probably in the July issue).

Digital Library

[20]

Nielsen, J. (1993). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods. Book in preparation.

Digital Library

[21]

Nielsen, J., and Molich, R. (1990). Heuristic evaluation of user interfaces. Proc. ACM CHI'90 (Seattle, WA, 1-5 April), 249-256. "

Digital Library

[22]

Virzi, R.A. (1990). Streamlining the design process: Running fewer subjects. Proceedings of the Human Factors Society 34th Annual Meeting (Orlando, FL, 8-12 October), 291-294.

[23]

Virzi, R.A. (1992). Refining the test phase of usability evaluation: How many subjects are enough? Human Factors 34, 4 (August), 457-468.

Digital Library

[24]

Wright, P.C., and Monk, A.F. (1991). A cost-effective evaluation method for use by designers. Intl. J. Man- Machine Studies 35, 6 (December), 891-912.

Digital Library

Cited By

Carrión-Toro MMorales-Martínez DSantórum MVizuete AMaldonado-Garcés VAcosta-Vargas P(2025)PictoAndes: A Customizable Communication Board for Inclusive Education and Multicultural AccessibilitySustainability10.3390/su1703095617:3(956)Online publication date: 24-Jan-2025
https://doi.org/10.3390/su17030956
Marelli DBianco SCiocca G(2025)Acquisition and Modeling of Material Appearance Using a Portable, Low Cost, DeviceSensors10.3390/s2504114325:4(1143)Online publication date: 13-Feb-2025
https://doi.org/10.3390/s25041143
Bobrova PPerego P(2025)The Development of User-Centric Design Guidelines for Web3 Applications: An Empirical StudyComputers10.3390/computers1402004614:2(46)Online publication date: 1-Feb-2025
https://doi.org/10.3390/computers14020046
Show More Cited By

Index Terms

A mathematical model of the finding of usability problems
1. Computing methodologies
  1. Modeling and simulation
    1. Simulation types and techniques
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Estimating the relative usability of two interfaces: heuristic, formal, and empirical methods compared
CHI '93: Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems

Two alternative user interface designs were subjected to user testing to measure user performance in a database query task. User performance was also estimated heuristically in three different ways and by use of formal GOMS modelling. The estimated ...
Finding usability problems through heuristic evaluation
CHI '92: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Usability specialists were better than non-specialists at performing heuristic evaluation, and “double experts” with specific expertise in the kind of interface being evaluated performed even better. Major usability problems have a higher probability ...
Usability in practice: formative usability evaluations - evolution and revolution
CHI EA '02: CHI '02 Extended Abstracts on Human Factors in Computing Systems
Formative evaluation is a collection of "find-and-fix" usability engineering methods, focused on identifying usability problems before a product is completed. In this forum, four experienced usability professionals will address different aspects of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '93: Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems

May 1993

547 pages

ISBN:0897915755

DOI:10.1145/169059

Chairmen:
Bert Arnold
Delft Univ. of Technology, The Netherlands
,
Gerrit van der Veer
Univ. of Twente and Vrije Univ., The Netherlands
,
Ted White
Univ. of Twente and Univ. of Ultrecht, Utrecht, The Netherlands

Copyright © 1993 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCAPH: ACM SIGCAPH Computers and the Physically Handicapped
Austrian Comp Soc: Austrian Computer Society
SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques
Human Factors Soc: Human Factors Society
SIGCHI: ACM Special Interest Group on Computer-Human Interaction
SIGGROUP: ACM Special Interest Group on Supporting Group Work
IEEE-CS: Computer Society
IFIP: International Federation for Information Processing
NGI: Dutch Computer Soc - Nederlands Genoostschapvoor Informatica

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 May 1993

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

INTERCHI93

Sponsor:

SIGCAPH
Austrian Comp Soc
SIGGRAPH
Human Factors Soc
SIGCHI
SIGGROUP
IEEE-CS
IFIP
NGI

INTERCHI93: Conference on Human Factors in Computing

April 24 - 29, 1993

Amsterdam, The Netherlands

Acceptance Rates

CHI '93 Paper Acceptance Rate 62 of 330 submissions, 19%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

924
Total Citations
View Citations
23,160
Total Downloads

Downloads (Last 12 months)4,234
Downloads (Last 6 weeks)548

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Carrión-Toro MMorales-Martínez DSantórum MVizuete AMaldonado-Garcés VAcosta-Vargas P(2025)PictoAndes: A Customizable Communication Board for Inclusive Education and Multicultural AccessibilitySustainability10.3390/su1703095617:3(956)Online publication date: 24-Jan-2025
https://doi.org/10.3390/su17030956
Marelli DBianco SCiocca G(2025)Acquisition and Modeling of Material Appearance Using a Portable, Low Cost, DeviceSensors10.3390/s2504114325:4(1143)Online publication date: 13-Feb-2025
https://doi.org/10.3390/s25041143
Bobrova PPerego P(2025)The Development of User-Centric Design Guidelines for Web3 Applications: An Empirical StudyComputers10.3390/computers1402004614:2(46)Online publication date: 1-Feb-2025
https://doi.org/10.3390/computers14020046
Žilak MCar Ž(2025)A Framework for Improving Accessibility of Serious Games in Handheld Augmented Reality Based on User Interaction DataApplied Sciences10.3390/app1504216115:4(2161)Online publication date: 18-Feb-2025
https://doi.org/10.3390/app15042161
Watanabe SKizaki HHori S(2025)Development of a Patient-Centered Symptom-Reporting Application in Pharmacy Settings Using a Hierarchical Patient-Friendly Symptom List: Developmental and Usability StudyJMIR Human Factors10.2196/7143912(e71439)Online publication date: 6-Mar-2025
https://doi.org/10.2196/71439
Perez Ramirez AOrtega AStephenson NMuñoz Osorio AKazak APhan T(2025)mHealth App to Promote Healthy Lifestyles for Diverse Families Living in Rural Areas: Usability StudyJMIR Formative Research10.2196/604959(e60495)Online publication date: 11-Feb-2025
https://doi.org/10.2196/60495
Tang YGierc MLa HKim JLiu SLam RPuterman EFaulkner G(2025)MoodMover: Development and usability testing of an mHealth physical activity intervention for depressionDIGITAL HEALTH10.1177/2055207625131775611Online publication date: 3-Feb-2025
https://doi.org/10.1177/20552076251317756
Lin JAlkouz BBouguettaya AAbu Safia A(2025)Dynamic and Immersive Framework for Drone Delivery Services in Skyway NetworksACM Transactions on Internet Technology10.1145/370889225:1(1-29)Online publication date: 4-Jan-2025
https://dl.acm.org/doi/10.1145/3708892
Vona FWarsinke MKojić TVoigt-Antons JMöller S(2025)User-Centric Evaluation Methods for Digital Twin Applications in Extended Reality2025 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)10.1109/AIxVR63409.2025.00028(142-146)Online publication date: 27-Jan-2025
https://doi.org/10.1109/AIxVR63409.2025.00028
Pimentel DBurrell SMadanume ELorenzo SAhn SGattis LWalker L(2025)Augmented Philanthropy: A Case Study Exploring the Feasibility and Impact of Spatial Computing in Strategic CommunicationsInternational Journal of Strategic Communication10.1080/1553118X.2024.2448207(1-19)Online publication date: 9-Feb-2025
https://doi.org/10.1080/1553118X.2024.2448207
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten