More Web Proxy on the site http://driver.im/

research-article

Making use of business goals in usability evaluation: an experiment with novice evaluators

Authors:

Kasper Hornbæk,

Erik FrøkjærAuthors Info & Claims

CHI '08: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 903 - 912

https://doi.org/10.1145/1357054.1357197

Published: 06 April 2008 Publication History

Abstract

The utility and impact of a usability evaluation depend on how well its results align with the business goals of the system under evaluation. However, how to achieve such alignment is not well understood. We propose a simple technique that requires active consideration of a system's business goals in planning and reporting evaluations. The technique is tested in an experiment with 44 novice evaluators using think aloud testing. The evaluators considering business goals report fewer usability problems compared to evaluators that did not use the technique. The company commissioning the evaluation, however, assesses those problems 30-42% higher on four dimensions of utility. We discuss how the findings may generalize to usability professionals, and how the technique may be used in realistic usability evaluations. More generally, we discuss how our results illustrate one of a variety of ways in which business goals and other facets of a system's context may enter into usability evaluations.

Supplementary Material

ZIP File (p903-slides.zip)

Supplemental material for Making use of business goals in usability evaluation: an experiment with novice evaluators

Download
35.40 MB

FLV File (p903-talk.flv)

Download
251.29 MB

Audio Only (p903-audiochannel0.mp3)

Download
37.28 MB

References

[1]

Bloomer, S. & Croft, R. Pitching Usability to Your Organization, interactions, november+december (1997), 18--26.

Digital Library

[2]

Bødker, K., Kensing, F., & Simonsen, J. Participatory IT Design, MIT Press, Cambridge, MA, 2004.

Digital Library

[3]

Bødker, S. & Madsen, K. H. Context: An Active Choice in Usability Work, interactions, july+august (1998), 17--25.

Digital Library

[4]

Carter, P. Liberating Usability Testing, interactions, march+april (2007), 18--22.

Digital Library

[5]

Cockton, G. Designing Worth Is Worth Designing, Proc. NordiCHI 2006, (2006), 165--174.

Digital Library

[6]

Cockton G., Lavery, D., & Woolrych, A., Inspection-Based Evaluations, in Sears, A. & Jacko, J. The Human-Computer Interaction Handbook, 2nd edition, CRC, 2007.

[7]

Cockton, G., Woolrych, A., Hall, L., & Hindmarch, M. Changing Analysts' Tunes: The Surprising Impact of a New Instrument for Usability Inspection Method Assessment, Proc. HCI 2003, Springer (2003), 145--162.

[8]

Cockton, G., Woolrych, A., & Hindemarch, M. Reconditioned Merchandise: Extended Structured Report Formats in Usability Inspection, Proc. CHI Extended Abstracts 2004, (2004), 1433--1436.

Digital Library

[9]

Desurvire, H. W., Lawrence, D., & Atwood, M. E. Empiricism Versus Judgment: Comparing User Interface Evaluation Methods on a New Telephone-Based Interface, SIGCHI bulletin, 23, 4 (1991), 58--59.

Digital Library

[10]

Dumas, J., Molich, R., & Jefferies, R. Describing Usability Problems: Are We Sending the Right Message?, interactions, 4 (2004), 24--29.

Digital Library

[11]

Følstad, A. Work-Domain Experts as Evaluators: Usability Inspection of Domain-Specific Work-Support Systems, International Journal of Human-Computer Interaction, 22, 3 (2007), 217--245.

[12]

Furniss D., Blandford, A., & Curzon, P., Usability Work in Professional Website Design: Insights from Practitioners' Perspectives, in Law, E., Hvannberg, E., & Cockton, G. (eds.) Maturing Usability: Quality in Software, Interaction and Value, Springer, 2007

[13]

Gulliksen, J., Boivie, I., & Gööransson, B. Usability Professionals-Current Practices and Future Development, Interacting with Computers, 18 (2006), 568--600.

Digital Library

[14]

Hartson, H. R., Andre, T. S., & Williges, R. C. Criteria for Evaluating Usability Evaluation Methods, International Journal of Human-Computer Interaction, 13, 4 (2001), 373--410.

[15]

Hornbæk, K. & Frøøkjær, E. Comparing Usability Problems and Redesign Proposals as Input to Practical Systems Development, Proc. CHI'2005, ACM Press (2005), 391--400.

Digital Library

[16]

Hornbæk, K. & Stage, J. Special Issue on The Interplay Between Usability Evaluation and User Interaction Design, International Journal of Human Computer Interaction, 21, 5 (2006), 117--123.

[17]

Hornbæk, K. & Frøøkjær, E. (2004) Two Psychology-Based Usability Inspection Techniques Studied in a Diary Experiment, Proc. Nordichi 2004, ACM Press (2004), 3--12.

Digital Library

[18]

Janes, J. Relevance Judgments and the Incremental Presentation of Document Representations, Information Processing & Management, 27, 6 (1991), 629--646.

[19]

Jeffries, R., Miller, J., Wharton, C., & Uyeda, K. User Interface Evaluation in the Real World: A Comparison of Four Techniques, Proc. CHI'91, (1991), 119--124.

Digital Library

[20]

Jeffries R., Usability Problem Reports: Helping Evaluators Communicate Effectively With Developers, in Nielsen, J. & Mack, R. L. Usability Inspection Methods, John Wiley, 1994, 273--294.

Digital Library

[21]

John, B. E. & Marks, S. J. Tracking the Effectiveness of Usability Evaluation Methods, Behaviour & Information Technology, 16, 4/5 (1997), 188--202.

[22]

John, B. E. & Packer, H. Learning and Using the Cognitive Walkthrough Method: a Case Study Approach, Proc. CHI'95, ACM Press (1995), 429--436.

Digital Library

[23]

Kazman, R. & Bass, L. Categorizing Business Goals for Software Architectures, Technical report CMU/SEI-2005-TR-021 (2006).

[24]

Law, E. L. C. Evaluating the Downstream Utility of User Tests and Examining the Developer Effect: A Case Study, International Journal of Human-Computer Interaction, 21, 2 (2006), 147--172.

[25]

Leavitt, H. Applied Organizational Change in Industry: Structural, Technological and Humanistic Approaches, John Wiley, 1964.

[26]

Lewis, C. Using the "Thinking-Aloud" Method in Cognitive Interface Design, Research Report RC9265 (1982).

[27]

Maguire, M. Context of Use within Usability Activities, International Journal of Human-Computer Studies, 55 (2001), 453--483.

Digital Library

[28]

Molich, R. Usable Web Design, Nyt Teknisk Forlag, 2007.

[29]

Molich, R., Jeffries, R., & Dumas, J. Making Usability Recommendations Useful and Usable, Journal of Usability Studies, 2, 4 (2007), 162--179.

[30]

Newman, W. & Taylor, A. Towards a Methodology Employing Critical Parameters to Deliver Performance Improvements in Interactive Systems, Proc. INTERACT'99, IOS Press (1999), 605--612.

[31]

Nielsen, J. Finding Usability Problems through Heuristic Evaluation, Proc. CHI'92, ACM Press (1992), 373--380.

Digital Library

[32]

Nielsen, J. & Molich, R. Heuristic Evaluation of User Interfaces, Proc. CHI'90, ACM Press (1990), 249--256.

Digital Library

[33]

Nielsen, J., Molich, R., Snyder, C., & Farrell, S. E-Commerce User Experience, Nielsen Norman Group, Fremont, CA, 2001.

Digital Library

[34]

Nørgaard, M. & Hornbææk, K. What Do Usability Evaluators Do in Practice? An Explorative Study of Think-Aloud Testing, Proc. DIS 2006, (2006), 209--218.

Digital Library

[35]

Pinelle, D. & Gutwin, C. Groupware Walkthrough: Adding Context to Groupware Usability Evaluation, Proc. CHI 2002, ACM Press (2002), 455--462.

Digital Library

[36]

Redish, J., Bias, R., Bailey, R., Molich, R., Dumas, J., & Spool, J. Usability in Practice: Formative Usability Evaluations -- Evolution and Revolution, (2002), 885--890.

Digital Library

[37]

Rubin, J. Handbook of Usability Testing, John Wiley & Sons, New York, NY, 1994.

Digital Library

[38]

Simonsen, J. Involving Top Management in IT Projects, Communications of the ACM, 50, 8 (2007), 53--58.

Digital Library

[39]

Smith, S. L. & Mosier, J. N. Guidelines for Designing User Interface Software, ESD-TR-86-278 (1986).

[40]

Uldall-Espersen, T. & Frøøkjær, E. Usability and Software Development: Roles of the Stakeholders, Proc. HCI International, Lecture Notes in Computer Science 4550, (2007), 642--651.

Digital Library

[41]

Wharton C. et al., The Cognitive Walkthrough Method: a Practitioner's Guide, in Nielsen, J. & Mack, R. L. (eds.) Usability Inspection Methods, John Wiley & Sons, 1994, 105--140.

Digital Library

[42]

Whiteside J., Bennett, J., & Holtzblatt, K., Usability Engineering: Our Experience and Evolution, in Helander, M. (ed.) Handbook of Human-Computer Interaction, Elsevier, 1988, 791-817.

[43]

Wixon D. & Wilson, C., The Usability Engineering Framework, in Helander, M., Landauer, T., & Prabhu, P. (eds.) Handbook of Human-Computer Interaction, 2nd edition, Elsevier, 1997, 653--688.

Cited By

Kretzer FMaedche A(2023)Making Usability Test Data Actionable! A Quantitative Test-Driven Prototyping ApproachExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585659(1-6)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585659
Nasir MIkram NJalil Z(2022)Usability inspection: Novice crowd inspectors versus expertJournal of Systems and Software10.1016/j.jss.2021.111122183(111122)Online publication date: Jan-2022
https://doi.org/10.1016/j.jss.2021.111122
Clemmensen TClemmensen T(2021)Relation Artefacts Type IIIHuman Work Interaction Design10.1007/978-3-030-71796-4_5(81-105)Online publication date: 24-Sep-2021
https://doi.org/10.1007/978-3-030-71796-4_5
Show More Cited By

Index Terms

Making use of business goals in usability evaluation: an experiment with novice evaluators
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
2. Software and its engineering
  1. Software notations and tools
    1. Development frameworks and environments
      1. Integrated and visual development environments

Recommendations

Finding usability problems through heuristic evaluation
CHI '92: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Usability specialists were better than non-specialists at performing heuristic evaluation, and “double experts” with specific expertise in the kind of interface being evaluated performed even better. Major usability problems have a higher probability ...
Comparison of techniques for matching of usability problem descriptions

Matching of usability problem descriptions consists of determining which problem descriptions are similar and which are not. In most comparisons of evaluation methods matching helps determine the overlap among methods and among evaluators. However, ...
An AHP-Based Approach in the Early Design Evaluation via Usability Goals
IVIC 2013: Third International Visual Informatics Conference on Advances in Visual Informatics - Volume 8237

The usability engineering process has indicated the importance of setting usability criteria and a measureable level for important goals. Without goal-oriented practice, novice or inexperienced designers do not know how to deliver usable and quality ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '08: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

April 2008

1870 pages

ISBN:9781605580111

DOI:10.1145/1357054

General Chairs:
Mary Czerwinski
Microsoft Research, USA
,
Arnie Lund
Microsoft, USA
,
Program Chair:
Desney Tan
Microsoft Research, USA

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 April 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '08

Sponsor:

CHI '08: CHI Conference on Human Factors in Computing Systems

April 5 - 10, 2008

Florence, Italy

Acceptance Rates

CHI '08 Paper Acceptance Rate 157 of 714 submissions, 22%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
1,934
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kretzer FMaedche A(2023)Making Usability Test Data Actionable! A Quantitative Test-Driven Prototyping ApproachExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585659(1-6)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585659
Nasir MIkram NJalil Z(2022)Usability inspection: Novice crowd inspectors versus expertJournal of Systems and Software10.1016/j.jss.2021.111122183(111122)Online publication date: Jan-2022
https://doi.org/10.1016/j.jss.2021.111122
Clemmensen TClemmensen T(2021)Relation Artefacts Type IIIHuman Work Interaction Design10.1007/978-3-030-71796-4_5(81-105)Online publication date: 24-Sep-2021
https://doi.org/10.1007/978-3-030-71796-4_5
McDonald SCockton GIrons A(2020)The Impact of Thinking-Aloud on Usability InspectionProceedings of the ACM on Human-Computer Interaction10.1145/33978764:EICS(1-22)Online publication date: 18-Jun-2020
https://dl.acm.org/doi/10.1145/3397876
Clemmensen THertzum MAbdelnour-Nocera J(2020)Ordinary User Experiences at WorkACM Transactions on Computer-Human Interaction10.1145/338608927:3(1-31)Online publication date: 13-Jun-2020
https://dl.acm.org/doi/10.1145/3386089
Riihiaho S(2017)Usability TestingThe Wiley Handbook of Human Computer Interaction10.1002/9781118976005.ch14(255-275)Online publication date: 28-Dec-2017
https://doi.org/10.1002/9781118976005.ch14
Alshamsi AWilliams NAndras PFaily SJiang NDogan HJacqui T(2016)The trade-off between usability and security in the context of eGovernmentProceedings of the 30th International BCS Human Computer Interaction Conference: Fusion!10.14236/ewic/HCI2016.36(1-13)Online publication date: 11-Jul-2016
https://dl.acm.org/doi/10.14236/ewic/HCI2016.36
Bornoe NBruun AStage JDuh HLueg CBillinghust MHuang W(2016)Facilitating redesign with design cardsProceedings of the 28th Australian Conference on Computer-Human Interaction10.1145/3010915.3010921(452-461)Online publication date: 29-Nov-2016
https://dl.acm.org/doi/10.1145/3010915.3010921
Bruun AJensen KKristensen DKjeldskov J(2016)Escaping the Trough: Towards Real-World Impact of Tabletop ResearchInternational Journal of Human–Computer Interaction10.1080/10447318.2016.120676433:2(77-93)Online publication date: 3-Aug-2016
https://doi.org/10.1080/10447318.2016.1206764
Chilana PKo AWobbrock JBegole BKim JInkpen KWoo W(2015)From User-Centered to Adoption-Centered DesignProceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems10.1145/2702123.2702412(1749-1758)Online publication date: 18-Apr-2015
https://dl.acm.org/doi/10.1145/2702123.2702412
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents