More Web Proxy on the site http://driver.im/

article

Users' design feedback in usability evaluation: a literature review

Author:

AsbjØrn FØlstadAuthors Info & Claims

Human-centric Computing and Information Sciences, Volume 7, Issue 1

Article No.: 100, Pages 1 - 19

https://doi.org/10.1186/s13673-017-0100-y

Published: 01 December 2017 Publication History

Abstract

As part of usability evaluation, users may be invited to offer their reflections on the system being evaluated. Such reflections may concern the system's suitability for its context of use, usability problem predictions, and design suggestions. We term the data resulting from such reflections users' design feedback. Gathering users' design feedback as part of usability evaluation may be seen as controversial, and the current knowledge on users' design feedback is fragmented. To mitigate this, we have conducted a literature review. The review provides an overview of the benefits and limitations of users' design feedback in usability evaluations. Following an extensive search process, 31 research papers were identified as relevant and analysed. Users' design feedback is gathered for a number of distinct purposes: to support budget approaches to usability testing, to expand on interaction data from usability testing, to provide insight into usability problems in users' everyday context, and to benefit from users' knowledge and creativity. Evaluation findings based on users' design feedback can be qualitatively different from, and hence complement, findings based on other types of evaluation data. Furthermore, findings based on users' design feedback can hold acceptable validity, though the thoroughness of such findings may be questioned. Finally, findings from users' design feedback may have substantial impact in the downstream development process. Four practical implications are highlighted, and three directions for future research are suggested.

References

[1]

Rubin J, Chisnell D (2008) Handbook of usability testing: how to plan, design, and conduct effective tests, 2nd edn. Wiley, Indianapolis

Digital Library

[2]

Bias RG (1994) The pluralistic usability walkthrough: coordinated empathies. In: Nielsen J, Mack RL (eds) Usability inspection methods. Wiley, New York, pp 63---76

[3]

Henderson R, Podd J, Smith MC, Varela-Alvarez H (1995) An examination of four user-based software evaluation methods. Interact Comput 7(4):412---432

[4]

O'Donnel PJ, Scobie G, Baxter I (1991) The use of focus groups as an evaluation technique in HCI. In: Diaper D, Hammond H (eds) People and computers VI, proceedings of HCI 1991. Cambridge University Press, Cambridge, pp 212---224

[5]

Lewis JR (2006) Sample sizes for usability tests: mostly math, not magic. Interactions 13(6):29---33

Digital Library

[6]

Chattratichart J, Brodie J (2004) Applying user testing data to UEM performance metrics. In: Dykstra-Erickson E, Tscheligi M (eds) CHI'04 extended abstracts on human factors in computing systems. ACM, New York, pp 1119---1122

Digital Library

[7]

Hvannberg ET, Law EL-C, Lárusdóttir MK (2007) Heuristic evaluation: comparing ways of finding and reporting usability problems. Interact Comput 19(2):225---240

Digital Library

[8]

Nielsen J (2001) First rule of usability? don't listen to users. Jakob Nielsen's Alertbox: August 5, 2001. http://www.nngroup.com/articles/first-rule-of-usability-dont-listen-to-users/

[9]

Whitefield A, Wilson F, Dowell J (1991) A framework for human factors evaluation. Behav Inf Technol 10(1):65---79

[10]

Gould JD, Lewis C (1985) Designing for usability: key principles and what designers think. Commun ACM 28(3):300---311

Digital Library

[11]

Wilson GM, Sasse MA (2000) Do users always know what's good for them? Utilising physiological responses to assess media quality. People and computers XIV--usability or else!. Springer, London, pp 327---339.

[12]

Åborg C, Sandblad B, Gulliksen J, Lif M (2003) Integrating work environment considerations into usability evaluation methods--the ADA approach. Interact Comput 15(3):453---471

[13]

Muller MJ, Matheson L, Page C, Gallup R (1998) Methods & tools: participatory heuristic evaluation. Interactions 5(5):13---18

Digital Library

[14]

Wright PC, Monk AF (1991) A cost-effective evaluation method for use by designers. Int J Man Mach Stud 35(6):891---912

Digital Library

[15]

Dumas JS, Redish JC (1999) A practical guide to usability testing. Intellect Books, Exeter

Digital Library

[16]

FØlstad A, Law E, Hornbæk K (2012) Analysis in practical usability evaluation: a survey study. In: Chi E, Höök K (eds) Proceedings of the SIGCHI conference on human factors in computing systems, CHI '12. ACM, New York, pp 2127---2136

Digital Library

[17]

Smilowitz ED, Darnell MJ, Benson AE (1994) Are we overlooking some usability testing methods? A comparison of lab, beta, and forum tests. Behav Inf Technol 13(1---2):183---190

[18]

Vermeeren AP, Law ELC, Roto V, Obrist M, Hoonhout J, Väänänen-Vainio-Mattila K (2010) User experience evaluation methods: current state and development needs. In: Proceedings of the 6th Nordic conference on human-computer interaction: extending boundaries, ACM, New York, p 521---530

Digital Library

[19]

FØlstad A, Hornbæk K (2010) Work-domain knowledge in usability evaluation: experiences with cooperative usability testing. J Syst Softw 83(11):2019---2030

Digital Library

[20]

Cowley JA, Radford-Davenport J (2011) Qualitative data differences between a focus group and online forum hosting a usability design review: a case study. Proceedings of the human factors and ergonomics society annual meeting 55(1): 1356---1360

[21]

Jacobsen NE (1999) Usability evaluation methods: the reliability and usage of cognitive walkthrough and usability test. (Doctoral thesis. University of Copenhagen, Denmark)

[22]

Cockton G, Lavery D, Woolrych A (2008) Inspection-based evaluations. In: Sears A, Jacko J (eds) The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, 2nd edn. Lawrence Erlbaum Associates, New York, pp 1171---1190

[23]

Mack RL, Nielsen J (1994) Executive summary. In: Nielsen J, Mack RL (eds) Usability inspection methods. Wiley, New York, pp 1---23

[24]

Baauw E, Bekker MM, Barendregt W (2005) A structured expert evaluation method for the evaluation of children's computer games. In: Costabile MF, Paternò F (Eds.) Proceedings of human-computer interaction--INTERACT 2005, lecture notes in computer science 3585, Springer, Berlin, p 457---469

Digital Library

[25]

FØlstad A (2007) Work-domain experts as evaluators: usability inspection of domain-specific work support systems. Int J Human Comp Interact 22(3):217---245

[26]

FrØkjær E, Hornbæk K (2005) Cooperative usability testing: complementing usability tests with user-supported interpretation sessions. In: van der Veer G, Gale C (eds) CHI'05 extended abstracts on human factors in computing systems. ACM Press, New York, pp 1383---1386

Digital Library

[27]

Andreasen MS, Nielsen HV, SchrØder SO, Stage J (2007) What happened to remote usability testing? An empirical study of three methods. In: Rosson MB, Gilmore D (Eds.) CHI'97: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, p 1405---1414

Digital Library

[28]

Hertzum M (1999) User testing in industry: a case study of laboratory, workshop, and field tests. In: Kobsa A, Stephanidis C (Eds.) Proceedings of the 5th ERCIM Workshop on User Interfaces for All, Dagstuhl, Germany, November 28---December 1, 1999. http://www.interaction-design.org/references/conferences/proceedings_of_the_5th_ercim_workshop_on_user_interfaces_for_all.html

[29]

Rosenbaum S, Kantner L (2007) Field usability testing: method, not compromise. Proceedings of the IEEE international professional communication conference, IPCC 2007.

[30]

Choe P, Kim C, Lehto MR, Lehto X, Allebach J (2006) Evaluating and improving a self-help technical support web site: use of focus group interviews. Int J Human Comput Interact 21(3):333---354

[31]

Greenbaum J, Kyng M (eds) (1991) Design at work. Lawrence Erlbaum Associates, Hillsdale

[32]

Desurvire HW, Kondziela JM, Atwood ME (1992) What is gained and lost when using evaluation methods other than empirical testing. In: Monk A, Diaper D, Harrison MD (eds) People and computers VII: proceedings of HCI 92. Cambridge University Press, Cambridge, pp 89---102

[33]

Karat CM, Campbell R, Fiegel T (1992) Comparison of empirical testing and walkthrough methods in user interface evaluation. In: Bauersfeld P, Bennett J, Lynch G (Eds.) CHI'92: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, p 397---404

Digital Library

[34]

Gray WD, Salzman MC (1998) Damaged merchandise? A review of experiments that compare usability evaluation methods. Human Comput Interact 13(3):203---261

Digital Library

[35]

Hartson HR, Andre TS, Williges RC (2003) Criteria for evaluating usability evaluation methods. Int J Human Comput Interact 15(1):145---181

[36]

Law EL-C (2006) Evaluating the downstream utility of user tests and examining the developer effect: a case study. Int J Human Comput Interact 21(2):147---172

[37]

Uldall-Espersen T, FrØkjær E, Hornbæk K (2008) Tracing impact in a usability improvement process. Interact Comput 20(1):48---63

Digital Library

[38]

FrØkjær E, Hornbæk K (2008) Metaphors of human thinking for usability inspection and design. ACM Trans Comput Human Interact (TOCHI) 14(4):20:1---20:33

Digital Library

[39]

Fu L, Salvendy G, Turley L (2002) Effectiveness of user testing and heuristic evaluation as a function of performance classification. Behav Inf Technol 21(2):137---143

[40]

Kitchenham B (2004) Procedures for performing systematic reviews (Technical Report TR/SE-0401). Keele, UK: Keele University. http://www.scm.keele.ac.uk/ease/sreview.doc

[41]

Harzing AW (2013) A preliminary test of Google Scholar as a source for citation data: a longitudinal study of Nobel prize winners. Scientometrics 94(3):1057---1075

Digital Library

[42]

Meho LI, Yang K (2007) Impact of data sources on citation counts and rankings of LIS faculty: web of Science versus Scopus and Google Scholar. J Am Soc Inform Sci Technol 58(13):2105---2125

Digital Library

[43]

FØlstad A, Anda BC, SjØberg DIK (2010) The usability inspection performance of work-domain experts: an empirical study. Interact Comput 22:75---87

Digital Library

[44]

Bruun A, Gull P, Hofmeister L, Stage J (2009) Let your users do the testing: a comparison of three remote asynchronous usability testing methods. In: Hickley K, Morris MR, Hudson S, Greenberg S (Eds.) CHI'09: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, p 1619---1628

Digital Library

[45]

Christensen L, FrØkjær E (2010) Distributed usability evaluation: enabling large-scale usability evaluation with user-controlled Instrumentation. In: Blandford A, Gulliksen J (Eds.) NordiCHI'10: Proceedings of the 6th Nordic conference on human-computer interaction: extending boundaries, ACM, New York, p 118---127

Digital Library

[46]

Bruun A, Stage J (2012) The effect of task assignments and instruction types on remote asynchronous usability testing. In: Chi EH, Höök K (Eds.) CHI'12: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, p 2117---2126

Digital Library

[47]

Hartson H R, Castillo JC (1998) Remote evaluation for post-deployment usability improvement. In: Catarci T, Costabile MF, Santucci G, Tarafino L, Levialdi S (Eds.) AVI98: Proceedings of the working conference on advanced visual interfaces, ACM Press, New York, p 22---29

Digital Library

[48]

Petrie H, Hamilton F, King N, Pavan P (2006) Remote usability evaluations with disabled people. In: Grinter R, Rodden T, Aoki P, Cutrell E, Jeffries R, Olson G (Eds.) CHI'06: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, p 1133---1141

Digital Library

[49]

Cunliffe D, Kritou E, Tudhope D (2001) Usability evaluation for museum web sites. Mus Manag Curatorship 19(3):229---252

[50]

Sullivan P (1991) Multiple methods and the usability of interface prototypes: the complementarity of laboratory observation and focus groups. In: Proceedings of the Internetional Conference on Systems Documentation--SIGDOC'91, ACM, New York, p 106---112

Digital Library

[51]

Donker A, Markopoulos P (2002) A comparison of think-aloud, questionnaires and interviews for testing usability with children. In: Faulkner X, Finlay J, Détienne F (eds) People and computers XVI--memorable yet invisible, proceedings of HCI 202. Springer, London, pp 305---316

[52]

Horsky J, McColgan K, Pang JE, Melnikas AJ, Linder JA, Schnipper JL, Middleton B (2010) Complementary methods of system usability evaluation: surveys and observations during software design and development cycles. J Biomed Inform 43(5):782---790

Digital Library

[53]

Barcelos TS, Muñoz R, Chalegre V (2012) Gamers as usability evaluators: A study in the domain of virtual worlds. In: Anacleto JC, de Almeida Nedis VP (Eds.) IHC'12: Proceedings of the 11th brazilian symposium on human factors in computing systems, Brazilian Computer Society, Porto Alegre, p 301---304

Digital Library

[54]

Edwards PJ, Moloney KP, Jacko JA, Sainfort F (2008) Evaluating usability of a commercial electronic health record: a case study. Int J Hum Comput Stud 66:718---728

Digital Library

[55]

Kontio J, Lehtola L, Bragge J (2004) Using the focus group method in software engineering: obtaining practitioner and user experiences. In: Proceedings of the International Symposium on Empirical Software Engineering --- ISESE, IEEE, Washington, p 271---280

Digital Library

[56]

Obrist M, Moser C, Alliez D, Tscheligi M (2011) In-situ evaluation of users' first impressions on a unified electronic program guide concept. Entertain Comput 2:191---202

[57]

Marsh SL, Dykes J, Attilakou F (2006) Evaluating a geovisualization prototype with two approaches: remoteinstructional vs. face-to-face exploratory. In: Proceedings of information visualization 2006, IEEE, Washington, p 310---315

Digital Library

[58]

Ebenezer C (2003) Usability evaluation of an NHS library website. Health Inf Libr J 20(3):134---142

[59]

Yeo A (2001) Global-software development lifecycle: an exploratory study. In: Jacko J, Sears A (Eds.) CHI'01: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, p 104---111

Digital Library

[60]

Rector AL, Horan B, Fitter M, Kay S, Newton PD, Nowlan WA, Robinson D, Wilson A (1992) User centered development of a general practice medical workstation: The PEN&PAD experience. In: Bauersfeld P, Bennett J, Lunch G (Eds.) CHI `92: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, p 447---453

Digital Library

[61]

Smith A, Dunckley L (2002) Prototype evaluation and redesign: structuring the design space through contextual techniques. Interact Comput 14(6):821---843

[62]

Ross S, Ramage M, Ramage Y (1995) PETRA: participatory evaluation through redesign and analysis. Interact Comput 7(4):335---360

[63]

Lamanauskas L, Pribeanu C, Vilkonis R, Balog A, Iordache DD, Klangauskas A (2007) Evaluating the educational value and usability of an augmented reality platform for school environments: some preliminary results. In: Proceedings of the 4th WSEAS/IASME international conference on engineering education p 86---91

[64]

Sylaiou S, Economou M, Karoulis A, White M (2008) The evaluation of ARCO: a lesson in curatorial competence and intuition with new technology. ACM Comput Entertain 6(20):23

Digital Library

[65]

Hornbæk K (2010) Dogmas in the assessment of usability evaluation methods. Behav Inf Technol 29(1):97---111

Digital Library

Cited By

Kazemi SLiebchen GCetinkaya D(2022)A Survey on the Usability and User Experience of the Open Community Web PortalsHCI International 2022 - Late Breaking Papers. Design, User Experience and Interaction10.1007/978-3-031-17615-9_28(409-423)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-17615-9_28
Boyd KBond RMagee JMcCormack P(2018)Can users recall their user experience with a technology?Proceedings of the 32nd International BCS Human Computer Interaction Conference10.14236/ewic/HCI2018.25(1-6)Online publication date: 4-Jul-2018
https://dl.acm.org/doi/10.14236/ewic/HCI2018.25

Users' design feedback in usability evaluation: a literature review
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Usability Evaluation of FURS Robot Control Panel Interface Design Based on SUS
Intelligent Robotics and Applications
Abstract
The objective of this study is to compare the interfaces of two FURS (Flexible Ureterorenoscopy) robotic operation consoles across generations. Analyzing functional requirements during surgery, the first-generation interface was designed. ...
On-the-Fly Usability Evaluation of Mobile Adaptive UIs Through Instant User Feedback
Human-Computer Interaction – INTERACT 2019
Abstract
Adaptive User Interfaces (AUIs) have been promoted as a solution for context variability due to their ability to automatically adapt to the context-of-use at runtime. For the acceptance of AUIs, usability plays a crucial role. Classical usability ...
Is usability evaluation important: the perspective of novice software developers
BCS-HCI '13: Proceedings of the 27th International BCS Human Computer Interaction Conference

In this paper we present the results of a study which aims to explore the perspective of novice software developers about usability evaluation. It is important for a software organization to understand how novice developers perceive the role and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Human-centric Computing and Information Sciences

Human-centric Computing and Information Sciences Volume 7, Issue 1

December 2017

729 pages

ISSN:2192-1962

EISSN:2192-1962

Issue’s Table of Contents

Copyright © Copyright © 2017 The Author(s).

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 December 2017

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kazemi SLiebchen GCetinkaya D(2022)A Survey on the Usability and User Experience of the Open Community Web PortalsHCI International 2022 - Late Breaking Papers. Design, User Experience and Interaction10.1007/978-3-031-17615-9_28(409-423)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-17615-9_28
Boyd KBond RMagee JMcCormack P(2018)Can users recall their user experience with a technology?Proceedings of the 32nd International BCS Human Computer Interaction Conference10.14236/ewic/HCI2018.25(1-6)Online publication date: 4-Jul-2018
https://dl.acm.org/doi/10.14236/ewic/HCI2018.25

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents