More Web Proxy on the site http://driver.im/

research-article

Was it worth the hassle?: ten years of mobile HCI research discussions on lab and field evaluations

Authors:

Jesper Kjeldskov,

Mikael B. SkovAuthors Info & Claims

MobileHCI '14: Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services

Pages 43 - 52

https://doi.org/10.1145/2628363.2628398

Published: 23 September 2014 Publication History

Abstract

Evaluation is considered one of the major cornerstones of human-computer interaction (HCI). During the last decade, several studies have discussed pros and cons of lab and field evaluations. Based on these discussions, we conduct a review to explore the past decade of mobile HCI research on field and lab evaluation, investigating responses in the literature to the "is it worth the hassle?" paper from 2004. We find that while our knowledge and experience with both lab and field studies have grown considerably, there is still no definite answer to the lab versus field question. In response we suggest that the real question is not if -- but when and how -- to go into the field. In response we suggest moving beyond usability evaluations, and to engage with field studies that are truly in-the-wild, and longitudinal.

References

[1]

Abdulrazak, B. and Malik, Y. Review of Challenges, Requirements, and Approaches of Pervasive Computing System Evaluation. IETE Technical Review 29, 6 (2012), 506--522.

[2]

Alsos, O. A. and Dabelow, B. A comparative evaluation study of basic interaction techniques for PDAs in pointof-care situations. Proc. P-Health'10, IEEE (2010), 1--8.

[3]

Axup, J. Building a Path For Future Communities. In Handbook of Research on Socio-Technical Design, (2008), 3--20.

[4]

Baillie, L. and Schatz, R. Exploring Multimodality in the Laboratory and the Field. Proc. CMI'05, ACM (2005), 100--107.

Digital Library

[5]

Barnard, L., Y, J. S., Jacko, J. A. and Sears, A. An empirical comparison of use-in-motion evaluation scenarios for mobile computing devices. IJHCS 62 (2005), 487--520.

Digital Library

[6]

Barnard, L., Yi, J.S., Jacko, J. and Sears, A. Capturing the effect of context on human performance in mobile computing. Pers Ubiquit Comput 11 (2007), 81--96.

Digital Library

[7]

Billi, M., Burzagli, L., Catarci, T., Santucci, G., Bertini, E., Gabbanini, F. and Palchetti, E. Unified methodology for evaluation of accessibility and usability of mobile applications. Univ. Access Inf. Soc., 9 (2010), 337--356.

Digital Library

[8]

Brown, B., Reeves, S., and Sherwood, S. Into the Wild: Challenges and Opportunities for Field Trial Methods. Proc. CHI'11, ACM (2011), 1657--1666.

Digital Library

[9]

Burghardt, D. and Wirth, K. Comparison of Evaluation Methods for Field-Based Usability Studies of Mobile Map Applications. Proc. International Cartographic Conference (2011).

[10]

Carter, S. Techniques and tools for field-based early-stage study and iteration of ubicomp applications: A dissertation proposal. University of California, 2005.

[11]

Carter, S., Mankoff, J., Klemmer, S. R. and Matthews, T. Exiting the Cleanroom: On Ecological Validity and Ubiquitous Computing. Human-Computer Interaction 23, 1, (2008), 47--99.

[12]

Crabtree, A., Chamberlain, A., Grinter, R. E., Jones, M., Rodden, T. and Rogers, Y. Introduction to the Special Issue of "The Turn to The Wild". TOCHI 20, 3 (2013).

Digital Library

[13]

Dahl, Y., Alsos, O. A. and Svanæs, D. Evaluating Mobile Usability: The Role of Fidelity in Full-Scale Laboratory Simulations with Mobile ICT for Hospitals, Proc. HCII'09, Springer (2009), 232--241.

Digital Library

[14]

Dahl, Y. Seeking a Theoretical Foundation for Design of In Sitro Usability Assessments. Proc. NordiCHI'10, ACM (2010), 623--626.

Digital Library

[15]

Davies, N., Cheverst, K., Dix, A. and Hesse, A. Understanding the Role of Image Recognition in Mobile Tour Guides. Proc. Mobile HCI'05, ACM (2005), 191--198.

Digital Library

[16]

Dearman, D., Hawkey, K. and Inkpen, K.M. Rendezvousing with location-aware devices. IwC 17 (2005), 524--566.

Digital Library

[17]

Duh, H. B., Tan, G. and Chen, V. H. Usability Evaluation for Mobile Devices: A Comparison of Laboratory and Field Tests. Proc. Mobile HCI'06, ACM (2006), 181--186.

Digital Library

[18]

Fiotakis, G., Raptis, D. and Avouris, N. Considering Cost in Usability Evaluation of Mobile Applications: Who, Where and When. Proc. Interact'09, Springer (2009), 231--234.

Digital Library

[19]

Gelderblom, H., Bruin, J. and Singh, A. Three Methods for Evaluating Mobile Phone Applications Aimed Users in a Developing Environment: AComparative Case Study. Proc. M4D'12 (2012), 321--334.

[20]

Hagen, P., Robertson, T., Kan, M. and Sadler, K. Emerging research methods for understanding mobile technology use. Proc. OzCHI'05, CHISIG (2005), 1--10.

Digital Library

[21]

Holone, H., Mislund, G., Tolsby, H. and Kristoffersen, S. Aspects of personal navigation with collaborative feedback. Proc. NordiCHI'08, ACM (2008), 182--191.

Digital Library

[22]

Howell, M., Love, S. and Turner, M. The impact of interface metaphor and context of use on the usability of a speech-based mobile city guide service. Behaviour & Information Technology 24, 1 (2005): 67--78.

[23]

Holzinger, A., Schlögl, M., Peischl, B. and Debevc, M. Optimization of a handwriting recognition algorithm for a mobile enterprise health information system on the basis of real-life usability research. Proc. ICETE'10, Springer (2010), 97--111.

[24]

Høegh, R. T., Kjeldskov, J., Skov, M. B. and Stage J. Setting Up A Field Laboratory for Evaluating In Situ. In Handbook of Research on User Interface Design and Evaluation for Mobile Technology, ISR, 2008.

[25]

Iachello, G. and Terrenghi, L. Mobile HCI 2004: Experience and Reflection. Pervasive Computing, JanMar (2005), 88--91.

Digital Library

[26]

Jambon, F., Golanski, C. and Pommier, P. J. Meta-evaluation of a context-aware mobile device usability. Proc. UBICOMM, IEEE (2007), pp. 21--26.

Digital Library

[27]

Jambon, F. and Meillon, B. User Experience in the Wild. Proc. CHI'09 EA, ACM (2009), 4069--4074.

Digital Library

[28]

Johnson, P. Usability and Mobility; Interactions on the move. Proc. Mobile HCI'98, GIST Technical Report G98-1 (1998)

[29]

Jumisko-Pyykkö, S. and Utriainen, T. (2011) A Hybrid Method for Quality Evaluation in the Context of Use for Mobile (3D) Television. Multimedia Tools and Applications, 55(2): 185--225.

Digital Library

[30]

Kaikkonen, A., Kekäläinen, A., Cankar, M., Kallio, T. and Kankainen, A. Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing. Journal of Usability Studies 1, 1 (2005), 4--16.

[31]

Kaikkonen, A., Kekäläinen, A., Cankar, M., Kallio, T., and Kankainen, A. Will laboratory test results be valid in mobile contexts? In Handbook of Research on User Interface Design and Evaluation for Mobile Technology, ISR, 2008.

[32]

Kalnikaite, V., Bird, J. and Rogers, Y. Decision-making in the aisles: informing, overwhelming or nudging supermarket shoppers' Pers Ubiquit Comput 17 (2013), 1247--1259.

Digital Library

[33]

Kellar, M., Inkpen, K., Dearman, D., et al. Evaluation of Mobile Collaboration: Learning from our Mistakes. Technical Report 2004-13, Dalhousie University, 2004.

[34]

Khanum, M. A. and Trivedi, M. C. Comparison of Testing Environments with Children for Usability Problem Identification. International Journal of Engineering and Technology 5, 3 (2013), 2048--2053.

[35]

Kjeldskov J. and Graham C. A Review of Mobile HCI Research Methods. Proc. Mobile HCI'03, Springer (2003), 317--335.

[36]

Kjeldskov, J., Skov, M. B., Als, B. S. and Høegh, R. T. Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. Proc. Mobile HCI'04, Springer (2004), 61--73.

[37]

Kjeldskov, J., Graham, C., Pedell, S., Vetere, F., Howard, S., Balbo, S. and Davies, J. Evaluating the usability of a mobile guide: The influence of location, participants and resources. Behaviour & Information Technology 24, 1 (2005), 51--65.

[38]

Kjeldskov, J. and Stage, J. Exploring 'Canned Communication' for coordinating distributed mobile work activities. IwC 18 (2006) 1310--1335.

Digital Library

[39]

Kjeldskov, J. and Paay, J. A Longitudinal Review of Mobile HCI Research Methods. Proc. Mobile HCI'12, ACM (2012), 69--78.

Digital Library

[40]

Kondratova, I., Lumsden, J. and Langton, N. Multimodal Field Data Entry: Performance and Usability Issues. Proc. International Conference on Computing and Decision Making NRC-CNRC (2006).

[41]

Korn, M. and Bødker, S. Looking ahead: how field trials can work in iterative and exploratory design of ubicomp systems. Proc. UbiComp'12, ACM (2012), 21--30.

Digital Library

[42]

Kray, C., Olivier, P., Guo, A. W., Singh, P., Ha, H. N. and Blythe, P. Taming Context: A Key Challenge in Evaluating the Usability of Ubiquitous Systems. Proc. USE'07 Workshop at Ubicomp'07 (2007).

[43]

Larsen, J. E., Petersen, M. K., Handler, R. and Zandi, N. Observing the Context of Use of a Media Player on Mobile Phones using Embedded and Virtual Sensors. Proc. NordiCHI'10, ACM (2010), 33--36.

[44]

Leitner, G., Ahlström, D. and Hitz, M. Usability of Mobile Computing in Emergency Response Systems - Lessons Learned and Future Directions. Proc. USAB'07. Springer (2007), 241--254.

Digital Library

[45]

Lumsden, J., Kondratova, I. and Durling, S. Investigating microphone efficacy for mobile speech-based data entry. Proc. HCI'07, Springer (2007), 89--97.

Digital Library

[46]

Lumsden, J. and MacLean, R. A Comparison of PseudoPaper and Paper Prototyping Methods for Mobile Evaluations. Proc. MONET'08 (2008), 538--457.

Digital Library

[47]

Lumsden, J., Langton, N., and Kondratova, I. Bringing the High Seas into the Lab to Evaluate Speech Input Feasibility: A Case Study. Proc. SiMPE Workshop at Mobile HCI'10 (2010).

[48]

Maly, I., Mikovec, Z., Vystrcil, J., Franc, J. and Slavik, P. An evaluation tool for research of user behavior in a realistic mobile environment. Pers Ubiquit Comput 17 (2013), 3--14.

Digital Library

[49]

Morrison, A., McMillan, D., Reeves, S., Sherwood, S., and Chalmers, M. A Hybrid Mass Participation Approach to Mobile Software Trials. Proc. CHI'12, ACM (2012), 1311--1320.

Digital Library

[50]

Nielsen, C. M., Overgaard, M., Pedersen, M. B., Stage, J. and Stenild, S. It's Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field. Proc. NordiCHI'06, ACM (2006), 272--280.

Digital Library

[51]

Oulasvirta, A., Tamminen, S., Roto, V. and Kuorelahti. Interaction in 4-second Bursts: The Fragmented Nature of Attentional Resources in Mobile HCI. Proc. CHI'05, ACM (2005), 919--928.

Digital Library

[52]

Oulasvirta, A. and Nyyssönen, T. Flexible Hardware Configurations for Studying Mobile Usability. Journal of Usability Studies 4, 2 (2009), 93--105.

[53]

Oulasvirta, A. Rethinking Experimental Designs for Field Evaluations. Pervasive Computing, Oct-Dec (2012), 60--67.

Digital Library

[54]

Rogers, Y., Connelly, K., Tedesco, L., Hazlewood, W., Kurtz, A., Hall, R. E., Hursey, J., and Toscos, T. Why It's Worth the Hassle: The Value of In-Situ Studies When Designing Ubicomp. Proc. UbiComp'07, Springer (2007), 336--353.

Digital Library

[55]

Roto, V., Väätäjä, H., Jumisko-Pyykkö, S., and Väänänen-Vainio-Mattila, K. Best Practices for Capturing Context in User Experience Studies in the Wild. Proc. MindTrek'11 (2011), 91--98.

Digital Library

[56]

Skattør, B. Training and Deployment as a basis for Usability Engineering of Mobile Systems. Proc. ACHI, IEEE (2008), 277--284.

Digital Library

[57]

Streefkerk, J. W., van Esch-Bussemakers, M. P. and Neerincx, M. A. Field Evaluation of a Mobile Location-Based Notification System for Police Officers. Proc. Mobile HCI'08, ACM (2008), 101--108.

Digital Library

[58]

Sun, X. and May, A. A Comparison of Field-Based and Lab-Based Experiments to Evaluate User Experience of Personalised Mobile Devices. Adv. in Hum.-Comp. Int., Hindawi (2013), Article 2.

Digital Library

[59]

Tolmie, P., Crabtree, A., Egglestone, S., Humble, J., Greenhalgh, C. and Rodden, T. Digital Plumbing: the mundane work of deploying UbiComp in the home. Pers Ubiquit Comput 14 (2010), 181--196.

Digital Library

[60]

Vastenburg, M. H., Keyson, D. V. and de Ridder, H. Measuring User Experiences of Prototypical Autonomous Products in a Simulated Home Environment. Proc. HCII'07, Springer (2007), 998--1007.

Digital Library

[61]

Wilfinger, D., Pirker, M., Bernhaupt, R. and Tscheligi, M. Evaluating and Investigating an iTV Concept in the Field. Proc. EuroITV'09, ACM (2009), 175--178.

Digital Library

[62]

Wilson, M. L, Russel, A., Smith, D. A. and schraefel, m. c. mSpace Mobile: Exploring Support for Mobile Tasks. Proc. HCI'06, Springer (2006), 193--202.

[63]

Wilson, S., Galliers, J. and Fone, J. (2007) Cognitive Artifacts in Support of Medical Shift Handover: An in Use, in Situ Evaluation. IJHCS 22, 1&2 (2007), 59--80

[64]

Wilson, M. L., Mackay, W., Chi, E., Berstein, M., Russell, D. and Thimbleby, H. RepliCHI - CHI should be replicating and validating results more: discuss. Proc. CHI'11 EA, ACM (2011), 463--466.

Digital Library

Cited By

Kallio JKinnula AMäkelä SJärvinen SRäsänen PHosio SBordallo López M(2024)Lessons From 3 Longitudinal Sensor-Based Human Behavior Assessment Field Studies and an Approach to Support Stakeholder Management: Content AnalysisJournal of Medical Internet Research10.2196/5046126(e50461)Online publication date: 31-Oct-2024
https://doi.org/10.2196/50461
Boot MUlak MGeurs KHavinga P(2024)Using body sensors for evaluating the impact of smart cycling technologies on cycling experiences: a systematic literature review and conceptual frameworkEuropean Transport Research Review10.1186/s12544-024-00635-316:1Online publication date: 14-Feb-2024
https://doi.org/10.1186/s12544-024-00635-3
Fan JSaaty MMccrickard D(2024)Education in HCI Outdoors: A Diary Study ApproachProceedings of the 6th Annual Symposium on HCI Education10.1145/3658619.3658621(1-10)Online publication date: 5-Jun-2024
https://dl.acm.org/doi/10.1145/3658619.3658621
Show More Cited By

Index Terms

Was it worth the hassle?: ten years of mobile HCI research discussions on lab and field evaluations
1. Human-centered computing

Recommendations

Snooze!: investigating the user-defined deferral of mobile notifications
MobileHCI '18: Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services

Notifications on mobile devices are a prominent source of interruptions. Previous work suggests using opportune moments to deliver notifications to reduce negative effects. In this paper, we instead explore the manual deferral of notifications. We ...
“I Could Wear It All of the Time, Just Like My Wedding Ring:” Insights into Older People’s Perceptions of Smart Rings
CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

We present findings from an examination of older adults’ perceptions of and preferences for smart rings, a unique class of interactive wearables gaining traction in the consumer market. To this end, we conducted semi-structured interviews with nine ...
Annotif: A System for Annotating Mobile Notifcations in User Studies
MUM '19: Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia

Notifications are an essential feature of smartphones. While they support users in staying up-to-date, they are also a prominent source of interruptions. A deeper understanding of mobile notifications is required to avoid adverse effects. However, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MobileHCI '14: Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services

September 2014

664 pages

ISBN:9781450330046

DOI:10.1145/2628363

Conference Chairs:
Aaron Quigley
University of St. Andrews, UK
,
Sara Diamond
OCAD University, Canada
,
Program Chairs:
Pourang Irani
University of Manitoba, Canada
,
Sriram Subramanian
University of Bristol, UK

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 September 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MobileHCI '14

Sponsor:

SIGCHI

MobileHCI '14: 16th International Conference on Human-Computer Interaction with Mobile Devices and Services

September 23 - 26, 2014

ON, Toronto, Canada

Acceptance Rates

MobileHCI '14 Paper Acceptance Rate 35 of 124 submissions, 28%;

Overall Acceptance Rate 202 of 906 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

70
Total Citations
View Citations
1,425
Total Downloads

Downloads (Last 12 months)126
Downloads (Last 6 weeks)9

Reflects downloads up to 17 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kallio JKinnula AMäkelä SJärvinen SRäsänen PHosio SBordallo López M(2024)Lessons From 3 Longitudinal Sensor-Based Human Behavior Assessment Field Studies and an Approach to Support Stakeholder Management: Content AnalysisJournal of Medical Internet Research10.2196/5046126(e50461)Online publication date: 31-Oct-2024
https://doi.org/10.2196/50461
Boot MUlak MGeurs KHavinga P(2024)Using body sensors for evaluating the impact of smart cycling technologies on cycling experiences: a systematic literature review and conceptual frameworkEuropean Transport Research Review10.1186/s12544-024-00635-316:1Online publication date: 14-Feb-2024
https://doi.org/10.1186/s12544-024-00635-3
Fan JSaaty MMccrickard D(2024)Education in HCI Outdoors: A Diary Study ApproachProceedings of the 6th Annual Symposium on HCI Education10.1145/3658619.3658621(1-10)Online publication date: 5-Jun-2024
https://dl.acm.org/doi/10.1145/3658619.3658621
Haliburton LGrüning DRiedel FSchmidt ATerzimehić N(2024)A Longitudinal In-the-Wild Investigation of Design Frictions to Prevent Smartphone OveruseProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642370(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642370
Zhu QWang ZZeng WTong WLin WMa X(2024)Make Interaction Situated: Designing User Acceptable Interaction for Situated Visualization in Public EnvironmentsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642049(1-21)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642049
McLoughlin RMcDonnell MMcEvoy D(2024)The Impact of 3D and AR Product Presentation on Mental Imagery, Product Uncertainty and User ExperienceProceedings of the Future Technologies Conference (FTC) 2024, Volume 110.1007/978-3-031-73110-5_28(425-443)Online publication date: 5-Nov-2024
https://doi.org/10.1007/978-3-031-73110-5_28
Ntoa SMargetis GValakou AMakri FDimitriou NKarakostas IKokkinis GApostolakis KTzovaras DStephanidis C(2024)A Mixed-Methods Approach for the Evaluation of Situational Awareness and User Experience with Augmented Reality TechnologiesAugmented Cognition10.1007/978-3-031-61569-6_13(199-219)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-61569-6_13
Tran TParker CHoggenmüller MHespanhol LTomitsch M(2023)Simulating Wearable Urban Augmented Reality Experiences in VR: Lessons Learnt from Designing Two Future Urban InterfacesMultimodal Technologies and Interaction10.3390/mti70200217:2(21)Online publication date: 16-Feb-2023
https://doi.org/10.3390/mti7020021
Fangel Skov KAxel Nielsen PKjeldskov J(2023)Tuning Shared Hospital Spaces: Sound Zones in HealthcareProceedings of the 18th International Audio Mostly Conference10.1145/3616195.3616198(63-70)Online publication date: 30-Aug-2023
https://dl.acm.org/doi/10.1145/3616195.3616198
Stefanidi EBentvelzen MWoźniak PKosch TWoźniak MMildner TSchneegass SMüller HNiess J(2023)Literature Reviews in HCI: A Review of ReviewsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581332(1-24)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581332
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents