More Web Proxy on the site http://driver.im/

research-article

LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples

Authors:

Katharina Reinecke,

Krzysztof Z. GajosAuthors Info & Claims

CSCW '15: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing

Pages 1364 - 1378

https://doi.org/10.1145/2675133.2675246

Published: 28 February 2015 Publication History

Abstract

Web-based experimentation with uncompensated and unsupervised samples has the potential to support the replication, verification, extension and generation of new results with larger and more diverse sample populations than previously seen. We introduce the experimental online platform LabintheWild, which provides participants with personalized feedback in exchange for participation in behavioral studies. In comparison to conventional in-lab studies, LabintheWild enables the recruitment of participants at larger scale and from more diverse demographic and geographic backgrounds. We analyze Google Analytics data, participants' comments, and tweets to discuss how participants hear about the platform, and why they might choose to participate. Analyzing three example experiments, we additionally show that these experiments replicate previous in-lab study results with comparable data quality.

References

[1]

Arnett, J. The neglected 95%: Why American psychology needs to become less American. American Psychologist 63(7) (2008), 602--14.

[2]

Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., and Plumb, I. The "Reading the Mind in the Eyes" Test revised version: A study with normal adults, and adults with Asperger syndrome or high-functioning autism. Journal of child psychology and psychiatry, and allied disciplines 42, 2 (Feb. 2001), 241--251.

[3]

Berinsky, A. J., Huber, G. A., and Lenz, G. S. Using Mechanical Turk as a subject recruitment tool for experimental research. Political Analysis 20 (2012), 351--68.

[4]

Buhrmester, M., Kwang, T., and Gosling, S. D. Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data- Perspectives on Psychological Science 6, 1 (2011), 3--5.

[5]

Cavanagh, J. P. Relation between the immediate memory span and the memory search rate. Psychological Review 79, 6 (1972), 525--530.

[6]

Cavanagh, J. P., and Chase, W. G. The equivalence of target and nontarget processing in visual search. Perception & Psychophysics 9, 6 (Nov. 1971), 493--495.

[7]

Chase, W. G., and Calfee, R. C. Modality and similarity effects in short-term recognition memory. J Exp Psychol 81, 3 (1969), 510--514.

[8]

Corbin, L., and Marquer, J. Effect of a simple experimental control: The recall constraint in Sternberg's memory scanning task. European Journal of Cognitive Psychology 20, 5 (Sept. 2008), 913--935.

[9]

Cruse, D., and Clifton, Jr, C. Recoding strategies and the retrieval of information from memory. Cognitive Psychology 4, 2 (1973), 157--193.

[10]

Downs, J. S., Holbrook, M. B., Sheng, S., and Cranor, L. F. Are your participants gaming the system?: Screening Mechanical Turk workers. In Proc. CHI'10, ACM (2010), 2399--2402.

Digital Library

[11]

Eriksson, K., and Simpson, B. Emotional reactions to losing explain gender differences in entering a risky lottery. Judgment and Decision Making 5, 3 (2010), 159--163.

[12]

Festinger, L. A theory of social comparison processes. Human relations 7(2) (1954), 117--140.

[13]

Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., and Wilmer, J. B. Is the web as good as the lab? comparable performance from web and lab in cognitive/perceptual experiments. Psychonomic bulletin & review 19, 5 (2012), 847--857.

[14]

Glaser, B. G., and Strauss, A. L. The discovery of grounded theory: Strategies for qualitative research. Transaction Publishers, 2009.

[15]

Goodman, J. K., Cryder, C. E., and Cheema, A. Data collection in a flat world: The strengths and weaknesses of mechanical turk samples. Journal of Behavioral Decision Making 26, 3 (2013), 213--224.

[16]

Heer, J., and Bostock, M. Crowdsourcing graphical perception: Using Mechanical Turk to assess visualization design. In Proc. CHI'10 (2010), 203--212.

Digital Library

[17]

Henrich, J., Heine, S. J., and Norenzayan, A. The weirdest people in the world. Behavioral and Brain Sciences 33 (2010), 61--83.

[18]

Horton, J. J., Rand, D. G., and Zeckhauser, R. J. The online laboratory: Conducting experiments in a real labor market. Experimental Economics (2011).

[19]

Ipeirotis, P. Demographics of Mechanical Turk. NYU Working Paper No. CEDER-10-01, March 2010.

[20]

Ipeirotis, P. G. Analyzing the Amazon Mechanical Turk marketplace. XRDS: Crossroads, The ACM Magazine for Students 17, 2 (2010), 16--21.

Digital Library

[21]

Kapelner, A., and Chandler, D. Preventing satisficing in online surveys. In Proceedings of CrowdConf (2010).

[22]

Kittur, A., Chi, E. H., and Suh, B. Crowdsourcing user studies with Mechanical Turk. In Proc. CHI'08, ACM (2008), 453--456.

Digital Library

[23]

Komarov, S., Reinecke, K., and Gajos, K. Z. Crowdsourcing performance evaluations of user interfaces. In Proc. CHI'13 (2013), 207--216.

Digital Library

[24]

Kristofferson, M. W. Effects of practice on character-classification performance. Canadian Journal of Psychology/Revue canadienne de psychologie 26, 1 (1972), 54--60.

[25]

Lindgaard, G., Fernandes, G., Dudek, C., and Brown, J. Attention web designers: You have 50 milliseconds to make a good first impression! Behaviour & Information Technology 25, 2 (2006), 115--126.

[26]

Mason, W., and Suri, S. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods 44, 1 (2012), 1--23.

[27]

Nickerson, R. S. Response times with a memory-dependent decision task. J Exp Psychol 72, 5 (Nov. 1966), 761--769.

[28]

Oppenheimer, D., Meyvis, T., and Davidenko, N. Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology 45, 4 (2009), 867--872.

[29]

Orne, M. T. Demand characteristics and the concept of quasi-controls. In Artifacts in Behavioral Research, R. Rosenthal and R. L. Rosnow, Eds. Oxford University Press, July 2009, ch. 5.

[30]

Paolacci, G., Chandler, J., and Ipeirotis, P. G. Running experiments on Amazon Mechanical Turk. Judgment and Decision making 5, 5 (2010), 411--419.

[31]

Rand, D. G. The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of theoretical biology (2012).

[32]

Reinecke, K., and Gajos, K. Z. Quantifying visual preferences around the world. In Proc. CHI'14, ACM (2014), 11--20.

Digital Library

[33]

Reinecke, K., Nguyen, M. K., Bernstein, A., Näf, M., and Gajos, K. Z. Doodle around the world: Online scheduling behavior reflects cultural differences in time perception and group decision-making. In Proc. CSCW'13, ACM (2013), 45--54.

Digital Library

[34]

Shaw, A. D., Horton, J. J., and Chen, D. L. Designing incentives for inexpert human raters. In Proc. CSCW'11, ACM (2011), 275--284.

Digital Library

[35]

Sternberg, S. High-speed scanning in human memory. Science 153, 3736 (Aug. 1966), 652--654.

[36]

Sternberg, S. Two operations in character recognition: Some evidence from reaction-time measurements. Perception & Psychophysics 2, 2 (1967), 45--53.

[37]

Sternberg, S. Memory scanning: New findings and current controversies. The Quarterly journal of experimental psychology 27, 1 (1975), 1--32.

[38]

Suri, S., and Watts, D. Cooperation and contagion in networked public goods experiments. Arxiv preprint arXiv10081276 (2010).

[39]

Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., and Malone, T. W. Evidence for a collective intelligence factor in the performance of human groups. science 330, 6004 (2010), 686--688.

Cited By

Lin HHan TGajos KGupta A(2024)Hevelius Report: Visualizing Web-Based Mobility Test Data For Clinical Decision and Learning SupportProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688490(1-10)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3688490
Rodd J(2024)Moving experimental psychology online: How to obtain high quality data when we can’t see our participantsJournal of Memory and Language10.1016/j.jml.2023.104472134(104472)Online publication date: Feb-2024
https://doi.org/10.1016/j.jml.2023.104472
Warburton MCampagnoli CMon-Williams MMushtaq FMorehead J(2024)Input device matters for measures of behaviour in online experimentsPsychological Research10.1007/s00426-024-02065-189:1Online publication date: 28-Nov-2024
https://doi.org/10.1007/s00426-024-02065-1
Show More Cited By

Index Terms

LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples
1. Human-centered computing

Recommendations

LabintheWild: How to Design Uncompensated, Feedback-driven Online Experiments
CSCW '17 Companion: Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

We present LabintheWild, an online experiment platform that provides participants with the opportunity to learn about themselves and compare themselves to others. In the past four years, LabintheWild has attracted approximately 3.5 million participants ...
Volunteer-Based Online Studies With Older Adults and People with Disabilities
ASSETS '18: Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility

There are few large-scale empirical studies with people with disabilities or older adults, mainly because recruiting participants with specific characteristics is even harder than recruiting young and/or non-disabled populations. Analyzing four online ...
Types of Motivation Affect Study Selection, Attention, and Dropouts in Online Experiments

Understanding whether and how motivation affects participation in online experiments is critical because who contributes and how they contribute can affect the validity of findings. Analyzing data from 7,674 participants across three different studies on ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '15: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing

February 2015

1956 pages

ISBN:9781450329224

DOI:10.1145/2675133

General Chairs:
Dan Cosley
Cornell University, USA
,
Andrea Forte
Drexel University, USA
,
Program Chairs:
Luigina Ciolfi
Sheffield Hallam University, UK
,
David McDonald
University of Washington, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 February 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Alfred P. Sloan Foundation
Harvard Mind Brain and Behavior Initiative
Swiss National Science Foundation

Conference

CSCW '15

Sponsor:

SIGCHI

CSCW '15: Computer Supported Cooperative Work and Social Computing

March 14 - 18, 2015

BC, Vancouver, Canada

Acceptance Rates

CSCW '15 Paper Acceptance Rate 161 of 575 submissions, 28%;

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Sponsor:
sigchi

Computer-Supported Cooperative Work and Social Computing

October 18 - 22, 2025

Bergen , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

84
Total Citations
View Citations
1,247
Total Downloads

Downloads (Last 12 months)158
Downloads (Last 6 weeks)21

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lin HHan TGajos KGupta A(2024)Hevelius Report: Visualizing Web-Based Mobility Test Data For Clinical Decision and Learning SupportProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688490(1-10)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3688490
Rodd J(2024)Moving experimental psychology online: How to obtain high quality data when we can’t see our participantsJournal of Memory and Language10.1016/j.jml.2023.104472134(104472)Online publication date: Feb-2024
https://doi.org/10.1016/j.jml.2023.104472
Warburton MCampagnoli CMon-Williams MMushtaq FMorehead J(2024)Input device matters for measures of behaviour in online experimentsPsychological Research10.1007/s00426-024-02065-189:1Online publication date: 28-Nov-2024
https://doi.org/10.1007/s00426-024-02065-1
Chamunorwa MWozniak MKrämer SMüller HBoll S(2023)An Empirical Comparison of Moderated and Unmoderated Gesture Elicitation Studies on Soft Surfaces and Objects for Smart Home ControlProceedings of the ACM on Human-Computer Interaction10.1145/36042457:MHCI(1-24)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604245
Pang RCenatempo JGraham FKuehn BWhisenant MBotchway PStone Perez KKoenecke A(2023)Auditing Cross-Cultural Consistency of Human-Annotated Labels for Recommendation SystemsProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594098(1531-1552)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594098
Pandey VKhan NGupta AGajos K(2023)Accuracy and Reliability of At-Home Quantification of Motor Impairments Using a Computer-Based Pointing Task with Children with Ataxia-TelangiectasiaACM Transactions on Accessible Computing10.1145/358179016:1(1-25)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3581790
Bradford BSpikol DViberg OMartínez-Monés AGuo P(2023)Does STEM Success Start Young? Exploring Higher Ed Students' Early Academic Experiences in Science and Math at ScaleProceedings of the Tenth ACM Conference on Learning @ Scale10.1145/3573051.3596189(368-372)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1145/3573051.3596189
Do KPang RJiang JReinecke K(2023)“That’s important, but...”: How Computer Science Researchers Anticipate Unintended Consequences of Their Research InnovationsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581347(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581347
Stokes JAugust TMarver RCzeskis ARoesner FKohno TReinecke K(2023)How Language Formality in Security and Privacy Interfaces Impacts Intended ComplianceProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581275(1-12)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581275
Choi JOh CKim YKim N(2023)VisLab: Enabling Visualization Designers to Gather Empirically Informed Design FeedbackProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581132(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581132
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents