More Web Proxy on the site http://driver.im/

research-article

Subcontracting Microwork

Authors:

Meredith Ringel Morris,

Jeffrey P. Bigham,

Jonathan Bragg,

Anand Kulkarni,

Saiph SavageAuthors Info & Claims

CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

Pages 1867 - 1876

https://doi.org/10.1145/3025453.3025687

Published: 02 May 2017 Publication History

Abstract

Mainstream crowdwork platforms treat microtasks as indivisible units; however, in this article, we propose that there is value in re-examining this assumption. We argue that crowdwork platforms can improve their value proposition for all stakeholders by supporting subcontracting within microtasks. After describing the value proposition of subcontracting, we then define three models for microtask subcontracting: real-time assistance, task management, and task improvement, and reflect on potential use cases and implementation considerations associated with each. Finally, we describe the outcome of two tasks on Mechanical Turk meant to simulate aspects of subcontracting. We reflect on the implications of these findings for the design of future crowd work platforms that effectively harness the potential of subcontracting workflows.

Supplementary Material

suppl.mov (pn2180p.mp4)

Supplemental video

Download
721.93 KB

References

[1]

Bederson, B. B., Quinn A. J. Web workers unite! Addressing challenges of online laborers. Proceedings of CHI 2011

Digital Library

[2]

Belkin, N. J. Helping People Find What They Don't Know. Communications of the ACM, 43(8), August 2000, p. 58--61.

Digital Library

[3]

Berg, J. Income Security in the On-Demand Economy: Findings and Policy Lessons from a Survey of Crowdworkers. Comparative Labor Law & Policy Journal, Vol. 37, No. 3, March 2016.

[4]

Blei, D., Ng., A., and Jordan, M. 2003. Latent Dirichlet Allocation. The Journal of Machine Learning Research, 3, 993--1022.

[5]

Brady, E., Morris, M. R., and Bigham, J. P. Gauging Receptiveness to Social Microvolunteering. Proceedings of CHI 201

Digital Library

[6]

Brewer, R., Morris, M. R., and Piper, A. M. "Why would anybody do this?": Older Adults' Understanding of and Experiences with Crowdwork. Proceedings of CHI 201

Digital Library

[7]

Callison-Burch, C. Crowd-Workers: Aggregating Information Across Turkers To Help Them Find Higher Paying Work. HCOMP 2014 Extended Abstracts.

[8]

Callison-Burch, C. Fast, cheap, and creative: evaluating translation quality using Amazon's Mechanical Turk. Proceedings of EMNLP 2009.

[9]

Chilana, P. K., Ko, A. J., and Wobbrock, J. O. LemonAid: Selection-Based Crowdsourced Contextual Help for Web Applications. Proceedings of CHI 2012.

Digital Library

[10]

Clery, D. Galaxy Zoo Volunteers Share Pain and Glory of Research. Science, July 2011.

[11]

Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popovic, Z., and Foldit Players. Predicting Structures with a Multiplayer Online Game. Nature (2010).

[12]

Dow, S. P., Kulkarni, A., Klemmer, S. R., and Hartmann, B. Shepherding the Crowd Yields Better Work. Proceedings of CSCW 20

Digital Library

[13]

Dynamo Wiki. "Fair Payment". Retrieved December 12, 2016 at http://wiki.wearedynamo.org/index.php?title=Fair_payment

[14]

Flanagan, J. The critical incident technique. Psychological Bulletin, 51:327--358, 1954.

[15]

Gray, M. L., Suri, S., Ali, S. S., Kulkarni, D. The Crowd is a Collaborative Network. Proceedings of CSCW 2016.

Digital Library

[16]

Hanrahan, B. V., Willamowski, J. K., Swaminathan, S., and Martin, D. B. TurkBench: Rendering the Market for Turkers. Proceedings of CHI 2015.

Digital Library

[17]

Ipeirotis, P. G. Analyzing the Amazon Mechanical Turk Marketplace. ACM Crossroads, 17(2), Winter 2010, p. 16--21.

Digital Library

[18]

Irani, L. and Silberman, M. S. Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk. Proceedings of CHI 2013.

Digital Library

[19]

Kittur, A., Khamkar, S., André, P., and Kraut, R. CrowdWeaver: Visually Managing Complex Crowd Work. Proceedings of CSCW 2012.

Digital Library

[20]

Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., and Horton, J. The Future of Crowd Work. Proceedings of CSCW 2013.

Digital Library

[21]

Kobayashi, M., Ishihara, T., and Itoko, T. Age-Based Task Specialization for Crowdsourced Proofreading. Proceedings of HCII 2013.

Digital Library

[22]

Kulkarni, A., Can, M., and Hartmann, B. Collaboratively Crowdsourcing Workflows with Turkomatic. Proceedings of CSCW 2012.

Digital Library

[23]

Lakhani, K. R. and von Hippel, E. How Open Source Software Works: "Free" User-to-User Assistance. Research Policy, 32, 2003, p. 923--943.

[24]

Lasecki, W. S., Murray, K. I., White, S., Miller, R. C., and Bigham, J. P. Real-Time Crowd Control of Existing Interfaces. Proceedings of UIST 2011.

Digital Library

[25]

Luther, K. Fiesler, C., and Bruckman, A. Redistributing Leadership in Online Creative Collaboration. Proceedings of CSCW 2013.

Digital Library

[26]

Luther, K., Tolentino, J., Wu, W., Pavel, A., Agrawala, M., Bailey, B., Hartmann, B., and Dow, S. Structuring, Aggregating, and Evaluating Crowdsourced Design Critique. Proceedings of CSCW 2015.

Digital Library

[27]

Martin, D., Hanrahan, B. V., O'Neill, J., and Gupta, N. Being a Turker. Proceedings of CSCW 2014.

Digital Library

[28]

McInnis, B., Cosley, D., Nam, C., and Leshed, G. Taking a HIT: Designing around Rejection, Mistrust, Risk, and Workers' Experiences in Amazon Mechanical Turk. Proceedings of CHI 2016.

Digital Library

[29]

Morris, M. R., Teevan, J., and Panovich, K. What Do People Ask Their Social Networks, and Why? A Survey Study of Status Message Q&A Behavior. Proceedings of CHI 2010.

[30]

Papoutsaki, A., Guo, H., Metaxa-Kakavouli, D., Gramazio, C., Rasley, J., Xie, W., Wang, G., and Huang, J. Crowdsourcing from Scratch: A Pragmatic Experiment in Data Collection by Novice Requesters. Proceedings of HCOMP 2015.

[31]

Plante, C. Doritos and the Decade-Long Scame for Free Super Bowl Commercials. The Verge, Feb 3., 2016.

[32]

Rzeszotarski, J. and Kittur, A. CrowdScape: Interactively Visualizing User Behavior and Output. Proceedings of UIST 2012.

Digital Library

[33]

Singh, V., Twidale, M. B., Rathi D. Open Source Technical Support: A Look at Peer Help-Giving. Proceedings of HICSS 2006.

Digital Library

[34]

Stanford Crowd Research Collective. Daemo: A Self-Governed Crowdsourcing Marketplace. UIST 2015 Extended Abstracts.

[35]

Suzuki, R., Salehi, N., Lam, M. S., Marroquin, J. C., and Bernstein, M. S. Atelier: Repurposing Expert Crowdsourcing Tasks as Micro-internships. Proceedings of CHI 2016.

Digital Library

[36]

United Nations. Convention on the Rights of Persons with Disabilities. December 2006.

[37]

Vakharia, D. and Lease, M. Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms. Proceedings of iConference 2015.

[38]

Whiting, M. et al. Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms. Proceedings of CSCW 2017.

Digital Library

[39]

Zyskowski, K., Morris, M. R., Bigham, J. P., Gray, M. L., and Kane, S. Accessible Crowdwork? Understanding the Value in and Challenge of Microtask Employment for People with Disabilities. Proceedings of CSCW 2015.

Digital Library

Cited By

Stureborg RDhingra BYang J(2023)Interface Design for Crowdsourcing Hierarchical Multi-Label Text AnnotationsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581431(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581431
YILDIZ S(2022)A Research on Freelancers as Digital Advertising Practitionersİletişim Kuram ve Araştırma Dergisi10.47998/ikad.1114445Online publication date: 14-Jun-2022
https://doi.org/10.47998/ikad.1114445
Veezhinathan K(2022)Building the SHAKTI microprocessorCommunications of the ACM10.1145/355663265:11(48-51)Online publication date: 20-Oct-2022
https://dl.acm.org/doi/10.1145/3556632
Show More Cited By

Index Terms

Subcontracting Microwork
1. Human-centered computing
  1. Collaborative and social computing

Recommendations

Exploring microwork opportunities through cybercafés
ACM DEV '12: Proceedings of the 2nd ACM Symposium on Computing for Development

Microwork in cybercafés is a promising tool for poverty alleviation. Using experiments in three cafes in Pune, India, and two cafes in Nairobi, Kenya, we investigate questions of practical feasibility. In surveys, 99% of potential workers do want the ...
How many crowdsourced workers should a requester hire?

Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a ...
TasRec: a framework for task recommendation in crowdsourcing
ICGSE '20: Proceedings of the 15th International Conference on Global Software Engineering

Lately, crowdsourcing has emerged as a viable option of getting work done by leveraging the collective intelligence of the crowd. With many tasks posted every day, the size of crowdsourcing platforms is growing exponentially. Hence, workers face an ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

May 2017

7138 pages

ISBN:9781450346559

DOI:10.1145/3025453

General Chairs:
Gloria Mark
University of California Irvine
,
Susan Fussell
Cornell University
,
Program Chairs:
Cliff Lampe
University of Michigan
,
m.c. schraefel
University of Southampton
,
Juan Pablo Hourcade
University of Iowa
,
Caroline Appert
Université Paris-Sud
,
Daniel Wigdor
University of Toronto

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '17

Sponsor:

SIGCHI

CHI '17: CHI Conference on Human Factors in Computing Systems

May 6 - 11, 2017

Colorado, Denver, USA

Acceptance Rates

CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
535
Total Downloads

Downloads (Last 12 months)79
Downloads (Last 6 weeks)1

Reflects downloads up to 31 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Stureborg RDhingra BYang J(2023)Interface Design for Crowdsourcing Hierarchical Multi-Label Text AnnotationsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581431(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581431
YILDIZ S(2022)A Research on Freelancers as Digital Advertising Practitionersİletişim Kuram ve Araştırma Dergisi10.47998/ikad.1114445Online publication date: 14-Jun-2022
https://doi.org/10.47998/ikad.1114445
Veezhinathan K(2022)Building the SHAKTI microprocessorCommunications of the ACM10.1145/355663265:11(48-51)Online publication date: 20-Oct-2022
https://dl.acm.org/doi/10.1145/3556632
Rechkemmer AYin M(2022)Understanding the Microtask Crowdsourcing Experience for Workers with Disabilities: A Comparative ViewProceedings of the ACM on Human-Computer Interaction10.1145/35551376:CSCW2(1-30)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555137
Mukhopadhyay D(2022)Cybersecurity in IndiaCommunications of the ACM10.1145/355491065:11(98-102)Online publication date: 20-Oct-2022
https://dl.acm.org/doi/10.1145/3554910
Darke PMetta RMedicherla RVenkatesh R(2022)Impactful research and tooling for program correctnessCommunications of the ACM10.1145/355166565:11(52-53)Online publication date: 20-Oct-2022
https://dl.acm.org/doi/10.1145/3551665
Balakrishnan M(2022)Computing and assistive technology solutions for the visually impairedCommunications of the ACM10.1145/355163465:11(44-47)Online publication date: 20-Oct-2022
https://dl.acm.org/doi/10.1145/3551634
Aiswarya CArvind VSaurabh S(2022)Theory research in IndiaCommunications of the ACM10.1145/355049665:11(88-93)Online publication date: 20-Oct-2022
https://dl.acm.org/doi/10.1145/3550496
Balasubramanian V(2022)Toward explainable deep learningCommunications of the ACM10.1145/355049165:11(68-69)Online publication date: 20-Oct-2022
https://dl.acm.org/doi/10.1145/3550491
Karkare AKar P(2022)PrutorCommunications of the ACM10.1145/355048165:11(62-64)Online publication date: 20-Oct-2022
https://dl.acm.org/doi/10.1145/3550481
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents