More Web Proxy on the site http://driver.im/

research-article

Don't hide in the crowd!: increasing social transparency between peer workers improves crowdsourcing outcomes

Authors:

Shih-Wen Huang,

Wai-Tat FuAuthors Info & Claims

CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 621 - 630

https://doi.org/10.1145/2470654.2470743

Published: 27 April 2013 Publication History

Abstract

This paper studied how social transparency and different peer-dependent reward schemes (i.e., individual, teamwork, and competition) affect the outcomes of crowdsourcing. The results showed that when social transparency was increased by asking otherwise anonymous workers to share their demographic information (e.g., name, nationality) to the paired worker, they performed significantly better. A more detailed analysis showed that in a teamwork reward scheme, in which the reward of the paired workers depended only on the collective outcomes, increasing social transparency could offset effects of social loafing by making them more accountable to their teammates. In a competition reward scheme, in which workers competed against each other and the reward depended on how much they outperformed their opponent, increasing social transparency could augment effects of social facilitation by providing more incentives for them to outperform their opponent. The results suggested that a careful combination of methods that increase social transparency and different reward schemes can significantly improve crowdsourcing outcomes.

References

[1]

Alonso, O., Rose, D. E., and Stewart, B. Crowdsourcing for relevance evaluation. SIGIR Forum 42, 2 (Nov. 2008), 9--15.

Digital Library

[2]

Bandiera, O., Barankay, I., and Rasul, I. Social preferences and the response to incentives: Evidence from personnel data. The Quarterly Journal of Economics 120, 3 (2005), 917--962.

[3]

Dow, S., Kulkarni, A., Klemmer, S., and Hartmann, B. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, CSCW '12, ACM (2012), 1013--1022.

Digital Library

[4]

Erickson, T., Halverson, C., Kellogg, W. A., Laff, M., and Wolf, T. Social translucence: designing social infrastructures that make collective activity visible. Commun. ACM 45, 4 (Apr. 2002), 40--44.

Digital Library

[5]

Erickson, T., and Kellogg, W. A. Social translucence: an approach to designing systems that support social processes. ACM Trans. Comput.-Hum. Interact. 7, 1 (Mar. 2000), 59--83.

Digital Library

[6]

Festinger, L. A theory of social comparison processes. Human relations 7, 2 (1954), 117--140.

[7]

Harkins, S. G. Social loafing and social facilitation. Journal of Experimental Social Psychology 23, 1 (1987), 1--18.

[8]

Howe, J. The rise of crowdsourcing. Wired Magazine (06 2006).

[9]

Huang, S.-W., and Fu, W.-T. Systematic analysis of output agreement games: Effects of gaming environment, social interaction, and feedback. In Proceedings of HCOMP12: The 4th Workshop on Human Computation (2012).

[10]

Huang, S.-W., and Fu, W.-T. Enhancing reliability using peer consistency evaluation in human computation. In Proceedings of the ACM 2013 conference on Computer Supported Cooperative Work, CSCW '13, ACM (2013).

Digital Library

[11]

Ipeirotis, P. G., Provost, F., and Wang, J. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP '10, ACM (2010), 64--67.

Digital Library

[12]

Karau, S., and Williams, K. Social loafing: A meta-analytic review and theoretical integration. Journal of Personality and Social Psychology 65, 4 (1993), 681--706.

[13]

Kerr, N. L. Motivation losses in small groups: A social dilemma analysis. Journal of Personality and Social Psychology 45 (1983), 819--828.

[14]

Kittur, A., Smus, B., Khamkar, S., and Kraut, R. E. Crowdforge: crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology, UIST '11, ACM (2011), 43--52.

Digital Library

[15]

Kittur, A., Suh, B., and Chi, E. H. Can you ever trust a wiki?: impacting perceived trustworthiness in wikipedia. In Proceedings of the 2008 ACM conference on Computer supported cooperative work, CSCW '08, ACM (2008), 477--480.

Digital Library

[16]

Law, E., and Von Ahn, L. Human Computation. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2011.

Digital Library

[17]

Mas, A., and Moretti, E. Peers at work. Working Paper 12508, National Bureau of Economic Research, September 2006.

[18]

Mason, W., and Watts, D. J. Financial incentives and the "performance of crowds". SIGKDD Explor. Newsl. 11, 2 (May 2010), 100--108.

Digital Library

[19]

Millen, D. R., and Patterson, J. F. Identity disclosure and the creation of social capital. In CHI '03 extended abstracts on Human factors in computing systems, CHI EA '03, ACM (2003), 720--721.

Digital Library

[20]

Postmes, T., Spears, R., Lee, A., and Novak, R. Individuality and social influence in groups: inductive and deductive routes to group identity. Journal of personality and social psychology 89, 5 (2005), 747.

[21]

Quinn, A. J., and Bederson, B. B. Human computation: a survey and taxonomy of a growing field. In Proceedings of the 2011 annual conference on Human factors in computing systems, CHI '11, ACM (2011), 1403--1412.

Digital Library

[22]

Robertson, S., Vojnovic, M., and Weber, I. Rethinking the esp game. In Proceedings of the 27th international conference extended abstracts on Human factors in computing systems, CHI EA '09, ACM (2009), 3937--3942.

Digital Library

[23]

Snow, R., O'Connor, B., Jurafsky, D., and Ng, A. Y. Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '08, Association for Computational Linguistics (Stroudsburg, PA, USA, 2008), 254--263.

Digital Library

[24]

Stuart, H. C., Dabbish, L., Kiesler, S., Kinnaird, P., and Kang, R. Social transparency in networked information exchange: a theoretical framework. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, CSCW '12, ACM (2012), 451--460.

Digital Library

[25]

Suh, B., Chi, E. H., Kittur, A., and Pendleton, B. A. Lifting the veil: improving accountability and social transparency in wikipedia with wikidashboard. In Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, CHI '08 (2008).

Digital Library

[26]

von Ahn, L., and Dabbish, L. Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems, CHI '04, ACM (2004), 319--326.

Digital Library

[27]

von Ahn, L., and Dabbish, L. Designing games with a purpose. Commun. ACM 51 (Aug. 2008), 58--67.

Digital Library

[28]

Zajonc, R. Social facilitation. Science 17 (1965), 65.

Cited By

Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Liu Y(2023)A New Framework for Evaluating Social Transparency Factors and Personal Brands in Social NetworksSocial Indicators Research10.1007/s11205-023-03289-1171:2(701-728)Online publication date: 30-Dec-2023
https://doi.org/10.1007/s11205-023-03289-1
Lykourentzou IVinella FAhmed FPapastathis CPapangelis KKhan VMasthoff J(2022)Self-organization in online collaborative work settingsCollective Intelligence10.1177/263391372210780051:1(263391372210780)Online publication date: 9-Sep-2022
https://doi.org/10.1177/26339137221078005
Show More Cited By

Index Terms

Don't hide in the crowd!: increasing social transparency between peer workers improves crowdsourcing outcomes
1. Human-centered computing

Recommendations

Motivating crowds using social facilitation and social transparency
CSCW '13: Proceedings of the 2013 conference on Computer supported cooperative work companion

We reported results from an experiment using an image labeling task, when workers were able to compare their own labels with the labels generated by another worker, they were motivated to generate more labels. In addition, when the workers shared their ...
Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool

Real-time collaboration between a requester and crowd workers expands the scope of tasks that crowdsourcing can be used for by letting requesters and crowd workers interactively create various artifacts (e.g., a sketch prototype, writing, or program code)...
Who's the boss?: requester transparency and motivation in a microtask marketplace
CHI EA '14: CHI '14 Extended Abstracts on Human Factors in Computing Systems

Workers on crowdsourcing platforms such as Amazon's Mechanical Turk often receive little to no information about who is requesting the task they are asked to perform. This can lead to psychological distance and reduced work quality as a result. In this ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

April 2013

3550 pages

ISBN:9781450318990

DOI:10.1145/2470654

General Chair:
Wendy E. Mackay
INRIA
,
Program Chairs:
Stephen Brewster
Glasgow University
,
Susanne Bødker
University of Aarhus

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 April 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '13

Sponsor:

SIGCHI

CHI '13: CHI Conference on Human Factors in Computing Systems

April 27 - May 2, 2013

Paris, France

Acceptance Rates

CHI '13 Paper Acceptance Rate 392 of 1,963 submissions, 20%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

43
Total Citations
View Citations
1,109
Total Downloads

Downloads (Last 12 months)35
Downloads (Last 6 weeks)1

Reflects downloads up to 19 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Liu Y(2023)A New Framework for Evaluating Social Transparency Factors and Personal Brands in Social NetworksSocial Indicators Research10.1007/s11205-023-03289-1171:2(701-728)Online publication date: 30-Dec-2023
https://doi.org/10.1007/s11205-023-03289-1
Lykourentzou IVinella FAhmed FPapastathis CPapangelis KKhan VMasthoff J(2022)Self-organization in online collaborative work settingsCollective Intelligence10.1177/263391372210780051:1(263391372210780)Online publication date: 9-Sep-2022
https://doi.org/10.1177/26339137221078005
Sun YMa XYe KHe L(2022)Investigating Crowdworkers' Identify, Perception and Practices in Micro-Task CrowdsourcingProceedings of the ACM on Human-Computer Interaction10.1145/34928546:GROUP(1-20)Online publication date: 14-Jan-2022
https://dl.acm.org/doi/10.1145/3492854
Ko EChoi EJang JKim J(2022)Capturing Diverse and Precise Reactions to a Comment with User-Generated LabelsProceedings of the ACM Web Conference 202210.1145/3485447.3512243(1731-1740)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3485447.3512243
Ehsan ULiao QMuller MRiedl MWeisz JKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Expanding Explainability: Towards Social Transparency in AI systemsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445188(1-19)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445188
Zhao YZheng KYin HLiu GFang JZhou X(2020)Preference-aware Task Assignment in Spatial Crowdsourcing: from Individuals to GroupsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2020.3021028(1-1)Online publication date: 2020
https://doi.org/10.1109/TKDE.2020.3021028
Li TManns CNorth CLuther K(2019)Dropping the Baton?Proceedings of the ACM on Human-Computer Interaction10.1145/33592383:CSCW(1-26)Online publication date: 7-Nov-2019
https://dl.acm.org/doi/10.1145/3359238
d'Eon GGoh JLarson KLaw E(2019)Paying Crowd Workers for Collaborative WorkProceedings of the ACM on Human-Computer Interaction10.1145/33592273:CSCW(1-24)Online publication date: 7-Nov-2019
https://dl.acm.org/doi/10.1145/3359227
Haltofová B(2019)Critical success factors of geocrowdsourcing use in e-government: a case study from the Czech RepublicUrban Research & Practice10.1080/17535069.2019.158699013:4(434-451)Online publication date: 25-Mar-2019
https://doi.org/10.1080/17535069.2019.1586990
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents