More Web Proxy on the site http://driver.im/

research-article

In What Mood Are You Today?: An Analysis of Crowd Workers' Mood, Performance and Engagement

Authors:

Mengdie Zhuang,

Ujwal GadirajuAuthors Info & Claims

WebSci '19: Proceedings of the 10th ACM Conference on Web Science

Pages 373 - 382

https://doi.org/10.1145/3292522.3326010

Published: 26 June 2019 Publication History

Abstract

The mood of individuals in the workplace has been well-studied due to its influence on task performance, and work engagement. However, the effect of mood has not been studied in detail in the context of microtask crowdsourcing. In this paper, we investigate the influence of one's mood, a fundamental psychosomatic dimension of a worker's behaviour, on their interaction with tasks, task performance and perceived engagement. To this end, we conducted two comprehensive studies; (i) a survey exploring the perception of crowd workers regarding the role of mood in shaping their work, and (ii) an experimental study to measure and analyze the actual impact of workers' moods in information findings microtasks. We found evidence of the impact of mood on a worker's perceived engagement through the feeling of reward or accomplishment, and we argue as to why the same impact is not perceived in the evaluation of task performance. Our findings have broad implications on the design and workflow of crowdsourcing systems.

References

[1]

Arnold B Bakker, Wilmar B Schaufeli, Michael P Leiter, and Toon W Taris. 2008. Work engagement: An emerging concept in occupational health psychology. Work & Stress, Vol. 22, 3 (2008), 187--200.

[2]

Christopher Beedie, Peter Terry, and Andrew Lane. 2005. Distinctions between emotion and mood. Cognition & Emotion, Vol. 19, 6 (2005), 847--878.

[3]

Borong Chen, Weiping Hu, and Jonathan A Plucker. 2016. The effect of mood on problem finding in scientific creativity. The Journal of Creative Behavior, Vol. 50, 4 (2016), 308--320.

[4]

Michael S. Christian, Adela S. Garza, and Jerel E. Slaughter. 2011. Work engagement: A quantitative review and test of its relations with task and contextual performance. Personnel Psychology, Vol. 64, 1 (2011), 89--136.

[5]

David M Clark, John D Teasdale, Donald E Broadbent, and Maryanne Martin. 1983. Effect of mood on lexical decisions. Bulletin of the Psychonomic Society, Vol. 21, 3 (1983), 175--178.

[6]

Peng Dai, Jeffrey M Rzeszotarski, Praveen Paritosh, and Ed H Chi. 2015. And now for something completely different: Improving crowdsourcing workflows with micro-diversions. In Proceeding of The 18th ACM Conference on Computer-Supported Cooperative Work and Social Computing. ACM, 628--638.

Digital Library

[7]

Richard J Davidson. 1994. On emotion, mood, and related affective constructs. The nature of emotion: Fundamental questions (1994), 51--55.

[8]

Gianluca Demartini, Djellel Eddine Difallah, Ujwal Gadiraju, Michele Catasta, et almbox. 2017. An introduction to hybrid human-machine information systems. Foundations and Trends® in Web Science, Vol. 7, 1 (2017), 1--87.

Digital Library

[9]

Pieter MA Desmet, Martijn H Vastenburg, and Natalia Romero. 2016. Mood measurement with Pick-A-Mood: review of current methods and design of a pictorial self-report scale. Journal of Design Research, Vol. 14, 3 (2016), 241--279.

[10]

Fernando Diaz, Ryen White, Georg Buscher, and Dan Liebling. 2013. Robust models of mouse movement on dynamic web search results pages. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management. ACM, 1451--1460.

Digital Library

[11]

Djellel Difallah, Elena Filatova, and Panos Ipeirotis. 2018. Demographics and Dynamics of Mechanical Turk Workers. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, New York, NY, USA, 135--143.

Digital Library

[12]

Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, and Philippe Cudré-Mauroux. 2014. Scaling-up the crowd: Micro-task pricing schemes for worker retention and latency improvement. In Second AAAI Conference on Human Computation and Crowdsourcing.

[13]

Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G Ipeirotis, and Philippe Cudré-Mauroux. 2015. The dynamics of micro-task crowdsourcing: The case of amazon mturk. In Proceedings of the 24th International Conference on World Wide Web. 238--247.

Digital Library

[14]

Peter Eckersley. 2010. How unique is your web browser?. In Privacy Enhancing Technologies. Springer, 1--18.

Digital Library

[15]

Alek Felstiner. 2011. Working the crowd: employment and labor law in the crowdsourcing industry. Berkeley J. Emp. & Lab. L., Vol. 32 (2011), 143.

[16]

Oluwaseyi Feyisetan, Elena Simperl, Max Van Kleek, and Nigel Shadbolt. 2015. Improving paid microtasks through gamification and adaptive furtherance incentives. In Proc. WWW'15. 333--343.

Digital Library

[17]

Nico H Frijda et almbox. 1994. Varieties of affect: Emotions and episodes, moods, and sentiments. (1994).

[18]

Ujwal Gadiraju, Alessandro Checco, Neha Gupta, and Gianluca Demartini. 2017a. Modus operandi of crowd workers: The invisible role of microtask work environments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 1, 3, 49.

Digital Library

[19]

Ujwal Gadiraju, Gianluca Demartini, Djellel Eddine Difallah, and Michele Catasta. 2016. It's getting crowded!: how to use crowdsourcing effectively for web science research. In Proceedings of the 8th ACM Conference on Web Science. ACM, 11--11.

Digital Library

[20]

Ujwal Gadiraju and Stefan Dietze. 2017. Improving learning through achievement priming in crowdsourced information finding microtasks. In proceedings of the Seventh International Learning Analytics & Knowledge Conference. ACM, 105--114.

Digital Library

[21]

Ujwal Gadiraju, Besnik Fetahu, Ricardo Kawase, Patrick Siehndel, and Stefan Dietze. 2017b. Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks. ACM Transactions on Computer-Human Interaction, Vol. 24, 4, Article 30 (Aug. 2017), bibinfonumpages26 pages.

Digital Library

[22]

Ujwal Gadiraju, Ricardo Kawase, and Stefan Dietze. 2014. A taxonomy of microtasks on the web. In Proceedings of the 25th ACM conference on Hypertext and social media. ACM, 218--223.

Digital Library

[23]

Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, and Gianluca Demartini. 2015. Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1631--1640.

Digital Library

[24]

Ujwal Gadiraju, Jie Yang, and Alessandro Bozzon. 2017c. Clarity is a worthwhile quality: On the role of task clarity in microtask crowdsourcing. In Proceedings of the 28th ACM Conference on Hypertext and Social Media. ACM, 5--14.

Digital Library

[25]

J Richard Hackman and Greg R Oldham. 1976. Motivation through the design of work: Test of a theory. Organizational behavior and human performance, Vol. 16, 2 (1976), 250--279.

[26]

Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics (1979), 65--70.

[27]

Stephen E Humphrey, Jennifer D Nahrgang, and Frederick P Morgeson. 2007. Integrating motivational, social, and contextual work design features: a meta-analytic summary and theoretical extension of the work design literature. Journal of applied psychology, Vol. 92, 5 (2007), 1332.

[28]

William A Kahn. 1990. Psychological conditions of personal engagement and disengagement at work. Academy of management journal, Vol. 33, 4 (1990), 692--724.

[29]

Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk. In AMCIS, Vol. 11. 1--11.

[30]

Gabriella Kazai, Jaap Kamps, and Natasa Milic-Frayling. 2013. An analysis of human factors and label accuracy in crowdsourcing relevance judgments. Information retrieval, Vol. 16, 2 (2013), 138--178.

Digital Library

[31]

Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work. ACM, 1301--1318.

Digital Library

[32]

Walter S Lasecki, Samuel C White, Kyle I Murray, and Jeffrey P Bigham. 2012. Crowd memory: Learning in the collective. In Proceedings of the Collective Intelligence.

[33]

Richard Lazarus. 1994. The stable and the unstable in emotion. The nature of emotion: Fundamental questions (1994), 79--85.

[34]

Sheena Lewis, Mira Dontcheva, and Elizabeth Gerber. 2011. Affective computational priming and creativity. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 735--744.

Digital Library

[35]

Nangyeon Lim. 2016. Cultural differences in emotion: differences in emotional arousal level between the East and the West. Integrative Medicine Research, Vol. 5, 2 (2016), 105--109.

[36]

Andrew Mao, Ece Kamar, and Eric Horvitz. 2013. Why stop now? predicting worker engagement in online crowdsourcing. In First AAAI Conference on Human Computation and Crowdsourcing.

[37]

Catherine C Marshall and Frank M Shipman. 2013. Experiences surveying the crowd: Reflections on methods, participation, and reliability. In Proceedings of the 5th Annual ACM Web Science Conference. ACM, 234--243.

Digital Library

[38]

Panagiotis Mavridis, David Gross-Amblard, and Zoltán Miklós. 2016. Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourcing. In Proceedings of the 25th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 843--853.

Digital Library

[39]

Brian McInnis, Dan Cosley, Chaebong Nam, and Gilly Leshed. 2016. Taking a HIT: Designing around rejection, mistrust, risk, and workers' experiences in Amazon Mechanical Turk. In Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, 2271--2282.

Digital Library

[40]

Andrew G. Miner and Theresa M. Glomb. 2010. State mood, task performance, and behavior at work: A within-persons approach. Organizational Behavior and Human Decision Processes, Vol. 112, 1 (2010), 43 -- 57.

[41]

Robert R Morris, Mira Dontcheva, Adam Finkelstein, and Elizabeth Gerber. 2013a. Affect and creative performance on crowdsourcing platforms. In Proceedings of the 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. 67--72.

Digital Library

[42]

Robert R Morris, Mira Dontcheva, Adam Finkelstein, and Elizabeth Gerber. 2013b. Affect and creative performance on crowdsourcing platforms. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. IEEE, 67--72.

Digital Library

[43]

Robert R Morris, Mira Dontcheva, and Elizabeth M Gerber. 2012. Priming for better performance in microtask crowdsourcing environments. IEEE Internet Computing, Vol. 16, 5 (2012), 13--19.

Digital Library

[44]

William N Morris. 2012. Mood: The frame of mind .Springer Sci. & Business Media.

[45]

Heather L. O'Brien, Paul Cairns, and Mark Hall. 2018. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. Intl. J. Human-Computer Studies, Vol. 112 (2018), 28 -- 39.

[46]

Peter Organisciak, Jaime Teevan, Susan Dumais, Robert C Miller, and Adam Tauman Kalai. 2014. A crowd of your own: Crowdsourcing for on-demand personalization. In Second AAAI Conference on Human Computation and Crowdsourcing.

[47]

Andrew J Oswald, Eugenio Proto, and Daniel Sgroi. 2015. Happiness and productivity. Journal of Labor Economics, Vol. 33, 4 (2015), 789--822.

[48]

Antonia Pilar Pacheco-Unguetti and Fabrice B. R. Parmentier. 2016. Happiness increases distraction by auditory deviant stimuli. British Journal of Psychology, Vol. 107, 3 (2016), 419--433.

[49]

Jesse J Prinz. 2004. Gut reactions: A perceptual theory of emotion .Oxford UP.

[50]

Markus Rokicki, Sergiu Chelaru, Sergej Zerr, and Stefan Siersdorfer. 2014. Competitive game designs for improving the cost effectiveness of crowdsourcing. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. ACM, 1469--1478.

Digital Library

[51]

Jeffrey Rzeszotarski and Aniket Kittur. 2012. CrowdScape: interactively visualizing user behavior and output. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 55--62.

Digital Library

[52]

Jeffrey M Rzeszotarski, Ed Chi, Praveen Paritosh, and Peng Dai. 2013. Inserting micro-breaks into crowdsourcing workflows. In First AAAI Conference on Human Computation and Crowdsourcing.

[53]

Jeffrey M Rzeszotarski and Aniket Kittur. 2011. Instrumenting the crowd: using implicit behavioral measures to predict task performance. In Proceedings of the 24th Annual ACM Symposium on User Interface Software & Technology. ACM, 13--22.

Digital Library

[54]

Pennie S Seibert and Henry C Ellis. 1991. Irrelevant thoughts, emotional mood states, and cognitive task performance. Memory & Cognition, Vol. 19, 5 (1991), 507--513.

[55]

Han Yu Zhiqi Shen, Simon Fauvel, and Lizhen Cui. 2017. Efficient scheduling in crowdsourcing based on workers' mood. In Agents (ICA), 2017 IEEE International Conference on. IEEE, 121--126.

[56]

Melanie J Taylor and Peter J Cooper. 1992. An experimental study of the effect of mood on body size perception. Behaviour research and therapy, Vol. 30, 1 (1992), 53--58.

[57]

Elizabeth R Tenney, Jared M Poole, and Ed Diener. 2016. Does positivity enhance work performance?: Why, when, and what we don't know. Research in Organizational Behavior, Vol. 36 (2016), 27--46.

[58]

Wei-Chi Tsai, Chien-Cheng Chen, and Hui-Lu Liu. 2007. Test of a model linking employee positive moods and task performance. Journal of Applied Psychology, Vol. 92, 6 (2007), 1570.

[59]

Viswanath Venkatesh and Cheri Speier. 1999. Computer technology training in the workplace: A longitudinal investigation of the effect of mood. Organizational behavior and human decision processes, Vol. 79, 1 (1999), 1--28.

[60]

Philippe Verduyn, Iven Van Mechelen, and Francis Tuerlinckx. 2011. The relation between event processing and the duration of emotional experience. Emotion, Vol. 11, 1 (2011), 20.

[61]

Bjørn Villa, Katrien De Moor, Poul Heegaard, and Anders Instefjord. 2013. Investigating quality of experience in the context of adaptive video streaming: findings from an experimental user study. In Norsk informatikkonferanse NIK 2013, Universitetet i Stavanger, 18.-20. november 2013. Akademika forlag, 122--133.

[62]

David Watson and Auke Tellegen. 1985. Toward a consensual structure of mood. Psychological bulletin, Vol. 98, 2 (1985), 219.

[63]

Jie Yang, Judith Redi, Gianluca Demartini, and Alessandro Bozzon. 2016. Modeling task complexity in crowdsourcing. In Fourth AAAI Conference on Human Computation and Crowdsourcing.

Cited By

de Groot EGadiraju U(2024)"Are we all in the same boat?" Customizable and Evolving Avatars to Improve Worker Engagement and Foster a Sense of Community in Online Crowd WorkProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642429(1-26)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642429
N V GM RN JLokesh A(2024)Human AI Collaboration for Backend Text Generation: Dynamic Content Recommendation (DCR) for Websites Based on Keywords2024 International Conference on Computing and Data Science (ICCDS)10.1109/ICCDS60734.2024.10560437(1-6)Online publication date: 26-Apr-2024
https://doi.org/10.1109/ICCDS60734.2024.10560437
Mourelatos E(2024)Mood matters: the interplay of personality in ethical perceptions in crowdsourcingBehaviour & Information Technology10.1080/0144929X.2024.2349786(1-23)Online publication date: 17-May-2024
https://doi.org/10.1080/0144929X.2024.2349786
Show More Cited By

Index Terms

In What Mood Are You Today?: An Analysis of Crowd Workers' Mood, Performance and Engagement
1. Human-centered computing
2. Information systems

Recommendations

Understanding Worker Moods and Reactions to Rejection in Crowdsourcing
HT '19: Proceedings of the 30th ACM Conference on Hypertext and Social Media

Requesters on crowdsourcing platforms typically exercise the power to decide the fate of tasks completed by crowd workers. Rejecting work has a direct impact on workers; (i) they may not be rewarded for the work completed and for their effort that has ...
Improving Reactions to Rejection in Crowdsourcing Through Self-Reflection
WebSci '21: Proceedings of the 13th ACM Web Science Conference 2021

In popular crowdsourcing marketplaces like Amazon Mechanical Turk, crowd workers complete tasks posted by requesters in return for monetary rewards. Task requesters are solely responsible for deciding whether to accept or reject submitted work. ...
Generating ground truth for music mood classification using mechanical turk
JCDL '12: Proceedings of the 12th ACM/IEEE-CS joint conference on Digital Libraries

Mood is an important access point in music digital libraries and online music repositories, but generating ground truth for evaluating various music mood classification algorithms is a challenging problem. This is because collecting enough human ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WebSci '19: Proceedings of the 10th ACM Conference on Web Science

June 2019

395 pages

ISBN:9781450362023

DOI:10.1145/3292522

General Chairs:
Paolo Boldi
Università degli Studi, Milano, Italy
,
Brooke Foucault Welles
Northeastern University, Boston, USA
,
Katharina Kinder-Kurlanda
GESIS - Leibniz Institute for the Social Sciences, Cologne, Germany
,
Christo Wilson
Northeastern University, Boston, USA
,
Program Chairs:
Isabella Peters
ZBW Leibniz Information Center for Economics & Kiel University, Kiel, Germany
,
Wagner Meira
Universidade Federal de Minas Gerais, Belo Horizonte, Brazil

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

The Erasmus+ project DISKOW
the EU Horizon 2020 transnational access program under SoBigData

Conference

WebSci '19

Sponsor:

SIGWEB

WebSci '19: 11th ACM Conference on Web Science

June 30 - July 3, 2019

Massachusetts, Boston, USA

Acceptance Rates

WebSci '19 Paper Acceptance Rate 41 of 130 submissions, 32%;

Overall Acceptance Rate 245 of 933 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
566
Total Downloads

Downloads (Last 12 months)55
Downloads (Last 6 weeks)6

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

de Groot EGadiraju U(2024)"Are we all in the same boat?" Customizable and Evolving Avatars to Improve Worker Engagement and Foster a Sense of Community in Online Crowd WorkProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642429(1-26)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642429
N V GM RN JLokesh A(2024)Human AI Collaboration for Backend Text Generation: Dynamic Content Recommendation (DCR) for Websites Based on Keywords2024 International Conference on Computing and Data Science (ICCDS)10.1109/ICCDS60734.2024.10560437(1-6)Online publication date: 26-Apr-2024
https://doi.org/10.1109/ICCDS60734.2024.10560437
Mourelatos E(2024)Mood matters: the interplay of personality in ethical perceptions in crowdsourcingBehaviour & Information Technology10.1080/0144929X.2024.2349786(1-23)Online publication date: 17-May-2024
https://doi.org/10.1080/0144929X.2024.2349786
Vasconcelos LZahn JTrevisan DViterbo J(2024)Engagement by Design Cards: A tool to involve designers and non-experts in the design of crowdsourcing initiativesInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103166182(103166)Online publication date: Feb-2024
https://doi.org/10.1016/j.ijhcs.2023.103166
Zhao YZhu ZChen BQiu S(2023)Leveraging Human-AI Collaboration in Crowd-Powered Source Search: A Preliminary StudyJournal of Social Computing10.23919/JSC.2023.00024:2(95-111)Online publication date: Jun-2023
https://doi.org/10.23919/JSC.2023.0002
Maddalena EIbáñez LReeves NSimperl E(2023)Qrowdsmith: Enhancing Paid Microtask Crowdsourcing with Gamification and Furtherance IncentivesACM Transactions on Intelligent Systems and Technology10.1145/360494014:5(1-26)Online publication date: 21-Jun-2023
https://dl.acm.org/doi/10.1145/3604940
Starke AAsotic ETrattner CVan Loo E(2023)Examining the User Evaluation of Multi-List Recommender Interfaces in the Context of Healthy Recipe ChoicesACM Transactions on Recommender Systems10.1145/35819301:4(1-31)Online publication date: 24-Feb-2023
https://dl.acm.org/doi/10.1145/3581930
Wijenayake SHettiachchi DGoncalves J(2023)Combining Worker Factors for Heterogeneous Crowd Task AssignmentProceedings of the ACM Web Conference 202310.1145/3543507.3583190(3794-3805)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583190
Elbouzidi APellerin REl Cadi ALamouri SBoubaker S(2023)Digital Twin Modeling Framework for Manual Warehouses2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC53992.2023.10393966(3528-3533)Online publication date: 1-Oct-2023
https://doi.org/10.1109/SMC53992.2023.10393966
Samiotis IQiu SLofi CYang JGadiraju UBozzon A(2022)An Analysis of Music Perception Skills on Crowdsourcing PlatformsFrontiers in Artificial Intelligence10.3389/frai.2022.8287335Online publication date: 14-Jun-2022
https://doi.org/10.3389/frai.2022.828733
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents