Classification Model based on Intelligence for Crowdsourced Data Labeling: Intelligence Elicitation by Information Interaction Mechanism
Pages 12 - 17
Abstract
In crowdsourced data labeling tasks, the labels given by annotators are considered as deviated versions of the ground-truth. We believe that the intelligence of annotators is the factor that determines the degree of deviation. Intelligence can be regarded as a measure of the reliability of the annotators, and it can be indirectly estimated through the effect of social interaction. Accordingly, we propose an estimation method for the intelligence of annotators in the crowdsourced labeling process, to improve the accuracy of label inferring through the mutual heuristic effect of information interaction mechanism. Finally, we build a classification model based on annotator-intelligence, which aggregates the various knowledge of multiple annotators, hence the ground-truth of the label can be inferred through a probabilistic approach.
References
[1]
Bahador Bahrami, Karsten Olsen, Peter E Latham, Andreas Roepstorff, Geraint Rees, and Chris D Frith. 2010. Optimally interacting minds. Science 329, 5995 (2010), 1081–1085.
[2]
Felix C Brodbeck, Rudolf Kerschreiter, Andreas Mojzisch, and Stefan Schulz-Hardt. 2007. Group decision making under conditions of distributed knowledge: The information asymmetries model.Academy of Management Review 32, 2 (2007), 459–479.
[3]
Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics) 28, 1(1979), 20–28.
[4]
Meng Fang, Jie Yin, and Dacheng Tao. 2014. Active Learning for Crowdsourcing Using Knowledge Transfer. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence (Québec City, Québec, Canada) (AAAI’14). AAAI Press, 1809–1815.
[5]
Burcu Gürçay, Barbara A Mellers, and Jonathan Baron. 2015. The power of social influence on estimation accuracy. Journal of Behavioral Decision Making 28, 3 (2015), 250–261.
[6]
Ralph Hertwig. 2012. Tapping into the wisdom of the crowd—with confidence. Science 336, 6079 (2012), 303–304.
[7]
David R. Karger, Sewoong Oh, and Devavrat Shah. 2011. Iterative Learning for Reliable Crowdsourcing Systems. In Proceedings of the 24th International Conference on Neural Information Processing Systems (Granada, Spain) (NIPS’11). Curran Associates Inc., Red Hook, NY, USA, 1953–1961.
[8]
Asher Koriat. 2012. When are two heads better than one and why?Science 336, 6079 (2012), 360–362.
[9]
James R Larson Jr. 2013. In search of synergy in small group performance. Psychology Press.
[10]
Jing Li, Suiyi Ling, Junle Wang, Zhi Li, and Patrick Le Callet. 2020. GPM: A Generic Probabilistic Model to Recover Annotator’s Behavior and Ground Truth Labeling. arXiv preprint arXiv:2003.00475(2020).
[11]
Z. Li and C. G. Bampis. 2017. Recover Subjective Quality Scores from Noisy Measurements. In 2017 Data Compression Conference (DCC). IEEE Computer Society, Los Alamitos, CA, USA, 52–61.
[12]
Qiang Liu, Jian Peng, and Alexander Ihler. 2012. Variational Inference for Crowdsourcing. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1 (Lake Tahoe, Nevada) (NIPS’12). Curran Associates Inc., Red Hook, NY, USA, 692–700.
[13]
Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why is that relevant? collecting annotator rationales for relevance judgments. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 4.
[14]
Joaquin Navajas, Tamara Niella, Gerry Garbulsky, Bahador Bahrami, and Mariano Sigman. 2018. Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds. Nature Human Behaviour 2, 2 (2018), 126–132.
[15]
Vikas C Raykar and Shipeng Yu. 2011. Ranking annotators for crowdsourced labeling tasks. In Advances in neural information processing systems. 1809–1817.
[16]
Rion Snow, Brendan O’connor, Dan Jurafsky, and Andrew Y Ng. 2008. Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing. 254–263.
[17]
Wei Tang, Ming Yin, and Chien-Ju Ho. 2019. Leveraging peer communication to enhance crowdsourcing. In The World Wide Web Conference. 1794–1805.
[18]
Tian Tian and Jun Zhu. 2015. Max-Margin Majority Voting for Learning from Crowds. In Nips. 1621–1629.
[19]
Peter Welinder, Steve Branson, Pietro Perona, and Serge Belongie. 2010. The multidimensional wisdom of crowds. Advances in neural information processing systems 23 (2010), 2424–2432.
[20]
Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems 22 (2009), 2035–2043.
[21]
Yan Yan, Rómer Rosales, Glenn Fung, Mark Schmidt, Gerardo Hermosillo, Luca Bogoni, Linda Moy, and Jennifer Dy. 2010. Modeling annotator expertise: Learning when everybody knows a bit of something. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 932–939.
[22]
Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. Using “annotator rationales” to improve machine learning for text categorization. In Human language technologies 2007: The conference of the North American chapter of the association for computational linguistics; proceedings of the main conference. 260–267.
[23]
Denny Zhou, John C Platt, Sumit Basu, and Yi Mao. 2012. Learning from the wisdom of crowds by minimax entropy. (2012).
Recommendations
Label Aggregation with Clustering for Biased Crowdsourced Labeling
ICMLC '22: Proceedings of the 2022 14th International Conference on Machine Learning and ComputingWith the rapid development of crowdsourcing learning, amount of label aggregation methods are proposed to infer the true labels of instances from multiple noisy labels provided by inexpert crowd workers. Most of the label aggregation methods take the ...
Learning from biased crowdsourced labeling with deep clustering
Highlights- The phenomenon of biased labeling usually existing in the scenario of crowdsourcing.
- Biased labeling is a critical factor that effects label aggregation performance.
- Deep clustering estimates the underlying label distribution and ...
AbstractWith the rapid development of crowdsourcing learning, amount of labels can be obtained from crowd workers fast and cheaply. However, crowdsourcing learning also faces challenges due to the varied qualities of amateurish crowd workers. To improve ...
Eliminating spammers and ranking annotators for crowdsourced labeling tasks
With the advent of crowdsourcing services it has become quite cheap and reasonably effective to get a data set labeled by multiple annotators in a short amount of time. Various methods have been proposed to estimate the consensus labels by correcting ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2021
182 pages
ISBN:9781450395540
DOI:10.1145/3503181
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 March 2022
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
ICCSE '21
ICCSE '21: 5th International Conference on Crowd Science and Engineering
October 16 - 18, 2021
Jinan, China
Acceptance Rates
Overall Acceptance Rate 92 of 247 submissions, 37%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 174Total Downloads
- Downloads (Last 12 months)95
- Downloads (Last 6 weeks)19
Reflects downloads up to 14 Jan 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in