More Web Proxy on the site http://driver.im/

research-article

Learning from biased crowdsourced labeling with deep clustering

Authors:

Victor S. Sheng,

Jun HouAuthors Info & Claims

Volume 211, Issue C

https://doi.org/10.1016/j.eswa.2022.118608

Published: 01 January 2023 Publication History

Highlights

•

The phenomenon of biased labeling usually existing in the scenario of crowdsourcing.

•

Biased labeling is a critical factor that effects label aggregation performance.

•

Deep clustering estimates the underlying label distribution and detect the bias.

Abstract

With the rapid development of crowdsourcing learning, amount of labels can be obtained from crowd workers fast and cheaply. However, crowdsourcing learning also faces challenges due to the varied qualities of amateurish crowd workers. To improve the quality of crowd labels, many researchers focus on inferring the ground truth from noisy labels, and take different factors, e.g. the reliability of workers and the difficulty of instances, into consideration to infer the aggregated labels. Nevertheless, to the best of our knowledge, label aggregation for biased crowdsourced labeling scenarios has not been sufficiently studied. Actually, the phenomenon of biased labeling exists in many crowdsourcing annotation tasks and affects the performance of label aggregation. To this end, this paper proposes a novel framework termed Biased Crowdsourcing Learning with Deep Clustering (BCLDC), which involves label aggregation and prediction using deep clustering to improve the quality of aggregated labels and learned models in biased labeling scenarios. BCLDC utilizes a deep clustering method to detect the labeling bias and then eliminates the bias by adjusting the number of labels belonging to the minority class which has fewer labels. Finally, a classifier is trained simultaneously with the aggregated labels inferred by an EM algorithm. Experimental results on six real-world datasets and five synthetic datasets consistently show that the proposed BCLDC outperforms other state-of-the-art algorithms in terms of ground truth inference and prediction.

References

[1]

Abassi, L., & Boukhris, I. (2019a). An evidential imprecise answer aggregation approach based on worker clustering. In International Conference on Intelligent Data Engineering and Automated Learning (pp. 341–349). Springer.

[2]

L. Abassi, I. Boukhris, A worker clustering-based approach of label aggregation under the belief function theory, Applied Intelligence 49 (2019) 53–62.

[3]

S. Albarqouni, C. Baur, F. Achilles, V. Belagiannis, S. Demirci, N. Navab, Aggnet: Deep learning from crowds for mitosis detection in breast cancer histology images, IEEE Transactions on Medical Imaging 35 (2016) 1313–1321.

[4]

M. Ankerst, M.M. Breunig, H.-P. Kriegel, J. Sander, Optics: Ordering points to identify the clustering structure, ACM Sigmod Record 28 (1999) 49–60.

[5]

Arthur, D., & Vassilvitskii, S. (2006). k-means++: The advantages of careful seeding. Technical Report Stanford.

[6]

A.P. Dawid, A.M. Skene, Maximum likelihood estimation of observer error-rates using the em algorithm, Journal of the Royal Statistical Society: Series C (Applied Statistics) 28 (1979) 20–28.

[7]

G. Demartini, D.E. Difallah, P. Cudre-Mauroux, Zencrowd: Leveraging probabilistic reasoning and crowd-sourcing techniques for large-scale entity linking, in: In Proceedings of the 21st international conference on World Wide Web, 2012, pp. 469–478.

[8]

A.P. Dempster, Maximum likelihood from incomplete data via the em algorithm, Journal of the Royal Statistical Society 39 (1977).

[9]

Y. Dong, L. Jiang, C. Li, Improving data and model quality in crowdsourcing using co-training-based noise correction, Information Sciences 583 (2022) 174–188.

[10]

Duran, B. S., & Odell, P. L. (2013). Cluster analysis: a survey volume 100. Springer Science & Business Media.

[11]

Ester, M., Kriegel, H.-P., Sander, J., Xu, X. et al. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd (pp. 226–231). volume 96.

[12]

V. Estivill-Castro, Why so many clustering algorithms – a position paper, Acm Sigkdd Explorations Newsletter 4 (2002) 65–75.

[13]

Fan, J., Li, G., Ooi, B. C., Tan, K.-l., & Feng, J. (2015). Icrowd: An adaptive crowdsourcing framework. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (pp. 1015–1030). ACM.

[14]

Fang, M., Yin, J., & Zhu, X. (2013). Knowledge transfer for multi-labeler active learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 273–288). Springer.

[15]

Fang, M., Zhu, X., Li, B., Ding, W., & Wu, X. (2012). Self-taught active learning from crowds. In 2012 IEEE 12th international conference on data mining (pp. 858–863). IEEE.

[16]

X. He, D. Cai, Y. Shao, H. Bao, J. Han, Laplacian regularized gaussian mixture model for data clustering, IEEE Transactions on Knowledge and Data Engineering 23 (2010) 1406–1418.

[17]

Huang, S.-J., Chen, J.-L., Mu, X., & Zhou, Z.-H. (2017). Cost-effective active learning from diverse labelers. In IJCAI (pp. 1879–1885).

[18]

Imamura, H., Sato, I., & Sugiyama, M. (2018). Analysis of minimax error rate for crowdsourcing and its application to worker clustering model. In International Conference on Machine Learning (pp. 2147–2156). PMLR.

[19]

Jagabathula, S., Subramanian, L., & Venkataraman, A. (2014). Reputation-based worker filtering in crowdsourcing. Advances in Neural Information Processing Systems, 27.

[20]

L. Jiang, H. Zhang, F. Tao, C. Li, Learning from crowds with multiple noisy label distribution propagation, IEEE Transactions on Neural Networks and Learning Systems (2021),.

[21]

Jiang, Z., Zheng, Y., Tan, H., Tang, B., & Zhou, H. (2017). Variational deep embedding: An unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (pp. 1965– 1972).

[22]

Kajino, H., Tsuboi, Y., & Kashima, H. (2012). A convex formulation for learning from crowds. In Twenty-Sixth AAAI Conference on Artificial Intelligence.

[23]

Karger, D. R., Oh, S., & Shah, D. (2011). Budget-optimal crowdsourcing using low-rank matrix approximations. In 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton) (pp. 284–291). IEEE.

[24]

Kim, H.-C., & Ghahramani, Z. (2012). Bayesian classifier combination. In Artificial Intelligence and Statistics (pp. 619–627). PMLR.

[25]

A. Kurve, D.J. Miller, G. Kesidis, Multicategory crowdsourcing accounting for variable task difficulty, worker skill, and worker intention, IEEE Transactions on Knowledge and Data Engineering 27 (2014) 794–809.

[26]

Li, S. Y., Jiang, Y., & Zhou, Z. H. (2015). Multi-label active learning from crowds. Computer Science.

[27]

Liu, Q., Peng, J., & Ihler, A. T. (2012). Variational inference for crowdsourcing. Advances in neural information processing systems, 25.

[28]

Li’ang Yin, J. H., Zhang, W., & Yu, Y. (2017). Aggregating crowd wisdoms with label-aware autoencoders. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. AAAI Press (pp. 1325–1331).

[29]

C. Long, G. Hua, A. Kapoor, Active visual recognition with expertise estimation in crowdsourcing, in: Proceedings of the IEEE International Conference on Computer Vision, 2013, pp. 3000–3007.

[30]

MacQueen, J. et al. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability (pp. 281–297). Oakland, CA, USA volume 1.

[31]

Mallah, C., Cope, J., Orwell, J. et al. (2013). Plant leaf classification using probabilistic integration of shape, texture and margin features. Signal Processing, Pattern Recognition and Applications, 5.

[32]

Mo, K., Zhong, E., & Yang, Q. (2013). Cross-task crowdsourcing. In Proceedings of the 19th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 677–685).

[33]

Ok, J., Oh, S., Shin, J., & Yi, Y. (2016). Optimality of belief propagation for crowdsourced classification. In International Conference on Machine Learning (pp. 535–544). PMLR.

[34]

Raykar, V. C., & Yu, S. (2011). Ranking annotators for crowdsourced labeling tasks. Advances in neural information processing systems, 24.

[35]

V.C. Raykar, S. Yu, L.H. Zhao, A. Jerebko, C. Florin, G.H. Valadez, et al., Supervised learning from multiple experts: Whom to trust when everyone lies a bit, in: Proceedings of the 26th Annual international conference on machine learning, 2009, pp. 889–896.

[36]

V.C. Raykar, S. Yu, L.H. Zhao, G.H. Valadez, C. Florin, L. Bogoni, et al., Learning from crowds, Journal of Machine Learning Research 11 (2010).

[37]

Rodrigues, F., & Pereira, F. (2018). Deep learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 32.

[38]

F. Rodrigues, F. Pereira, B. Ribeiro, Learning from multiple annotators: Distinguishing good from random labelers, Pattern Recognition Letters 34 (2013) 1428–1436.

[39]

Rodrigues, F., Pereira, F., & Ribeiro, B. (2014). Gaussian process classification and active learning with multiple annotators. In International conference on machine learning (pp. 433–441). PMLR.

[40]

P. Ruiz, P. Morales-Alvarez, R. Molina, A.K. Katsaggelos, Learning from crowds with variational gaussian ́ processes, Pattern Recognition 88 (2019) 298–311.

[41]

Sheng, V. S. (2011). Simple multiple noisy label utilization strategies. In 2011 IEEE 11th International Conference on Data Mining (pp. 635–644). IEEE.

[42]

V.S. Sheng, F. Provost, P.G. Ipeirotis, Get another label? improving data quality and data mining using multiple, noisy labelers, in: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, 2008, pp. 614–622.

[43]

Sheng, V. S., & Zhang, J. (2019). Machine learning with crowdsourcing: A brief summary of the past research and future directions. In Proc. 33rd AAAI Conf. Artif. Intell. (pp. 9837–9843).

[44]

R. Snow, B. O’connor, D. Jurafsky, A.Y. Ng, Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks, in: In Proceedings of the 2008 conference on empirical methods in natural language processing, 2008, pp. 254–263.

[45]

R. Tanno, A. Saeedi, S. Sankaranarayanan, D.C. Alexander, N. Silberman, Learning from noisy labels by regularized estimation of annotator confusion, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 11244–11253.

[46]

D. Tao, J. Cheng, Z. Yu, K. Yue, L. Wang, Domain-weighted majority voting for crowdsourcing, IEEE Transactions on Neural Networks and Learning Systems 30 (2018) 163–174.

[47]

Thierry, C., Dubois, J.-C., Le Gall, Y., & Martin, A. (2019). Modeling uncertainty and inaccuracy on data from crowdsourcing platforms: Monitor. In 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI) (pp. 776–783). IEEE.

[48]

S. Wallace, T. Cai, B. Le, L.A. Leiva, Debiased label aggregation for subjective crowdsourcing tasks, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), 2022, pp. 1–8.

[49]

Welinder, P., Branson, S., Belongie, S., & Perona, P. (2010). The multidimensional wisdom of crowds. volume 23.

[50]

Welinder, P., & Perona, P. (2010). Online crowdsourcing: rating annotators and obtaining cost-effective labels. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops (pp. 25–32). IEEE.

[51]

J. Whitehill, T.-F. Wu, J. Bergsma, J. Movellan, P. Ruvolo, Whose vote should count more: Optimal integration of labels from labelers of unknown expertise, Advances in Neural Information Processing Systems 22 (2009) 2035–2043.

[52]

Wu, M., Li, Q., Wang, S., & Hou, J. (2019). A subjectivity-aware algorithm for label aggregation in crowdsourcing. In 2019 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC) (pp. 373–378). IEEE.

[53]

Wu, M., Li, Q., Zhang, J., Cui, S., Li, D., & Qi, Y. (2017). A robust inference algorithm for crowd sourced categorization. In 2017 12th International Conference on Intelligent Systems and Knowledge Engineering (ISKE) (pp. 1–6). IEEE.

[54]

Xie, J., Girshick, R., & Farhadi, A. (2016). Unsupervised deep embedding for clustering analysis. In International conference on machine learning (pp. 478–487). PMLR.

[55]

Yan, Y., Rosales, R., Fung, G., & Dy, J. G. (2011). Active learning from crowds. In ICML.

[56]

Y. Yan, R. Rosales, G. Fung, R. Subramanian, J. Dy, Learning from multiple annotators with varying expertise, Machine learning 95 (2014) 291–327.

[57]

M.-S. Yang, C.-Y. Lai, C.-Y. Lin, A robust em clustering algorithm for gaussian mixture models, Pattern Recognition 45 (2012) 3950–3961.

[58]

J. Zhang, V.S. Sheng, Q. Li, J. Wu, X. Wu, Consensus algorithms for biased labeling in crowdsourcing, Information Sciences 382 (2017) 254–273.

[59]

J. Zhang, V.S. Sheng, B. Nicholson, X. Wu, Ceka: A tool for mining the wisdom of crowds, Journal of Machine Learning Research 16 (2015) 2853–2858.

[60]

J. Zhang, V.S. Sheng, J. Wu, X. Wu, Multi-class ground truth inference in crowdsourcing with clustering, IEEE Transactions on Knowledge and Data Engineering 28 (2015) 1080–1085.

[61]

J. Zhang, X. Wu, V.S. Sheng, Imbalanced multiple noisy labeling, IEEE Transactions on Knowledge and Data Engineering 27 (2015) 489–503.

[62]

J. Zhang, X. Wu, V.S. Sheng, Learning from crowdsourced labeled data: A survey, Artificial Intelligence Review 46 (2016) 543–576.

[63]

J. Zhang, X. Wu, V.S. Shengs, Active learning with imbalanced multiple noisy labeling, IEEE Transactions on Cybernetics 45 (2015) 1095–1107.

[64]

Y. Zhang, X. Chen, D. Zhou, M.I. Jordan, Spectral methods meet em: A provably optimal algorithm for crowdsourcing, Advances in Neural Information Processing Systems 27 (2014) 1260–1268.

[65]

Z.-H. Zhou, A brief introduction to weakly supervised learning, National Science Review 5 (2018) 44–53.

Cited By

Gong FRaghunathan DGupta AApostolaki MSekar VYu MSeneviratne AVeitch D(2024)Zoom2Net: Constrained Network Telemetry ImputationProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672225(764-777)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672225

Index Terms

Learning from biased crowdsourced labeling with deep clustering

Index terms have been assigned to the content through auto-classification.

Recommendations

Label Aggregation with Clustering for Biased Crowdsourced Labeling
ICMLC '22: Proceedings of the 2022 14th International Conference on Machine Learning and Computing

With the rapid development of crowdsourcing learning, amount of label aggregation methods are proposed to infer the true labels of instances from multiple noisy labels provided by inexpert crowd workers. Most of the label aggregation methods take the ...
Label Aggregation for Crowdsourcing with Bi-Layer Clustering
SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

This paper proposes a novel general label aggregation method for both binary and multi-class labeling in crowdsourcing, namely Bi-Layer Clustering (BLC), which clusters two layers of features - the conceptual-level and the physical-level features - to ...
Learning with partly labeled data

Learning with partly labeled data aims at combining labeled and unlabeled data in order to boost the accuracy of a classifier. This paper outlines the two main classes of learning methods to deal with partly labeled data: pre-labeling-based learning and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Expert Systems with Applications: An International Journal

Expert Systems with Applications: An International Journal Volume 211, Issue C

Jan 2023

1635 pages

ISSN:0957-4174

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 January 2023

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gong FRaghunathan DGupta AApostolaki MSekar VYu MSeneviratne AVeitch D(2024)Zoom2Net: Constrained Network Telemetry ImputationProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672225(764-777)Online publication date: 4-Aug-2024
https://dl.acm.org/doi/10.1145/3651890.3672225

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents