More Web Proxy on the site http://driver.im/

research-article

Bi-convex Optimization to Learn Classifiers from Multiple Biomedical Annotations

Authors:

Jinbo BiAuthors Info & Claims

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB), Volume 14, Issue 3

Pages 564 - 575

https://doi.org/10.1109/TCBB.2016.2576457

Published: 01 May 2017 Publication History

Abstract

The problem of constructing classifiers from multiple annotators who provide inconsistent training labels is important and occurs in many application domains. Many existing methods focus on the understanding and learning of the crowd behaviors. Several probabilistic algorithms consider the construction of classifiers for specific tasks using consensus of multiple labelers annotations. These methods impose a prior on the consensus and develop an expectation-maximization algorithm based on logistic regression loss. We extend the discussion to the hinge loss commonly used by support vector machines. Our formulations form bi-convex programs that construct classifiers and estimate the reliability of each labeler simultaneously. Each labeler is associated with a reliability parameter, which can be a constant, or class-dependent, or varies for different examples. The hinge loss is modified by replacing the true labels by the weighted combination of labelers’ labels with reliabilities as weights. Statistical justification is discussed to motivate the use of linear combination of labels. In parallel to the expectation-maximization algorithm for logistic-based methods, efficient alternating algorithms are developed to solve the proposed bi-convex programs. Experimental results on benchmark datasets and three real-world biomedical problems demonstrate that the proposed methods either outperform or are competitive to the state of the art.

References

[1]

S.L. Hui and X.H. Zhou,"Evaluation of diagnostic tests without gold standards," Stat. Methods Med. Res., vol. 7, no. 4, pp. 354-70, 1998.

[2]

M. Yetisgen-Yildiz, I. Solti, F. Xia, and S. R. Halgrim, "Preliminary experience with amazon's mechanical turk for annotating medical named entities," in Proc. NAACL HLT Workshop Creating Speech Lang. Data Amazon's Mech. Turk, 2010, pp. 180-183.

[3]

J. D. Burger, et al., "Validating candidate gene-mutation relations in MEDLINE abstracts via crowdsourcing," in Proc. 8th Int. Conf. Data Integr. Life Sci., 2012, pp. 83-91.

[4]

B. M. Good and A. I. Su, "Crowdsourcing for bioinformatics," Bioinf., vol. 29, no. 16, pp. 1925-1933, 2013.

[5]

T. B. Nguyen, et al., "Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography," Radiology, vol. 262, pp. 824-833, 2012.

[6]

M. Qazi, G. Fung, S. Krishnan, J. Bi, B. Rao, and A. Katz, "Automated heart abnormality detection using sparse linear classifiers," IEEE Eng. Med. Biol. Mag., vol. 26, no. 2, pp. 56-63, Mar./Apr. 2007.

[7]

R. Jin and Z. Ghahramani, "Learning with multiple labels," in Advances in Neural Information Processing Systems, S. T. S. Becker and K. Obermayer, Eds. Cambridge, MA, USA: MIT Press, 2003, pp. 897-904.

[8]

Y. Yan, et al., "Modeling annotator expertise: Learning when everybody knows a bit of something," in Proc. 13th Int. Conf. Artif. Intell. Statist., 2010, pp. 932-939.

[9]

V. C. Raykar, et al., "Supervised learning from multiple experts: Who to trust when everyone lies a bit," in Proc. 26th Int. Conf. Mach. Learn., pp. 96-103, 2009.

[10]

V. C. Raykar, et al., "Learning from crowds," J. Mach. Learn. Res., vol. 11, pp. 1297-1322, 2010.

[11]

H. Kajino and H. Kashima, "A convex formulation for learning from crowds," in Proc. 26th AAAI Conf. Artif. Intell., 2012, pp. 73-79.

[12]

H. Kajino, Y. Tsuboi, and H. Kashima, "Clustering crowds," in Proc. AAAI Conf. Artif. Intell., 2013, pp. 1120-1127.

[13]

N. Pochet and J. Suykens, "Support vector machines versus logistic regression: Improving prospective performance in clinical decision-making," Ultrasound Obstetrics Gynecology, vol. 27, no. 6, pp. 607-608, 2006.

[14]

D. A. Salazar, J. I. V_elez, and J. C. Salazar, "Comparison between SVM and logistic regression: Which one is better to discriminate?" Revista Colombiana de Estadística, vol. 35, no. 2, pp. 223-237, 2012.

[15]

T. Verplancke, et al., "Support vector machine versus logistic regression modeling for prediction of hospital mortality in critically ill patients with haematological malignancies," BMC Med. Informat. Decision Making, vol. 8, no. 1, 2008, Art. no. 56.

[16]

A. P. Dawid and A. M. Skeene, "Maximum likelihood estimation of observed error-rates using the EM algorithm," Appl. Statist., vol. 28, no. 1, pp. 20-28, 1979.

[17]

P. S. Albert and L. E. Dodd, "A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard," Biometrics, vol. 60, no. 2, pp. 427-435, 2004.

[18]

R. Snow, B. O'Connor, D. Jurafsky, and A. Ng, "Cheap and fastbut is it good?: Evaluating non-expert annotations for natural language tasks," in Proc. Conf. Empirical Methods Nat. Lang. Process., 2008, pp. 254-263.

[19]

D. Zhou, J. Platt, S. Basu, and Y. Mao, "Learning from the wisdom of crowds by minimax entropy," in Proc. Adv. Neural Inf. Process. Syst., vol. 25, 2012, pp. 2204-2212.

[20]

D. Zhou, Q. Liu, J. C. Platt, and C. Meek, "Aggregating ordinal labels from crowds by minimax conditional entropy," in Proc. 31st Int. Conf. Mach. Learn., 2014, pp. 262-270.

[21]

V. C. Raykar and S. Yu, "Ranking annotators for crowd-sourced labeling tasks," in Advances in Neural Information Processing Systems 20, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, Eds. Cambridge, MA, USA: MIT Press, 2011, pp. 1809-1817.

[22]

C. Liu and Y.-M. Wang, "Truelabel+confusion: A spectrum of probabilistic models in analyzing multiple ratings," in Proc. Int. Conf. Mach. Learn., 2012, pp. 225-232.

[23]

S. B. P. Welinder, S. Branson and P. Perona, "The multidimensional wisdom of crowds," in Proc. Neural Inf. Process. Syst. Conf., 2010, pp. 2424-2432.

[24]

Y. Tian and J. Zhu, "Learning from crowds in the presence of schools of thought," in Proc. 18th ACM SIGKDD Int. Conf. Knowl. Discovery Data Min., 2012, pp. 226-234.

[25]

T. W. J. B. J. Whitehill, P. Ruvolo and J. Movellan, "Whose vote should count more: Optimal integration of labels from labelers of unknown expertise," in Proc. Neural Inf. Process. Syst. Conf., 2009, pp. 2035-2043.

[26]

D. R. Karger, S. Oh, and D. Shah, "Iterative learning for reliable crowdsourcing systems," in Proc. Adv. Neural Inf. Process. Syst., 2011, pp. 1953-1961.

[27]

C.-J. Ho, S. Jabbari, and J. W. Vaughan, "Adaptive task assignment for crowdsourced classification," in Proc. Int. Conf. Mach. Learn., 2013, pp. 534-542.

[28]

A. Singla, I. Bogunovic, G. Bartók, A. Karbasi, and A. Krause, "Near-optimally teaching the crowd to classify," in Proc. 31st Int. Conf. Mach. Learn., 2014, pp. 154-162.

[29]

P. Smyth, U. Fayyad, M. Burl, P. Perona, and P. Baldi, "Inferring ground truth from subjective labeling of venus images," in Proc. Adv. Neural Inf. Process. Syst. 7, 1995, pp. 1085-1092.

[30]

V. S. Sheng, F. Provost, and P. G. Ipeirotis, "Get another label? improving data quality and data mining using multiple, noisy labelers," in Proc. ACM Conf. Knowl. Discovery Data Min., 2008, pp. 614-622.

[31]

O. Dekel and O. Shamir, "Good learners for evil teachers," in Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp. 233-240.

[32]

J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. Wortman, "Learning bounds for domain adaptation," in Advances in Neural Information Processing Systems 20, J. Platt, D. Koller, Y. Singer, and S. Roweis, Eds. Cambridge, MA: MIT Press, 2008, pp. 129-136.

[33]

K. Crammer, M. Kearns, and J. Wortman, "Learning from multiple sources," J. Mach. Learn. Res., vol. 9, no. 2, pp. 1757-1774, 2009.

[34]

Y. Yan, R. Rosales, G. Fung, and J. Dy, "Modeling multiple annotator expertise in the semi-supervised learning scenario," Proc. 26th Conf. Uncertainty Artif. Intell., pp. 241-248, 2010.

[35]

M. Fang, J. Yin, and D. Tao, "Active learning for crowdsourcing using knowledge transfer," in Proc. 28th AAAI Conf. Artif. Intell., 2014, pp. 1809-1815.

[36]

J. Bi and X. Wang, "Learning classifiers from dual annotation ambiguity via a minmax framework," Neurocomputing, vol. 151, Part 2, pp. 891-904, 2015.

[37]

J. C. Bezdek and R. J. Hathaway, "Some notes on alternating optimization," in Proc. AFSS Int. Conf. Fuzzy Syst. Adv. Soft Comput., vol. 2275, 2002, pp. 288-300.

[38]

J. Gorski, F. Pfeuffer, and K. Klamroth, " Biconvex sets and optimization with biconvex functions: A survey and extensions," Math. Methods Oper. Res., vol. 66, no. 3, pp. 373-407, 2007.

[39]

B. Mozafari, P. Sarkar, M. J. Franklin, M. I. Jordan, and S. Madden, "Active learning for crowd-sourced databases," Comput. Res. Repository, vol. abs/1209.3686, 2012. [Online]. Available: http://arvix.org/abs/1209.3686

[40]

T. Ojala, M. Pietikainen, and D. Harwood, "Performance evaluation of texture measures with classification based on kullback discrimination of distributions," in Proc. 12th Int. Conf. Pattern Recognit., vol. 1, 1994, pp. 582-585.

Cited By

Aung AWhitehill J(2018)Harnessing Label Uncertainty to Improve Modeling: An Application to Student Engagement Recognition2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)10.1109/FG.2018.00033(166-170)Online publication date: 15-May-2018
https://dl.acm.org/doi/10.1109/FG.2018.00033

Bi-convex Optimization to Learn Classifiers from Multiple Biomedical Annotations
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches

Recommendations

Convex formulation of multiple instance learning from positive and unlabeled bags
Abstract
Multiple instance learning (MIL) is a variation of traditional supervised learning problems where data (referred to as bags) are composed of sub-elements (referred to as instances) and only bag labels are available. MIL has a variety ...
Semi-Supervised Learning on Single-View Datasets by Integration of Multiple Co-trained Classifiers
ICMLA '12: Proceedings of the 2012 11th International Conference on Machine Learning and Applications - Volume 01

We propose a novel semi-supervised learning algorithm, called IMCC, designed for co-training classifiers on single-view datasets. Our method runs the co-training algorithm for a predefined number of times, each time using a different random split of ...
Learning sparse classifiers with difference of convex functions algorithms
the 8th International Conference on Optimization: Techniques and Applications

Sparsity of a classifier is a desirable condition for high-dimensional data and large sample sizes. This paper investigates the two complementary notions of sparsity for binary classification: sparsity in the number of features and sparsity in the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE/ACM Transactions on Computational Biology and Bioinformatics

IEEE/ACM Transactions on Computational Biology and Bioinformatics Volume 14, Issue 3

May 2017

248 pages

ISSN:1545-5963

Issue’s Table of Contents

Copyright © 2017.

Publisher

IEEE Computer Society Press

Washington, DC, United States

Publication History

Published: 01 May 2017

Published in TCBB Volume 14, Issue 3

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
31
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Aung AWhitehill J(2018)Harnessing Label Uncertainty to Improve Modeling: An Application to Student Engagement Recognition2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)10.1109/FG.2018.00033(166-170)Online publication date: 15-May-2018
https://dl.acm.org/doi/10.1109/FG.2018.00033

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents