More Web Proxy on the site http://driver.im/

research-article

Relationships of Cohen's Kappa, Sensitivity, and Specificity for Unbiased Annotations

Authors:

Bin XiaAuthors Info & Claims

ICBIP '19: Proceedings of the 4th International Conference on Biomedical Signal and Image Processing

Pages 98 - 101

https://doi.org/10.1145/3354031.3354040

Published: 13 August 2019 Publication History

Abstract

For the binary classification tasks in supervised learning, the labels of data have to be available for classifier development. Cohen's kappa is usually employed as a quality measure for data annotation, which is inconsistent with its true functionality of assessing the inter-annotator consistency. However, the derived relationship functions of Cohen's kappa, sensitivity, and specificity in the literature are complicated, thus are unable to be employed to interpret classification performance from kappa values. In this study, based on an annotation generation model, we develop simple relationships of kappa, sensitivity, and specificity when there is no bias in the annotations. A relationship between kappa and Youden's J statistic, a performance metric for binary classification, is further obtained. The derived relationships are evaluated on a synthetic dataset using linear regression analysis. The results demonstrate the accuracy of the derived relationships. It suggests the potential of estimating classification performance from kappa values when bias is absent in the annotations.

References

[1]

Aickin, M., 1990. Maximum likelihood estimation of agreement in the constant predictive probability model, and its relation to Cohen's kappa. Biometrics, (June 1990), 293--302.

[2]

Beigman Klebanov, B. and Beigman, E., 2009. From annotator agreement to noise models. Computational Linguistics, 35, 4 (Dec. 2009), pp. 495--503.

Digital Library

[3]

Byrt, T., Bishop, J. and Carlin, J.B., 1993. Bias, prevalence and kappa. Journal of clinical epidemiology, 46, 5 (May 1993), 423--429.

[4]

Carletta, J., 1996. Assessing agreement on classification tasks: the kappa statistic. Computational linguistics, 22, 2 (June 1996), 249--254.

[5]

Cohen, J., 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20, 1 (April 1960), 37--46.

[6]

Eugenio, B.D. and Glass, M., 2004. The kappa statistic: A second look. Computational linguistics, 30, 1 (March 2004), 95--101.

[7]

Feuerman, M. and Miller, A.R., 2005. The kappa statistic as a function of sensitivity and specificity. International Journal of Mathematical Education in Science and Technology, 36,5 (July 2005), 517--527.

[8]

Feuerman, M. and Miller, A.R., 2007. Critical points for certain statistical measures of agreement. International Journal of Mathematical Education in Science and Technology, 38, 6, (Sep. 2007), 739--748.

[9]

Feuerman, M. and Miller, A.R., 2008. Relationships between statistical measures of agreement: sensitivity, specificity and kappa. Journal of evaluation in clinical practice, 14, 5(Oct. 2008), pp. 930--933.

[10]

Frénay, B. and Verleysen, M., 2014. Classification in the presence of label noise: a survey. IEEE Transactions on Neural Networks and Learning Systems, 25, 5 (May 2014), 845--869.

[11]

Ghosh, A., Manwani, N. and Sastry, P.S., 2015. Making risk minimization tolerant to label noise. Neurocomputing, 160 (July 2015), 93--107.

[12]

Natarajan, N., Dhillon, I.S., Ravikumar, P.K. and Tewari, A., 2013. Learning with noisy labels. In Advances in Neural Information Processing Systems (Lake Tahoe, USA, Dec. 05-10, 2013), Curran Associates, Inc., New York, NY, 1196--1204.

[13]

Passonneau, R.J. and Carpenter, B., 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2 (Dec. 2014), 311--326.

[14]

Schisterman, E.F., Perkins, N.J., Liu, A. and Bondell, H., 2005. Optimal cut-point and its corresponding Youden Index to discriminate individuals using pooled blood samples. Epidemiology, (Jan. 2005), 73--81.

[15]

Thompson, W.D. and Walter, S.D., 1988. A reappraisal of the kappa coefficient. Journal of clinical epidemiology, 41, 10, (Jan. 1988), 949--958.

Cited By

Harrag FMahdadi CZiad A(2023)Detecting and Measuring Social Bias of Arabic Generative Models in the Context of Search and RecommendationAdvances in Bias and Fairness in Information Retrieval10.1007/978-3-031-37249-0_13(155-168)Online publication date: 15-Jul-2023
https://doi.org/10.1007/978-3-031-37249-0_13
Eni MMordoh VZigel Y(2022)Cough detection using a non-contact microphone: A nocturnal cough studyPLOS ONE10.1371/journal.pone.026224017:1(e0262240)Online publication date: 19-Jan-2022
https://doi.org/10.1371/journal.pone.0262240
Ferdous ZAkhter RTahsin ANuha Mustafina STabassum N(2022)Sentiment Analysis on COVID-19 Vaccine Twitter Data using Neural Network ModelsProceedings of the 2nd International Conference on Computing Advancements10.1145/3542954.3543064(435-441)Online publication date: 10-Mar-2022
https://dl.acm.org/doi/10.1145/3542954.3543064
Show More Cited By

Index Terms

Relationships of Cohen's Kappa, Sensitivity, and Specificity for Unbiased Annotations
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
        Supervised learning by classification

Recommendations

Cohen's weighted kappa with additive weights

Cohen's weighted kappa is a popular descriptive statistic for summarizing interrater agreement on an ordinal scale. An agreement table with $$n\in \mathbb N _{\ge 3}$$ ordered categories can be collapsed into $$n-1$$ distinct $$2\times 2$$ tables by combining adjacent categories. Weighted kappa with ...
About the relationship between ROC curves and Cohen's kappa

Receiver operating characteristic (ROC) curves are very powerful tools for measuring classifiers' accuracy in binary-class problems. However, their usefulness in real-world multi-class problems has not been demonstrated yet. In these frequently ...
Generalized Cohen’s Kappa: A Novel Inter-rater Reliability Metric for Non-mutually Exclusive Categories
Human Interface and the Management of Information
Abstract
Qualitative coding of large datasets has been a valuable tool for qualitative researchers. In terms of inter-rater reliability, existing metrics have not evolved to fit current approaches, presenting a variety of restrictions. In this paper, we ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICBIP '19: Proceedings of the 4th International Conference on Biomedical Signal and Image Processing

August 2019

149 pages

ISBN:9781450372244

DOI:10.1145/3354031

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

Graduate School of Library, Information, and Media Studies, University of Tsukuba, Japan: Graduate School of Library, Information, and Media Studies, University of Tsukuba, Japan
Sichuan University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 August 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICBIP '19

ICBIP '19: 2019 4th International Conference on Biomedical Signal and Image Processing

August 13 - 15, 2019

Chengdu, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
251
Total Downloads

Downloads (Last 12 months)34
Downloads (Last 6 weeks)4

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Harrag FMahdadi CZiad A(2023)Detecting and Measuring Social Bias of Arabic Generative Models in the Context of Search and RecommendationAdvances in Bias and Fairness in Information Retrieval10.1007/978-3-031-37249-0_13(155-168)Online publication date: 15-Jul-2023
https://doi.org/10.1007/978-3-031-37249-0_13
Eni MMordoh VZigel Y(2022)Cough detection using a non-contact microphone: A nocturnal cough studyPLOS ONE10.1371/journal.pone.026224017:1(e0262240)Online publication date: 19-Jan-2022
https://doi.org/10.1371/journal.pone.0262240
Ferdous ZAkhter RTahsin ANuha Mustafina STabassum N(2022)Sentiment Analysis on COVID-19 Vaccine Twitter Data using Neural Network ModelsProceedings of the 2nd International Conference on Computing Advancements10.1145/3542954.3543064(435-441)Online publication date: 10-Mar-2022
https://dl.acm.org/doi/10.1145/3542954.3543064
AbdelMaksoud EBarakat SElmogy M(2022)A computer-aided diagnosis system for detecting various diabetic retinopathy grades based on a hybrid deep learning techniqueMedical & Biological Engineering & Computing10.1007/s11517-022-02564-660:7(2015-2038)Online publication date: 11-May-2022
https://doi.org/10.1007/s11517-022-02564-6
Yu SLi XFeng YZhang XChen S(2021)An instance-oriented performance measure for classificationInformation Sciences10.1016/j.ins.2021.08.094580(598-619)Online publication date: Nov-2021
https://doi.org/10.1016/j.ins.2021.08.094

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents