[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3447548.3470826acmconferencesArticle/Chapter ViewAbstractPublication PageskddConference Proceedingsconference-collections
abstract
Public Access

KDD 2021 Tutorial on Systemic Challenges and Solutions on Bias and Unfairness in Peer Review

Published: 14 August 2021 Publication History

Abstract

No abstract available.

References

[1]
S. Price and P. Flach. 2017. Computational support for academic peer review: a perspective from artificial intelligence. Communications of the ACM.
[2]
A. Mulligan, L. Hall, and E. Raphael. 2013. Peer review in a changing world: an international study measuring the attitudes of researchers. Journal of the Association for Information Science and Technology.
[3]
D. Nicholas, A. Watkinson, H. Jamali, E. Herman, C. Tenopir, R. Volentine, S. Allard, and K. Levine. 2015. Peer review: still king in the digital age. Learned Publishing.
[4]
M. Ware. 2008. Peer review: benefits, perceptions and alternatives. Publishing Research Consortium.
[5]
R. Smith. 2006. Peer review: a flawed process at the heart of science and journals. Journal of the royal society of medicine.
[6]
R. K. Merton. 1968. The Matthew effect in science. Science.
[7]
C. Triggle and D. Triggle. 2007. What is the future of peer review? Vascular health and risk management.
[8]
W. Thorngate and W. Chowdhury. 2014. By the numbers: track record, flawed reviews, journal space, and the fate of talented authors. In Advances in Social Simulation. Springer.
[9]
F. Squazzoni and C. Gandelli. 2012. Saint matthew strikes again: an agentbased model of peer review and the scientific community structure. Journal of Informetrics.
[10]
A. Tomkins, M. Zhang, and W. D. Heavlin. 2017. Reviewer bias in single-versus double-blind peer review. Proceedings of the National Academy of Sciences.
[11]
I. Stelmakh, N. Shah, and A. Singh. 2019. On testing for biases in peer review. In NeurIPS.
[12]
E. Manzoor and N. B. Shah. 2021. Uncovering latent biases in text: method and application to peer review. In AAAI.
[13]
K. Okike, K. T. Hug, M. S. Kocher, and S. S. Leopold. 2016. Single-blind vs Double-blind Peer Review in the Setting of Author Prestige. JAMA.
[14]
A. E. Budden, T. Tregenza, L. W. Aarssen, J. Koricheva, R. Leimu, and C. J. Lortie. 2008. Double-blind review favours increased representation of female authors. Trends in Ecology and Evolution.
[15]
T. Webb, B. O'Hara, and R. Freckleton. 2008. Does double-blind review benefit female authors? Trends in Ecology and Evolution.
[16]
S. Hill and F. J. Provost. 2003. The myth of the double-blind review? author identification using only citations. SIGKDD Explorations.
[17]
S. Siegelman. 1991. Assassins and zealots: variations in peer review. Radiology.
[18]
L. Brenner, D. Griffin, and D. J. Koehler. 2005. Modeling patterns of probability calibration with random support theory: diagnosing case-based judgment. Organizational Behavior and Human Decision Processes.
[19]
S. R. Paul. 1981. Bayesian methods for calibration of examiners. British Journal of Mathematical and Statistical Psychology.
[20]
P. A. Flach, S. Spiegler, B. Golénia, S. Price, J. Guiver, R. Herbrich, T. Graepel, and M. J. Zaki. 2010. Novel tools to streamline the conference review process: experiences from SIGKDD'09. SIGKDD Explor. Newsl.
[21]
M. Roos, J. Rothe, and B. Scheuermann. 2011. How to calibrate the scores of biased reviewers by quadratic programming. In AAAI.
[22]
H. Ge, M. Welling, and Z. Ghahramani. 2013. A Bayesian model for calibrating conference review scores. Manuscript. http : / /mlg . eng . cam . ac . uk / hong / unpublished/nips-review-model.pdf Last accessed: April 4, 2021. (2013).
[23]
Y. Baba and H. Kashima. 2013. Statistical quality estimation for general crowdsourcing tasks. In KDD.
[24]
R. S. MacKay, R. Kenna, R. J. Low, and S. Parker. 2017. Calibration with confidence: a principled method for panel assessment. Royal Society Open Science.
[25]
M. Rokeach. 1968. The role of values in public opinion research. Public Opinion Quarterly.
[26]
Y. Freund, R. D. Iyer, R. E. Schapire, and Y. Singer. 2003. An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research.
[27]
A.-W. Harzing et al. 2009. Rating versus ranking: what is the best way to reduce response and language bias in cross-national research? International Business Review.
[28]
I. Mitliagkas, A. Gopalan, C. Caramanis, and S. Vishwanath. 2011. User rankings from comparisons: learning permutations in high dimensions. In Allerton Conference.
[29]
A. Ammar and D. Shah. 2012. Efficient rank aggregation using partial data. In SIGMETRICS.
[30]
S. Negahban, S. Oh, and D. Shah. 2012. Iterative ranking from pair-wise comparisons. In Advances in Neural Information Processing Systems.
[31]
J. Wang and N. B. Shah. 2019. Your 2 is my 1, your 3 is my 9: handling arbitrary miscalibrations in ratings. In AAMAS.
[32]
N. B. Shah, B. Tabibian, K. Muandet, I. Guyon, and U. Von Luxburg. 2018. Design and analysis of the NIPS 2016 review process. JMLR.
[33]
W. Ding, N. B. Shah, and W. Wang. 2020. On the privacy-utility tradeoff in peer-review data analysis. In AAAI Privacy-Preserving Artificial Intelligence (PPAI-21) workshop.
[34]
S. Balietti, R. Goldstone, and D. Helbing. 2016. Peer review and competition in the art exhibition game. Proceedings of the National Academy of Sciences.
[35]
N. Alon, F. Fischer, A. Procaccia, and M. Tennenholtz. 2011. Sum of us: strategyproof selection from the selectors. In Proceedings of the 13th Conference on Theoretical Aspects of Rationality and Knowledge. ACM.
[36]
G. De Clippel, H. Moulin, and N. Tideman. 2008. Impartial division of a dollar. Journal of Economic Theory.
[37]
R. Holzman and H. Moulin. 2013. Impartial nominations for a prize. Econometrica.
[38]
F. Fischer and M. Klimm. 2015. Optimal impartial selection. SIAM Journal on Computing.
[39]
D. Kurokawa, O. Lev, J. Morgenstern, and A. D. Procaccia. 2015. Impartial peer review. In IJCAI.
[40]
H. Aziz, O. Lev, N. Mattei, J. S. Rosenschein, and T. Walsh. 2016. Strategyproof peer selection: mechanisms, analyses, and experiments. In AAAI.
[41]
A. Kahng, Y. Kotturi, C. Kulkarni, D. Kurokawa, and A. Procaccia. 2018. Ranking wily people who rank each other. In AAAI.
[42]
Y. Xu, H. Zhao, X. Shi, and N. Shah. 2019. On strategyproof conference review. In IJCAI.
[43]
I. Stelmakh, N. Shah, and A. Singh. 2021. Catch me if i can: detecting strategic behaviour in peer assessment. In AAAI.
[44]
T. N. Vijaykumar. 2020. Potential organized fraud in on-going asplos reviews. https://medium.com/@tnvijayk/potential- organized- fraud-in- on- goingasplos-reviews-874ce14a3ebe. (2020).
[45]
T. N. Vijaykumar. 2020 (accessed November 17, 2020). Potential organized fraud in ACM/IEEE computer architecture conferences. https://medium.com/ @tnvijayk/potential-organized- fraud-in-acm-ieee-computer-architectureconferences-ccd61169370d. (2020 (accessed November 17, 2020)).
[46]
S. Jecmen, H. Zhang, R. Liu, N. B. Shah, V. Conitzer, and F. Fang. 2020. Mitigating manipulation in peer review via randomized reviewer assignments. In NeurIPS.
[47]
R. Wu, C. Guo, F. Wu, R. Kidambi, L. van der Maaten, and K. Q. Weinberger. 2021. Making paper reviewing robust to bid manipulation attacks. arXiv preprint arXiv:2102.06020.
[48]
L. Charlin and R. S. Zemel. 2013. The Toronto Paper Matching System: an automated paper-reviewer assignment system. In ICML Workshop on Peer Reviewing and Publishing Models.
[49]
I. Stelmakh, N. Shah, and A. Singh. 2019. PeerReview4All: fair and accurate reviewer assignment in peer review. In ALT.
[50]
A. Kobren, B. Saha, and A. McCallum. 2019. Paper matching with local fairness constraints. In KDD.
[51]
N. Garg, T. Kavitha, A. Kumar, K. Mehlhorn, and J. Mestre. 2010. Assigning papers to referees. Algorithmica.
[52]
J. Goldsmith and R. Sloan. 2007. The AI conference paper assignment problem.
[53]
W. Tang, J. Tang, and C. Tan. 2010. Expertise matching via constraint-based optimization. In IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.
[54]
C. J. Taylor. 2008. On the optimal assignment of conference papers to reviewers.
[55]
C. Long, R. Wong, Y. Peng, and L. Ye. 2013. On good and fair paper-reviewer assignment. In ICDM.
[56]
T Fiez, N Shah, and L Ratliff. 2020. A SUPER* algorithm to optimize paper bidding in peer review. In Conference on Uncertainty in Artificial Intelligence.
[57]
C. J. Lee. 2015. Commensuration bias in peer review. Philosophy of Science.
[58]
S. Kerr, J. Tolliver, and D. Petree. 1977. Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal.
[59]
R. Noothigattu, N. Shah, and A. Procaccia. 2021. Loss functions, axioms, and peer review. Journal of Artificial Intelligence Research.
[60]
I. Stelmakh, N. Shah, A. Singh, and H. Daumé III. 2021. Prior and prejudice: the novice reviewers' bias against resubmissions in conference peer review. In CSCW.
[61]
I. Stelmakh, C. Rastogi, N. B. Shah, A. Singh, and H. Daumé III. 2020. A large scale randomized controlled trial on herding in peer-review discussions. arXiv preprint arXiv:2011.15083.
[62]
I. Stelmakh, N. Shah, A. Singh, and H. Daumé III. 2021. A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences. In AAAI.
[63]
J. Wang and N. Shah. 2019. Gender distributions of paper awards. Research on Research blog. https : / / researchonresearch . blog / 2019 / 06 / 18 / gender - distributions-of-paper-awards/. (2019).
[64]
J. Wang and N. Shah. 2018. There's lots in a name (whereas there shouldn't be). Research on Research blog. https://researchonresearch.blog/2018/11/28/thereslots-in-a-name/. (2018).
[65]
N. Lawrence and C. Cortes. 2014. The NIPS Experiment. http://inverseprobability. com/2014/12/16/the-nips-experiment. [Online; accessed 11-June-2018]. (2014).
[66]
J. Wang, I. Stelmakh, Y. Wei, and N. Shah. 2021. Debiasing evaluations that are biased by evaluations. In AAAI.
[67]
N. B. Shah. 2021. WSDM 2021 tutorial on systematic challenges and computational solutions on bias and unfairness in peer review. In WSDM.

Cited By

View all

Index Terms

  1. KDD 2021 Tutorial on Systemic Challenges and Solutions on Bias and Unfairness in Peer Review

          Recommendations

          Comments

          Please enable JavaScript to view thecomments powered by Disqus.

          Information & Contributors

          Information

          Published In

          cover image ACM Conferences
          KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining
          August 2021
          4259 pages
          ISBN:9781450383325
          DOI:10.1145/3447548
          Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

          Sponsors

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          Published: 14 August 2021

          Check for updates

          Author Tags

          1. bias
          2. peer review

          Qualifiers

          • Abstract

          Funding Sources

          • NSF

          Conference

          KDD '21
          Sponsor:

          Acceptance Rates

          Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

          Upcoming Conference

          KDD '25

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • Downloads (Last 12 months)85
          • Downloads (Last 6 weeks)19
          Reflects downloads up to 03 Jan 2025

          Other Metrics

          Citations

          Cited By

          View all

          View Options

          View options

          PDF

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          Login options

          Media

          Figures

          Other

          Tables

          Share

          Share

          Share this Publication link

          Share on social media