More Web Proxy on the site http://driver.im/

research-article

Is three better than one?: simulating the effect of reviewer selection and behavior on the quality and efficiency of peer review

Authors:

Federico Bianchi,

Flaminio SquazzoniAuthors Info & Claims

WSC '15: Proceedings of the 2015 Winter Simulation Conference

Pages 4081 - 4089

Published: 06 December 2015 Publication History

Abstract

This paper looks at the effect of multiple reviewers and their behavior on the quality and efficiency of peer review. By extending a previous model, we tested various reviewer behavior, fair, random and strategic, and examined the impact of selecting multiple reviewers for the same author submission. We found that, when reviewer reliability is random or reviewers behave strategically, involving more than one reviewer per submission reduces evaluation bias. However, if scientists review scrupulously, multiple reviewers require an abnormal resource drain at the system level from research activities towards reviewing. This implies that reviewer selection mechanisms that protect the quality of the process against reviewer misbehavior might be economically unsustainable.

References

[1]

Alberts, B., B. Hanson, and K. L. Kelner. 2008. "Reviewing Peer Review." Science 321:15.

[2]

Allesina, S. 2012. "Modeling Peer Review: An Agent-Based Approach," Ideas in Ecology and Evolution 5(2): 27--35.

[3]

Birukou A., J. R. Wakeling, C. Bartolini, F. Casati, M. Marchese, K. Mirylenka, N. Osman, A. Ragone, C. Sierra, and A. Wassef. 2011. "Alternatives to Peer Review: Novel Approaches for Research Evaluation," Frontiers in Computational Neuroscience.

[4]

Bornmann, L. 2011. "Scientific Peer Review." Annual Review of Information Science and Technology, 45:199--245.

Digital Library

[5]

Couzin-Frankel, J. 2013. "Secretive and Subjective, Peer Review Proves Resistant to Study." Science 341: 1331.

[6]

Crocker, J. and M. L. Cooper. 2011. "Addressing Scientific Fraud." Science 334(6060):1182.

[7]

Gewin, V. 2012. "Uncovering Misconduct." Nature 485:137--139.

[8]

Grimaldo F. and M. Paolucci. 2014. "A Simulation of Disagreement for Control of Rational Cheating in Peer Review," Advances in Complex Systems, 99: 663--688.

[9]

Lee, C. J., C. R. Sugimoto, G. Zhang, and B. Cronin. 2013. "Bias in Peer Review," Journal of the American Society for Information Science and Technology 64: 2--17.

Digital Library

[10]

Leek, J. T., M. A. Taub, and F. J. Pineda. 2011. "Cooperation between Referees and Authors Increases Peer Review Accuracy," PLoS ONE 6(11): e26897.

[11]

Paolucci M. and F. Grimaldo. 2014. "Mechanism Change in a Simulation of Peer Review: From Junk Support to Elitism," Scientometrics, 99: 663--688.

Digital Library

[12]

Payette, N. 2012. "Agent-Based Models of Science." In Scharnhorst, A., K. Börner and P. Van den Besselaar (Eds.), Models of Science Dynamics: Encounters between Complexity Theory and Information Sciences. Springer Verlag, Heidelberg.

[13]

Roebber, P. J. and D. M. Schultz. 2011. "Peer Review, Program Officers and Science Funding." PLoS ONE 6(4):e18680: http://www.plosone.org/article/info:doi%2F10.1371%2Fjournal.pone.0018680.

[14]

Smith. R. 2006. "Peer Review. A Flawed Process at the Heart of Science and Journals." Journal of the Royal Society of Medicine 99: 759--760.

[15]

Squazzoni, F., G. Bravo and K. Takács, K. 2013. "Does Incentive Provision Increase the Quality of Peer Review? An Experimental Study." Research Policy 42(1): 287--294.

[16]

Squazzoni, F. and C. Gandelli. 2012. "Saint Matthews Strikes Again. An Agent-Based Model of Peer Review and the Scientific Community Structures." Journal of Informetrics 6:265--275.

[17]

Squazzoni, F. and C. Gandelli. 2013. "Opening the black box of peer review. An agent-based model of scientist behaviour." Journal of Artificial Societies and Social Simulation 16(2) 3: http://jasss.soc.surrey.ac.uk/16/2/3.html

[18]

Squazzoni, F. and K. Takács. 2011. "Social Simulation that 'Peers into Peer Review'." Journal of Artificial Societies and Social Simulation 14(4) 3: http://jasss.soc.surrey.ac.uk/14/4/3.html.

[19]

Thurner, S. and R. Hanel. 2011. "Peer Review in a World with Rational Scientists: Toward Selection of the Average." The European Physical Journal B 84:707--711.

Cited By

Bianchi FGrimaldo FBravo GSquazzoni F(2018)The peer review gameScientometrics10.1007/s11192-018-2825-4116:3(1401-1420)Online publication date: 1-Sep-2018
https://dl.acm.org/doi/10.1007/s11192-018-2825-4
Bacchelli ABeller MUchitel SOrso ARobillard M(2017)Double-blind review in software engineering venuesProceedings of the 39th International Conference on Software Engineering Companion10.1109/ICSE-C.2017.49(385-396)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.1109/ICSE-C.2017.49

Recommendations

Are You Reviewer 2: Three Ideas for Better Reviewing

Dear colleagues in the HRI community, I have been given the honor and privilege to write the editorial introduction for this final issue of the Journal of Human-Robot Interaction, before its reemergence as the ACM Transactions on Human-Robot ...
SEACHI 2016: Proceedings of the SEACHI 2016 on Smart Cities for Better Living with HCI and UX
CIVR '08: Proceedings of the 2008 international conference on Content-based image and video retrieval

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WSC '15: Proceedings of the 2015 Winter Simulation Conference

December 2015

4051 pages

ISBN:9781467397414

Sponsors

SIGSIM: ACM Special Interest Group on Simulation and Modeling

Publisher

IEEE Press

Publication History

Published: 06 December 2015

Check for updates

Qualifiers

Research-article

Conference

WSC '15

Sponsor:

SIGSIM

WSC '15: Winter Simulation Conference

December 6 - 9, 2015

California, Huntington Beach

Acceptance Rates

WSC '15 Paper Acceptance Rate 202 of 296 submissions, 68%;

Overall Acceptance Rate 3,413 of 5,075 submissions, 67%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
36
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bianchi FGrimaldo FBravo GSquazzoni F(2018)The peer review gameScientometrics10.1007/s11192-018-2825-4116:3(1401-1420)Online publication date: 1-Sep-2018
https://dl.acm.org/doi/10.1007/s11192-018-2825-4
Bacchelli ABeller MUchitel SOrso ARobillard M(2017)Double-blind review in software engineering venuesProceedings of the 39th International Conference on Software Engineering Companion10.1109/ICSE-C.2017.49(385-396)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.1109/ICSE-C.2017.49

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten