More Web Proxy on the site http://driver.im/

research-article

Peer Review on Open-Source Software Projects: Parameters, Statistical Models, and Theory

Authors:

Peter C. Rigby,

Daniel M. German,

Margaret-Anne StoreyAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 23, Issue 4

Article No.: 35, Pages 1 - 33

https://doi.org/10.1145/2594458

Published: 05 September 2014 Publication History

Abstract

Peer review is seen as an important quality-assurance mechanism in both industrial development and the open-source software (OSS) community. The techniques for performing inspections have been well studied in industry; in OSS development, software peer reviews are not as well understood.

To develop an empirical understanding of OSS peer review, we examine the review policies of 25 OSS projects and study the archival records of six large, mature, successful OSS projects. We extract a series of measures based on those used in traditional inspection experiments. We measure the frequency of review, the size of the contribution under review, the level of participation during review, the experience and expertise of the individuals involved in the review, the review interval, and the number of issues discussed during review. We create statistical models of the review efficiency, review interval, and effectiveness, the issues discussed during review, to determine which measures have the largest impact on review efficacy.

We find that OSS peer reviews are conducted asynchronously by empowered experts who focus on changes that are in their area of expertise. Reviewers provide timely, regular feedback on small changes. The descriptive statistics clearly show that OSS review is drastically different from traditional inspection.

References

[1]

A. Ackerman, L. Buchwald, and F. Lewski. 1989. Software inspections: An effective verification process. IEEE Softw. 6, 3, 31--36.

Digital Library

[2]

J. Asundi and R. Jayant. 2007. Patch review processes in open source software development communities: A comparative case study. In Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07). 10.

Digital Library

[3]

V. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Sørumgård, and M. Zelkowitz. 1996. The empirical investigation of perspective-based reading. Empir. Softw. Eng. 1, 2, 133--164.

[4]

C. Bird, A. Gourley, and P. Devanbu. 2007. Detecting patch submission and acceptance in OSS projects. In Proceedings of the 4th International Workshop on Mining Software Repositories (MSR). IEEE Computer Society, 4.

Digital Library

[5]

C. Bird, A. Gourley, P. Devanbu, M. Gertz, and A. Swaminathan. 2006. Mining email social networks. In Proceedings of the 3rd International Workshop on Mining Software Repositories (MSR). ACM Press, 137--143.

Digital Library

[6]

D. Bisant and J. Lyle. 1989. A two-person inspection method to improve programming productivity. IEEE Trans. Softw. Eng. 15, 10, 1294--1304.

Digital Library

[7]

S. Breu, R. Premraj, J. Sillito, and T. Zimmermann. 2010. Information needs in bug reports: Improving cooperation between developers and users. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. ACM Press, 301--310.

Digital Library

[8]

F. Buck. 1981. Indicators of quality inspections. Tech. rep. TR 21, IBM Syst. Commun. Division.

[9]

K. Burnham and D. Anderson. 2002. Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach. Springer Verlag.

[10]

A. Cockburn. 2004. Crystal Clear a Human-Powered Methodology for Small Teams. Addison-Wesley Professional.

Digital Library

[11]

J. Cohen. 2006. Best Kept Secrets of Peer Code Review. Smart Bear Inc.

[12]

M. Crawley. 2005. Statistics: An Introduction Using R. John Wiley and Sons.

[13]

T. Dinh-Trong and J. Bieman. 2005. The FreeBSD project: A replication case study of open source development. IEEE Trans. Softw. Eng. 31, 6, 481--494.

Digital Library

[14]

P. Eggert, M. Haertel, D. Hayes, R. Stallman, and L. Tower. 2002. diff -- Compare files line by line. Free Software Foundation, Inc.

[15]

S. G. Eick, C. R. Loader, M. D. Long, L. G. Votta, and S. V. Wiel. 1992. Estimating software fault content before coding. In Proceedings of the 14th International Conference on Software Engineering. 59--65.

Digital Library

[16]

M. Fagan. 1986. Advances in software inspections. IEEE Trans. Softw. Eng. 12, 7, 744--751.

Digital Library

[17]

M. Fagan. 2002. A history of software inspections. In Software Pioneers: Contributions to Software Engineering, Springer-Verlag, Inc., 562--573.

Digital Library

[18]

M. E. Fagan. 1976. Design and code inspections to reduce errors in program development. IBM Syst. J. 15, 3, 182--211.

Digital Library

[19]

R. T. Fielding and G. Kaiser. 1997. The Apache HTTP server project. IEEE Internet Comput. 1, 4, 88--90.

Digital Library

[20]

K. Fogel. 2005. Producing Open Source Software. O'Reilly.

Digital Library

[21]

Freshmeat. 2009. About Freshmeat: A catalog of software projects. http://freshmeat.net/about (Last accessed September 2009).

[22]

D. German. 2007. Using software distributions to understand the relationship among free and open source software projects. In Proceedings of the 4th International Workshop on Mining Software Repositories (MSR). IEEE, 8.

Digital Library

[23]

C. Gutwin, R. Penner, and K. Schneider. 2004. Group awareness in distributed software development. In Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW). 72--81.

Digital Library

[24]

S. Hambridge. 1995. Netiquette guidelines. http://edtech2.boisestate.edu/frankm/573/netiquette.html.

[25]

R. Hartill. 1998. Email. http://mail-archives.apache.org/mod_mbox/httpd-dev/199801.mbox&percnt;3CPine. NEB.3.96.980108232055.19805A-100000&commat;localhost&percnt;3E.

[26]

L. Hatton. 2008. Testing the value of checklists in code inspections. IEEE Softw. 25, 4, 82--88.

Digital Library

[27]

H. Heinzl and M. Mittlböck. 2003. Pseudo r-squared measures for poisson regression models with over- or underdispersion. Computat. Stat. Data Analy. 44, 1--2, 253--271.

[28]

A. Hindle, M. W. Godfrey, and R. C. Holt. 2008. Reading Beside the Lines: Indentation as a proxy for complexity metric. In Proceedings of the 16th IEEE International Conference on Program Comprehension (ICPC). IEEE Computer Society, 133--142.

Digital Library

[29]

J. Howison, M. Conklin, and K. Crowston. 2006. FLOSSmole: A collaborative repository for FLOSS research data and analyses. Int. J. Inf. Techno. Web Eng. 1, 17--26.

[30]

G. Jeong, S. Kim, T. Zimmermann, and K. Yi. 2009. Improving code review by predicting reviewers and acceptance of patches. ROSAEC Memo 2009-006, Research on Software Analysis for Error-Free Computing.

[31]

P. M. Johnson. 1998. Reengineering inspection. Commun. ACM 41, 2, 49--52.

Digital Library

[32]

P. M. Johnson and D. Tjahjono. 1998. Does every inspection really need a meeting&quest; Empir. Softw. Eng. 3, 1, 9--35.

Digital Library

[33]

P. Kampstra. 2008. Beanplot: A boxplot alternative for visual comparison of distributions. J. Stat. Softw. Code Snippets 1 28, 1--9.

[34]

P. Kennedy. 1981. Estimation with correctly interpreted dummy variables in semilogarithmic equations. Am. Econ. Rev. 71, 4, 801.

[35]

J. C. Knight and E. A. Myers. 1993. An improved inspection technique. Commun. ACM 36, 11, 51--61.

Digital Library

[36]

S. Kollanus and J. Koskinen. 2009. Survey of software inspection research. Open Softw. Eng. J. 3, 15--34.

[37]

O. Laitenberger and J. DeBaud. 2000. An encompassing life cycle centric survey of software inspection. J. Syst. Softw. 50, 1, 5--31.

Digital Library

[38]

C. Larman and V. Basili. 2003. Iterative and incremental developments: A brief history. Computer 36, 6, 47--56.

Digital Library

[39]

G. Lee and R. Cole. 2003. From a firm-based to a community-based model of knowledge creation: The case of the Linux kernel development. Org. Sci. 14, 6, 633--649.

Digital Library

[40]

S. Lussier. 2004. New tricks: How open source changed the way my team works. IEEE Softw. 21, 1, 68--72.

Digital Library

[41]

F. Macdonald and J. Miller. 1999. A comparison of computer support systems for software inspection. Autom. Softw. Eng. 6, 3, 291--313.

Digital Library

[42]

J. Maindonald and J. Braun. 2003. Data Analysis and Graphics Using R: An Example-Based Approach. Cambridge University Press.

[43]

J. Martin and W. T. Tsai. 1990. N-Fold inspection: A requirements analysis technique. ACM Commun. 33, 2, 225--232.

Digital Library

[44]

V. Mashayekhi, C. Feulner, and J. Riedl. 1994. CAIS: Collaborative asynchronous inspection of software. In Proceedings of the 2nd ACM SIGSOFT Symposium on Foundations of Software Engineering. ACM Press, 21--34.

Digital Library

[45]

P. McCullagh and J. Nelder. 1989. Generalized Linear Models (Monographs on Statistics and Applied Probability Book 37). Chapman & Hall/CRC.

[46]

A. Mockus, R. Fielding, and J. Herbsleb. 2000. A case study of open source software development: The Apache server. In Proceedings of the 22nd International Conference on Software Engineering (ICSE). 262--273.

Digital Library

[47]

A. Mockus, R. T. Fielding, and J. Herbsleb. 2002. Two case studies of open source software development: Apache and Mozilla. ACM Trans. Softw. Eng. Methodo. 11, 3, 1--38.

Digital Library

[48]

A. Mockus and J. D. Herbsleb. 2002. Expertise browser: A quantitative approach to identifying expertise. In Proceedings of the 24th International Conference on Software Engineering (ICSE'05). ACM Press, 503--512.

Digital Library

[49]

N. Nagappan and T. Ball. 2005. Use of relative code churn measures to predict system defect density. In Proceedings of the 27th International Conference on Software Engineering (ICSE). IEEE, 284--292.

Digital Library

[50]

Netcraft. 2010. Web Server Survey. http://news.netcraft.com/archives/2010/06/16/june-2010-web- server-survey.html.

[51]

M. Nurolahzade, S. M. Nasehi, S. H. Khandkar, and Shreya Rawal. 2009. The role of patch review in software evolution: An analysis of the Mozilla Firefox. In Proceedings of the International Workshop on Principles of Software Evolution. 9--18.

Digital Library

[52]

D. L. Parnas and D. M. Weiss. 1985. Active design reviews: Principles and practices. In Proceedings of the 8th International Conference on Software Engineering (ICSE). IEEE Computer Society Press, 132--136.

Digital Library

[53]

D. Perry, A. Porter, M. Wade, L. Votta, and J. Perpich. 2002. Reducing inspection interval in large-scale software development. IEEE Trans. Softw. Eng. 28, 7, 695--705.

Digital Library

[54]

A. Porter, H. Siy, A. Mockus, and L. Votta. 1998. Understanding the sources of variation in software inspections. ACM Trans. Softw. Eng. Methodol. 7, 1, 41--79.

Digital Library

[55]

A. Porter and L. Votta. 1994. An experiment to assess different defect detection methods for software requirements inspections. In Proceedings of the 16th International Conference on Software Engineering. IEEE Computer Society Press, 103--112.

Digital Library

[56]

A. A. Porter, H. P. Siy, and G. V. J. Lawrence. 1997. Understanding the effects of developer activities on inspection interval. In Proceedings of the 19th International Conference on Software Engineering (ICSE). 128--138.

Digital Library

[57]

R Development Core Team. 2011. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0.

[58]

E. S. Raymond. 1999. The Cathedral and the Bazaar. O'Reilly and Associates.

Digital Library

[59]

P. Resnick. 2001. RFC 2822: Internet message format. http:www.ietf.org/rfc/rfc2822.txt.

Digital Library

[60]

P. Rigby, B. Cleary, F. Painchaud, M. Storey, and D. German. 2012. Contemporary peer review in action: Lessons from open source development. IEEE Softw. 29, 6, 56--61.

Digital Library

[61]

P. C. Rigby. 2011. Understanding open source software peer review: Review processes, parameters and statistical models, and underlying behaviours and mechanisms. Ph.D. Dissertation, University of Victoria, Canada. http://hdl.handle.net/1828/3258.

Digital Library

[62]

P. C. Rigby and D. M. German. 2006. A preliminary examination of code review processes in open source projects. Tech. rep. DCS-305-IR. University of Victoria.

[63]

P. C. Rigby, D. M. German, and M.-A. Storey. 2008. Open source software peer review practices: A case study of the Apache server. In Proceedings of the 30th International Conference on Software Engineering (ICSE). 541--550.

Digital Library

[64]

P. C. Rigby and M.-A. Storey. 2011. Understanding broadcast based peer review on open source software projects. In Proceeding of the 33rd International Conference on Software Engineering (ICSE'11). ACM, New York, NY, 541--550.

Digital Library

[65]

C. Sauer, D. R. Jeffery, L. Land, and P. Yetton. 2000. The effectiveness of software development technical reviews: A behaviorally motivated program of research. IEEE Trans. Softw. Eng. 26, 1, 1--14.

Digital Library

[66]

E. Shihab, Z. Jiang, and A. Hassan. 2009. On the use of Internet relay chat (IRC) meetings by developers of the GNOME GTK+ project. In Proceedings of the 6th IEEE International Working Conference on Mining Software Repositories (MSR). IEEE Computer Society, 107--110.

Digital Library

[67]

SourceForge. 2010. SourceForge: Find and develop open source software. http://sourceforge.net/about (Last accessed December 2010).

[68]

L. G. Votta. 1993. Does every inspection need a meeting&quest; SIGSOFT Softw. Eng. Notes 18, 5, 107--114.

Digital Library

[69]

P. Weissgerber, D. Neu, and S. Diehl. 2008. Small patches get in&excl;. In Proceedings of the 5th IEEE International Working Conference on Mining Software Repositories (MSR). ACM, 67--76.

Digital Library

[70]

K. E. Wiegers. 2001. Peer Reviews in Software: A Practical Guide. Addison-Wesley Information Technology Series. Addison-Wesley.

[71]

R. K. Yin. 2003. Applications of Case Study Research (2nd Ed.). Applied Social Research Methods Series, Vol. 34. Sage Publications Inc.

Cited By

Chen KBao LHu XXia XYang X(2025)E-PRedictor: an approach for early prediction of pull request acceptanceScience China Information Sciences10.1007/s11432-022-3953-468:5Online publication date: 16-Jan-2025
https://doi.org/10.1007/s11432-022-3953-4
Wang SLin BChen LMao X(2024)Divide-and-Conquer: Automating Code Revisions via Localization-and-RevisionACM Transactions on Software Engineering and Methodology10.1145/369701334:3(1-26)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3697013
Zhang JWang LLi YJiang JWang TTao X(2024)Measuring and Mining Community Evolution in Developer Social Networks with Entropy-Based IndicesACM Transactions on Software Engineering and Methodology10.1145/368883234:1(1-43)Online publication date: 16-Aug-2024
https://dl.acm.org/doi/10.1145/3688832
Show More Cited By

Index Terms

Peer Review on Open-Source Software Projects: Parameters, Statistical Models, and Theory

Recommendations

Open source software peer review practices: a case study of the apache server
ICSE '08: Proceedings of the 30th international conference on Software engineering

Peer review is seen as an important quality assurance mechanism in both industrial development and the open source software (OSS) community. The techniques for performing inspections have been well studied in industry; in OSS development, peer reviews ...
Students' Engagement in Open Source Projects: An Analysis of Google Summer of Code
SBES '17: Proceedings of the XXXI Brazilian Symposium on Software Engineering

Several open source software (OSS) communities promote and participate in initiatives such as summers of code to foster contributions and attract new developers. However, little is known about how successful these initiatives are. As a case study, we ...
Contemporary Peer Review in Action: Lessons from Open Source Development

Do you use software peer reviews? Are you happy with your current code review practices? Even though formal inspection is recognized as one of the most effective ways to improve software quality, many software organizations struggle to effectively ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 23, Issue 4

Special Issue International Conference on Software Engineering (ICSE 2012) and Regular Papers

August 2014

232 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/2668018

Editors:
David S. Rosenblum
National University of Singapore, Singapore
,
Gail C. Murphy,
Mauro Pezzé

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 September 2014

Accepted: 01 March 2014

Revised: 01 December 2013

Received: 01 August 2011

Published in TOSEM Volume 23, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

79
Total Citations
View Citations
1,125
Total Downloads

Downloads (Last 12 months)90
Downloads (Last 6 weeks)15

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen KBao LHu XXia XYang X(2025)E-PRedictor: an approach for early prediction of pull request acceptanceScience China Information Sciences10.1007/s11432-022-3953-468:5Online publication date: 16-Jan-2025
https://doi.org/10.1007/s11432-022-3953-4
Wang SLin BChen LMao X(2024)Divide-and-Conquer: Automating Code Revisions via Localization-and-RevisionACM Transactions on Software Engineering and Methodology10.1145/369701334:3(1-26)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3697013
Zhang JWang LLi YJiang JWang TTao X(2024)Measuring and Mining Community Evolution in Developer Social Networks with Entropy-Based IndicesACM Transactions on Software Engineering and Methodology10.1145/368883234:1(1-43)Online publication date: 16-Aug-2024
https://dl.acm.org/doi/10.1145/3688832
Middleton JOre JStolee KRoychoudhury APaiva AAbreu RStorey M(2024)Barriers for Students During Code Change ComprehensionProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639227(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639227
Hajari FMalmir SMirsaeedi ERigby P(2024)Factoring Expertise, Workload, and Turnover Into Code Review RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.336675350:4(884-899)Online publication date: 23-Feb-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3366753
Tufano RDabić OMastropaolo ACiniselli MBavota G(2024)Code Review Automation: Strengths and Weaknesses of the State of the ArtIEEE Transactions on Software Engineering10.1109/TSE.2023.334817250:2(338-353)Online publication date: Feb-2024
https://doi.org/10.1109/TSE.2023.3348172
Tuna ESeaman CTüzün E(2024)Do code reviews lead to fewer code smells?Journal of Systems and Software10.1016/j.jss.2024.112101215(112101)Online publication date: Sep-2024
https://doi.org/10.1016/j.jss.2024.112101
Hong YTantithamthavorn CThongtanunam PAleti A(2024)Don’t forget to change these functions! recommending co-changed functions in modern code reviewInformation and Software Technology10.1016/j.infsof.2024.107547176:COnline publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107547
Coelho FTsantalis NMassoni TAlves E(2024)A qualitative study on refactorings induced by code reviewEmpirical Software Engineering10.1007/s10664-024-10560-730:1Online publication date: 31-Oct-2024
https://dl.acm.org/doi/10.1007/s10664-024-10560-7
Papotti AParamitha RMassacci F(2024)On the acceptance by code reviewers of candidate security patches suggested by Automated Program Repair toolsEmpirical Software Engineering10.1007/s10664-024-10506-z29:5Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1007/s10664-024-10506-z
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents