[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article

Peer Review on Open-Source Software Projects: Parameters, Statistical Models, and Theory

Published: 05 September 2014 Publication History

Abstract

Peer review is seen as an important quality-assurance mechanism in both industrial development and the open-source software (OSS) community. The techniques for performing inspections have been well studied in industry; in OSS development, software peer reviews are not as well understood.
To develop an empirical understanding of OSS peer review, we examine the review policies of 25 OSS projects and study the archival records of six large, mature, successful OSS projects. We extract a series of measures based on those used in traditional inspection experiments. We measure the frequency of review, the size of the contribution under review, the level of participation during review, the experience and expertise of the individuals involved in the review, the review interval, and the number of issues discussed during review. We create statistical models of the review efficiency, review interval, and effectiveness, the issues discussed during review, to determine which measures have the largest impact on review efficacy.
We find that OSS peer reviews are conducted asynchronously by empowered experts who focus on changes that are in their area of expertise. Reviewers provide timely, regular feedback on small changes. The descriptive statistics clearly show that OSS review is drastically different from traditional inspection.

References

[1]
A. Ackerman, L. Buchwald, and F. Lewski. 1989. Software inspections: An effective verification process. IEEE Softw. 6, 3, 31--36.
[2]
J. Asundi and R. Jayant. 2007. Patch review processes in open source software development communities: A comparative case study. In Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07). 10.
[3]
V. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Sørumgård, and M. Zelkowitz. 1996. The empirical investigation of perspective-based reading. Empir. Softw. Eng. 1, 2, 133--164.
[4]
C. Bird, A. Gourley, and P. Devanbu. 2007. Detecting patch submission and acceptance in OSS projects. In Proceedings of the 4th International Workshop on Mining Software Repositories (MSR). IEEE Computer Society, 4.
[5]
C. Bird, A. Gourley, P. Devanbu, M. Gertz, and A. Swaminathan. 2006. Mining email social networks. In Proceedings of the 3rd International Workshop on Mining Software Repositories (MSR). ACM Press, 137--143.
[6]
D. Bisant and J. Lyle. 1989. A two-person inspection method to improve programming productivity. IEEE Trans. Softw. Eng. 15, 10, 1294--1304.
[7]
S. Breu, R. Premraj, J. Sillito, and T. Zimmermann. 2010. Information needs in bug reports: Improving cooperation between developers and users. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. ACM Press, 301--310.
[8]
F. Buck. 1981. Indicators of quality inspections. Tech. rep. TR 21, IBM Syst. Commun. Division.
[9]
K. Burnham and D. Anderson. 2002. Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach. Springer Verlag.
[10]
A. Cockburn. 2004. Crystal Clear a Human-Powered Methodology for Small Teams. Addison-Wesley Professional.
[11]
J. Cohen. 2006. Best Kept Secrets of Peer Code Review. Smart Bear Inc.
[12]
M. Crawley. 2005. Statistics: An Introduction Using R. John Wiley and Sons.
[13]
T. Dinh-Trong and J. Bieman. 2005. The FreeBSD project: A replication case study of open source development. IEEE Trans. Softw. Eng. 31, 6, 481--494.
[14]
P. Eggert, M. Haertel, D. Hayes, R. Stallman, and L. Tower. 2002. diff -- Compare files line by line. Free Software Foundation, Inc.
[15]
S. G. Eick, C. R. Loader, M. D. Long, L. G. Votta, and S. V. Wiel. 1992. Estimating software fault content before coding. In Proceedings of the 14th International Conference on Software Engineering. 59--65.
[16]
M. Fagan. 1986. Advances in software inspections. IEEE Trans. Softw. Eng. 12, 7, 744--751.
[17]
M. Fagan. 2002. A history of software inspections. In Software Pioneers: Contributions to Software Engineering, Springer-Verlag, Inc., 562--573.
[18]
M. E. Fagan. 1976. Design and code inspections to reduce errors in program development. IBM Syst. J. 15, 3, 182--211.
[19]
R. T. Fielding and G. Kaiser. 1997. The Apache HTTP server project. IEEE Internet Comput. 1, 4, 88--90.
[20]
K. Fogel. 2005. Producing Open Source Software. O'Reilly.
[21]
Freshmeat. 2009. About Freshmeat: A catalog of software projects. http://freshmeat.net/about (Last accessed September 2009).
[22]
D. German. 2007. Using software distributions to understand the relationship among free and open source software projects. In Proceedings of the 4th International Workshop on Mining Software Repositories (MSR). IEEE, 8.
[23]
C. Gutwin, R. Penner, and K. Schneider. 2004. Group awareness in distributed software development. In Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW). 72--81.
[24]
S. Hambridge. 1995. Netiquette guidelines. http://edtech2.boisestate.edu/frankm/573/netiquette.html.
[25]
R. Hartill. 1998. Email. http://mail-archives.apache.org/mod_mbox/httpd-dev/199801.mbox%3CPine. NEB.3.96.980108232055.19805A-100000@localhost%3E.
[26]
L. Hatton. 2008. Testing the value of checklists in code inspections. IEEE Softw. 25, 4, 82--88.
[27]
H. Heinzl and M. Mittlböck. 2003. Pseudo r-squared measures for poisson regression models with over- or underdispersion. Computat. Stat. Data Analy. 44, 1--2, 253--271.
[28]
A. Hindle, M. W. Godfrey, and R. C. Holt. 2008. Reading Beside the Lines: Indentation as a proxy for complexity metric. In Proceedings of the 16th IEEE International Conference on Program Comprehension (ICPC). IEEE Computer Society, 133--142.
[29]
J. Howison, M. Conklin, and K. Crowston. 2006. FLOSSmole: A collaborative repository for FLOSS research data and analyses. Int. J. Inf. Techno. Web Eng. 1, 17--26.
[30]
G. Jeong, S. Kim, T. Zimmermann, and K. Yi. 2009. Improving code review by predicting reviewers and acceptance of patches. ROSAEC Memo 2009-006, Research on Software Analysis for Error-Free Computing.
[31]
P. M. Johnson. 1998. Reengineering inspection. Commun. ACM 41, 2, 49--52.
[32]
P. M. Johnson and D. Tjahjono. 1998. Does every inspection really need a meeting? Empir. Softw. Eng. 3, 1, 9--35.
[33]
P. Kampstra. 2008. Beanplot: A boxplot alternative for visual comparison of distributions. J. Stat. Softw. Code Snippets 1 28, 1--9.
[34]
P. Kennedy. 1981. Estimation with correctly interpreted dummy variables in semilogarithmic equations. Am. Econ. Rev. 71, 4, 801.
[35]
J. C. Knight and E. A. Myers. 1993. An improved inspection technique. Commun. ACM 36, 11, 51--61.
[36]
S. Kollanus and J. Koskinen. 2009. Survey of software inspection research. Open Softw. Eng. J. 3, 15--34.
[37]
O. Laitenberger and J. DeBaud. 2000. An encompassing life cycle centric survey of software inspection. J. Syst. Softw. 50, 1, 5--31.
[38]
C. Larman and V. Basili. 2003. Iterative and incremental developments: A brief history. Computer 36, 6, 47--56.
[39]
G. Lee and R. Cole. 2003. From a firm-based to a community-based model of knowledge creation: The case of the Linux kernel development. Org. Sci. 14, 6, 633--649.
[40]
S. Lussier. 2004. New tricks: How open source changed the way my team works. IEEE Softw. 21, 1, 68--72.
[41]
F. Macdonald and J. Miller. 1999. A comparison of computer support systems for software inspection. Autom. Softw. Eng. 6, 3, 291--313.
[42]
J. Maindonald and J. Braun. 2003. Data Analysis and Graphics Using R: An Example-Based Approach. Cambridge University Press.
[43]
J. Martin and W. T. Tsai. 1990. N-Fold inspection: A requirements analysis technique. ACM Commun. 33, 2, 225--232.
[44]
V. Mashayekhi, C. Feulner, and J. Riedl. 1994. CAIS: Collaborative asynchronous inspection of software. In Proceedings of the 2nd ACM SIGSOFT Symposium on Foundations of Software Engineering. ACM Press, 21--34.
[45]
P. McCullagh and J. Nelder. 1989. Generalized Linear Models (Monographs on Statistics and Applied Probability Book 37). Chapman & Hall/CRC.
[46]
A. Mockus, R. Fielding, and J. Herbsleb. 2000. A case study of open source software development: The Apache server. In Proceedings of the 22nd International Conference on Software Engineering (ICSE). 262--273.
[47]
A. Mockus, R. T. Fielding, and J. Herbsleb. 2002. Two case studies of open source software development: Apache and Mozilla. ACM Trans. Softw. Eng. Methodo. 11, 3, 1--38.
[48]
A. Mockus and J. D. Herbsleb. 2002. Expertise browser: A quantitative approach to identifying expertise. In Proceedings of the 24th International Conference on Software Engineering (ICSE'05). ACM Press, 503--512.
[49]
N. Nagappan and T. Ball. 2005. Use of relative code churn measures to predict system defect density. In Proceedings of the 27th International Conference on Software Engineering (ICSE). IEEE, 284--292.
[50]
Netcraft. 2010. Web Server Survey. http://news.netcraft.com/archives/2010/06/16/june-2010-web- server-survey.html.
[51]
M. Nurolahzade, S. M. Nasehi, S. H. Khandkar, and Shreya Rawal. 2009. The role of patch review in software evolution: An analysis of the Mozilla Firefox. In Proceedings of the International Workshop on Principles of Software Evolution. 9--18.
[52]
D. L. Parnas and D. M. Weiss. 1985. Active design reviews: Principles and practices. In Proceedings of the 8th International Conference on Software Engineering (ICSE). IEEE Computer Society Press, 132--136.
[53]
D. Perry, A. Porter, M. Wade, L. Votta, and J. Perpich. 2002. Reducing inspection interval in large-scale software development. IEEE Trans. Softw. Eng. 28, 7, 695--705.
[54]
A. Porter, H. Siy, A. Mockus, and L. Votta. 1998. Understanding the sources of variation in software inspections. ACM Trans. Softw. Eng. Methodol. 7, 1, 41--79.
[55]
A. Porter and L. Votta. 1994. An experiment to assess different defect detection methods for software requirements inspections. In Proceedings of the 16th International Conference on Software Engineering. IEEE Computer Society Press, 103--112.
[56]
A. A. Porter, H. P. Siy, and G. V. J. Lawrence. 1997. Understanding the effects of developer activities on inspection interval. In Proceedings of the 19th International Conference on Software Engineering (ICSE). 128--138.
[57]
R Development Core Team. 2011. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0.
[58]
E. S. Raymond. 1999. The Cathedral and the Bazaar. O'Reilly and Associates.
[59]
P. Resnick. 2001. RFC 2822: Internet message format. http:www.ietf.org/rfc/rfc2822.txt.
[60]
P. Rigby, B. Cleary, F. Painchaud, M. Storey, and D. German. 2012. Contemporary peer review in action: Lessons from open source development. IEEE Softw. 29, 6, 56--61.
[61]
P. C. Rigby. 2011. Understanding open source software peer review: Review processes, parameters and statistical models, and underlying behaviours and mechanisms. Ph.D. Dissertation, University of Victoria, Canada. http://hdl.handle.net/1828/3258.
[62]
P. C. Rigby and D. M. German. 2006. A preliminary examination of code review processes in open source projects. Tech. rep. DCS-305-IR. University of Victoria.
[63]
P. C. Rigby, D. M. German, and M.-A. Storey. 2008. Open source software peer review practices: A case study of the Apache server. In Proceedings of the 30th International Conference on Software Engineering (ICSE). 541--550.
[64]
P. C. Rigby and M.-A. Storey. 2011. Understanding broadcast based peer review on open source software projects. In Proceeding of the 33rd International Conference on Software Engineering (ICSE'11). ACM, New York, NY, 541--550.
[65]
C. Sauer, D. R. Jeffery, L. Land, and P. Yetton. 2000. The effectiveness of software development technical reviews: A behaviorally motivated program of research. IEEE Trans. Softw. Eng. 26, 1, 1--14.
[66]
E. Shihab, Z. Jiang, and A. Hassan. 2009. On the use of Internet relay chat (IRC) meetings by developers of the GNOME GTK+ project. In Proceedings of the 6th IEEE International Working Conference on Mining Software Repositories (MSR). IEEE Computer Society, 107--110.
[67]
SourceForge. 2010. SourceForge: Find and develop open source software. http://sourceforge.net/about (Last accessed December 2010).
[68]
L. G. Votta. 1993. Does every inspection need a meeting? SIGSOFT Softw. Eng. Notes 18, 5, 107--114.
[69]
P. Weissgerber, D. Neu, and S. Diehl. 2008. Small patches get in!. In Proceedings of the 5th IEEE International Working Conference on Mining Software Repositories (MSR). ACM, 67--76.
[70]
K. E. Wiegers. 2001. Peer Reviews in Software: A Practical Guide. Addison-Wesley Information Technology Series. Addison-Wesley.
[71]
R. K. Yin. 2003. Applications of Case Study Research (2nd Ed.). Applied Social Research Methods Series, Vol. 34. Sage Publications Inc.

Cited By

View all
  • (2025)A qualitative study on refactorings induced by code reviewEmpirical Software Engineering10.1007/s10664-024-10560-730:1Online publication date: 1-Feb-2025
  • (2024)Measuring and Mining Community Evolution in Developer Social Networks with Entropy-Based IndicesACM Transactions on Software Engineering and Methodology10.1145/368883234:1(1-43)Online publication date: 16-Aug-2024
  • (2024)Barriers for Students During Code Change ComprehensionProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639227(1-13)Online publication date: 20-May-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology
ACM Transactions on Software Engineering and Methodology  Volume 23, Issue 4
Special Issue International Conference on Software Engineering (ICSE 2012) and Regular Papers
August 2014
232 pages
ISSN:1049-331X
EISSN:1557-7392
DOI:10.1145/2668018
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 September 2014
Accepted: 01 March 2014
Revised: 01 December 2013
Received: 01 August 2011
Published in TOSEM Volume 23, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Peer review
  2. inspection
  3. mining software repositories
  4. open-source software

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)79
  • Downloads (Last 6 weeks)6
Reflects downloads up to 30 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2025)A qualitative study on refactorings induced by code reviewEmpirical Software Engineering10.1007/s10664-024-10560-730:1Online publication date: 1-Feb-2025
  • (2024)Measuring and Mining Community Evolution in Developer Social Networks with Entropy-Based IndicesACM Transactions on Software Engineering and Methodology10.1145/368883234:1(1-43)Online publication date: 16-Aug-2024
  • (2024)Barriers for Students During Code Change ComprehensionProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639227(1-13)Online publication date: 20-May-2024
  • (2024)Factoring Expertise, Workload, and Turnover Into Code Review RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.336675350:4(884-899)Online publication date: 23-Feb-2024
  • (2024)Code Review Automation: Strengths and Weaknesses of the State of the ArtIEEE Transactions on Software Engineering10.1109/TSE.2023.334817250:2(338-353)Online publication date: Feb-2024
  • (2024)Do code reviews lead to fewer code smells?Journal of Systems and Software10.1016/j.jss.2024.112101215(112101)Online publication date: Sep-2024
  • (2024)Don’t forget to change these functions! recommending co-changed functions in modern code reviewInformation and Software Technology10.1016/j.infsof.2024.107547176:COnline publication date: 1-Dec-2024
  • (2024)On the acceptance by code reviewers of candidate security patches suggested by Automated Program Repair toolsEmpirical Software Engineering10.1007/s10664-024-10506-z29:5Online publication date: 3-Aug-2024
  • (2024)Teaching Theorizing in Software Engineering ResearchHandbook on Teaching Empirical Software Engineering10.1007/978-3-031-71769-7_3(31-69)Online publication date: 25-Dec-2024
  • (2023)Hypothesis testing using RDigital Diagnostics10.17816/DD1213684:2(238-247)Online publication date: 12-Jul-2023
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media