More Web Proxy on the site http://driver.im/

research-article

A large scale study of programming languages and code quality in github

Authors:

Vladimir Filkov,

Premkumar DevanbuAuthors Info & Claims

FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering

Pages 155 - 165

https://doi.org/10.1145/2635868.2635922

Published: 11 November 2014 Publication History

Abstract

What is the effect of programming languages on software quality? This question has been a topic of much debate for a very long time. In this study, we gather a very large data set from GitHub (729 projects, 80 Million SLOC, 29,000 authors, 1.5 million commits, in 17 languages) in an attempt to shed some empirical light on this question. This reasonably large sample size allows us to use a mixed-methods approach, combining multiple regression modeling with visualization and text analytics, to study the effect of language features such as static v.s. dynamic typing, strong v.s. weak typing on software quality. By triangulating findings from different methods, and controlling for confounding effects such as team size, project size, and project history, we report that language design does have a significant, but modest effect on software quality. Most notably, it does appear that strong typing is modestly better than weak typing, and among functional languages, static typing is also somewhat better than dynamic typing. We also find that functional languages are somewhat better than procedural languages. It is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as project size, team size, and commit size. However, we hasten to caution the reader that even these modest effects might quite possibly be due to other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages.

References

[1]

Github archive, https://githubarchive.org/.

[2]

Github documentation, https://help.github.com/articles/stars.

[3]

Google big query, https://developers.google.com/bigquery/.

[4]

J. Armstrong, R. Virding, C. Wikström, and M. Williams. Concurrent programming in erlang. 1993.

Digital Library

[5]

P. Bhattacharya and I. Neamtiu. Assessing programming language impact on development and maintenance: A study on c and c++. In Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, pages 171–180, New York, NY, USA, 2011. ACM.

Digital Library

[6]

C. Bird, N. Nagappan, B. Murphy, H. Gall, and P. Devanbu. Don’t touch my code!: examining the effects of ownership on software quality. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pages 4–14. ACM, 2011.

Digital Library

[7]

D. M. Blei. Probabilistic topic models. Communications of the ACM, 55(4):77–84, 2012.

Digital Library

[8]

J. Cohen. Applied multiple regression/correlation analysis for the behavioral sciences. Lawrence Erlbaum, 2003.

[9]

H. CRAMÉR et al. Mathematical methods of statistics. Princeton University Press, 1946.

[10]

S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian. Selecting empirical methods for software engineering research. In Guide to advanced empirical software engineering, pages 285–311. Springer, 2008.

[11]

K. El Emam, S. Benlarbi, N. Goel, and S. N. Rai. The confounding effect of class size on the validity of object-oriented metrics. Software Engineering, IEEE Transactions on, 27(7):630–650, 2001.

Digital Library

[12]

GitHub. Linguist: https://github.com/github/linguist.

[13]

Google. http://golang.org/doc/effective_go. html#concurrency.

[14]

S. Hanenberg. An experiment about static and dynamic type systems: Doubts about the positive impact of static type systems on development time. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA ’10, pages 22–35, New York, NY, USA, 2010. ACM.

Digital Library

[15]

R. Harrison, L. Smaraweera, M. Dobie, and P. Lewis. Comparing programming paradigms: an evaluation of functional and object-oriented programs. Software Engineering Journal, 11(4):247–254, 1996.

[16]

D. E. Harter, M. S. Krishnan, and S. A. Slaughter. Effects of process maturity on quality, cycle time, and effort in software product development. Management Science, 46(4):451–466, 2000.

Digital Library

[17]

R. Hindley. The principal type-scheme of an object in combinatory logic. Transactions of the american mathematical society, pages 29–60, 1969.

[18]

M. Jump and K. S. McKinley. Cork: dynamic memory leak detection for garbage-collected languages. In ACM SIGPLAN Notices, volume 42, pages 31–38. ACM, 2007.

Digital Library

[19]

S. Kleinschmager, S. Hanenberg, R. Robbes, É. Tanter, and A. Stefik. Do static type systems improve the maintainability of software systems? an empirical study. In Program Comprehension (ICPC), 2012 IEEE 20th International Conference on, pages 153–162. IEEE, 2012.

[20]

Z. Li, L. Tan, X. Wang, S. Lu, Y. Zhou, and C. Zhai. Have things changed now? An empirical study of bug characteristics in modern open source software. In ASID ’06: Proceedings of the 1st workshop on Architectural and system support for improving software dependability, October 2006.

Digital Library

[21]

J. P. Marques De Sá. Applied statistics using spss, statistica and matlab. 2003.

[22]

C. Mayer, S. Hanenberg, R. Robbes, É. Tanter, and A. Stefik. An empirical study of the influence of static type systems on the usability of undocumented software. In ACM SIGPLAN Notices, volume 47, pages 683–702. ACM, 2012.

Digital Library

[23]

L. A. Meyerovich and A. S. Rabkin. Empirical analysis of programming language adoption. In Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications, pages 1––18. ACM, 2013.

Digital Library

[24]

R. Milner. A theory of type polymorphism in programming. Journal of computer and system sciences, 17(3):348–375, 1978.

[25]

A. Mockus and L. G. Votta. Identifying reasons for software changes using historic databases. In ICSM ’00: Proceedings of the International Conference on Software Maintenance, page 120. IEEE Computer Society, 2000.

Digital Library

[26]

M. Odersky, L. Spoon, and B. Venners. Programming in scala. Artima Inc, 2008.

Digital Library

[27]

V. Pankratius, F. Schmidt, and G. Garretón. Combining functional and imperative programming for multicore software: an empirical study evaluating scala and java. In Proceedings of the 2012 International Conference on Software Engineering, pages 123–133. IEEE Press, 2012.

Digital Library

[28]

T. Petricek and J. Skeet. Real World Functional Programming: With Examples in F# and C#. Manning Publications Co., 2009.

Digital Library

[29]

B. C. Pierce. Types and programming languages. MIT press, 2002.

Digital Library

[30]

A. A. Porter and L. G. Votta. An experiment to assess different defect detection methods for software requirements inspections. In Proceedings of the 16th International Conference on Software Engineering, ICSE ’94, pages 103–112, Los Alamitos, CA, USA, 1994. IEEE Computer Society Press.

Digital Library

[31]

D. Posnett, C. Bird, and P. Dévanbu. An empirical study on the influence of pattern roles on change-proneness. Empirical Software Engineering, 16(3):396–423, 2011.

Digital Library

[32]

F. Rahman and P. Devanbu. How, and why, process metrics are better. In Proceedings of the 2013 International Conference on Software Engineering, pages 432–441. IEEE Press, 2013.

Digital Library

[33]

L. Tan, C. Liu, Z. Li, X. Wang, Y. Zhou, and C. Zhai. Bug characteristics in open source software. Empirical Software Engineering, 2013.

Digital Library

[34]

V. Vapnik. The nature of statistical learning theory. springer, 2000.

Digital Library

[35]

Q. Vuong. Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica: Journal of the Econometric Society, pages 307–333, 1989.

[36]

E. J. Weyuker, T. J. Ostrand, and R. M. Bell. Do too many cooks spoil the broth? using the number of developers to enhance defect prediction models. Empirical Software Engineering, 13(5):539–559, 2008.

Digital Library

[37]

J. H. Zar. Significance Testing of the Spearman Rank Correlation Coefficient. Journal of the American Statistical Association, 67(339):578–580, 1972. Introduction Methodology Study Subjects Data Collection Categorizing Languages Identifying Project Domain Categorizing Bugs Statistical Methods Results Related Work Threats to Validity Conclusion Acknowledgements References

Cited By

Ahmad SChowdhury SHolmes R(2025)Impact of methodological choices on the analysis of code metrics and maintenanceJournal of Systems and Software10.1016/j.jss.2024.112263220(112263)Online publication date: Feb-2025
https://doi.org/10.1016/j.jss.2024.112263
Tran HAli NUnterkalmsteiner MBörstler JChatzipetrou P(2025)Quality attributes of test cases and test suites – importance & challenges from practitioners’ perspectivesSoftware Quality Journal10.1007/s11219-024-09698-w33:1Online publication date: 15-Jan-2025
https://doi.org/10.1007/s11219-024-09698-w
Al-Shameri Y(2024)Applications of Artificial Intelligence for Enhanced Bug Detection in Software DevelopmentIntegrating Technology in Problem-Solving Educational Practices10.4018/979-8-3693-6745-2.ch008(155-188)Online publication date: 25-Oct-2024
https://doi.org/10.4018/979-8-3693-6745-2.ch008
Show More Cited By

Index Terms

A large scale study of programming languages and code quality in github
1. Software and its engineering
  1. Software notations and tools
    1. General programming languages
      1. Language features

Recommendations

A large-scale study of programming languages and code quality in GitHub

What is the effect of programming languages on software quality? This question has been a topic of much debate for a very long time. In this study, we gather a very large data set from GitHub (728 projects, 63 million SLOC, 29,000 authors, 1.5 million ...
Definitional Interpreters for Higher-Order Programming Languages

Higher-order programming languages (i.e., languages in which procedures or labels can occur as values) are usually defined by interpreters that are themselves written in a programming language based on the lambda calculus (i.e., an applicative language ...
Definitional interpreters for higher-order programming languages
ACM '72: Proceedings of the ACM annual conference - Volume 2

Higher-order programming languages (i.e., languages in which procedures or labels can occur as values) are usually defined by interpreters which are themselves written in a programming language based on the lambda calculus (i.e., an applicative language ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering

November 2014

856 pages

ISBN:9781450330565

DOI:10.1145/2635868

General Chair:
Shing-Chi Cheung
Hong Kong University of Science and Technology, China
,
Program Chairs:
Alessandro Orso
Georgia Institute of Technology, USA
,
Margaret-Anne Storey
University of Victoria, Canada

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGSOFT/FSE'14

Sponsor:

SIGSOFT

SIGSOFT/FSE'14: 22nd ACM SIGSOFT Symposium on the Foundations of Software Engineering

November 16 - 21, 2014

Hong Kong, China

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

239
Total Citations
View Citations
4,349
Total Downloads

Downloads (Last 12 months)384
Downloads (Last 6 weeks)40

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ahmad SChowdhury SHolmes R(2025)Impact of methodological choices on the analysis of code metrics and maintenanceJournal of Systems and Software10.1016/j.jss.2024.112263220(112263)Online publication date: Feb-2025
https://doi.org/10.1016/j.jss.2024.112263
Tran HAli NUnterkalmsteiner MBörstler JChatzipetrou P(2025)Quality attributes of test cases and test suites – importance & challenges from practitioners’ perspectivesSoftware Quality Journal10.1007/s11219-024-09698-w33:1Online publication date: 15-Jan-2025
https://doi.org/10.1007/s11219-024-09698-w
Al-Shameri Y(2024)Applications of Artificial Intelligence for Enhanced Bug Detection in Software DevelopmentIntegrating Technology in Problem-Solving Educational Practices10.4018/979-8-3693-6745-2.ch008(155-188)Online publication date: 25-Oct-2024
https://doi.org/10.4018/979-8-3693-6745-2.ch008
Scheibel WBlum JLauterbach FAtzberger DDöllner J(2024)Integrated Visual Software Analytics on the GitHub PlatformComputers10.3390/computers1302003313:2(33)Online publication date: 25-Jan-2024
https://doi.org/10.3390/computers13020033
Sun DBoudouaia AZhu CLi Y(2024)Would ChatGPT-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? An empirical studyInternational Journal of Educational Technology in Higher Education10.1186/s41239-024-00446-521:1Online publication date: 22-Feb-2024
https://doi.org/10.1186/s41239-024-00446-5
Troppmann DFass AStaicu CFilkov VRay BZhou M(2024)Typed and Confused: Studying the Unexpected Dangers of Gradual TypingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695549(1858-1870)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695549
Aguiar LPaixao MCarmo RSoares ELeal AFreitas MGama E(2024)Multi-language Software Development in the LLM Era: Insights from Practitioners’ Conversations with ChatGPTProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3690755(489-495)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3690755
Wang HGao ZBi TGrundy JWang XWu MYang X(2024)What Makes a Good TODO Comment?ACM Transactions on Software Engineering and Methodology10.1145/366481133:6(1-30)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3664811
de Conto EGenest BEaswaran AAdams BZimmermann TOzkaya ILin DZhang J(2024)Function+Data Flow: A Framework to Specify Machine Learning Pipelines for Digital TwinningProceedings of the 1st ACM International Conference on AI-Powered Software10.1145/3664646.3664759(19-27)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3664646.3664759
Hassan MSalvador JSantu SRahman A(2024)State Reconciliation Defects in Infrastructure as CodeProceedings of the ACM on Software Engineering10.1145/36607901:FSE(1865-1888)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660790
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents