More Web Proxy on the site http://driver.im/

research-article

A large-scale study about quality and reproducibility of jupyter notebooks

Authors:

João Felipe Pimentel,

Leonardo Murta,

Vanessa Braganholo,

Juliana FreireAuthors Info & Claims

MSR '19: Proceedings of the 16th International Conference on Mining Software Repositories

Pages 507 - 517

https://doi.org/10.1109/MSR.2019.00077

Published: 26 May 2019 Publication History

Abstract

Jupyter Notebooks have been widely adopted by many different communities, both in science and industry. They support the creation of literate programming documents that combine code, text, and execution results with visualizations and all sorts of rich media. The self-documenting aspects and the ability to reproduce results have been touted as significant benefits of notebooks. At the same time, there has been growing criticism that the way notebooks are being used leads to unexpected behavior, encourage poor coding practices, and that their results can be hard to reproduce. To understand good and bad practices used in the development of real notebooks, we studied 1.4 million notebooks from GitHub. We present a detailed analysis of their characteristics that impact reproducibility. We also propose a set of best practices that can improve the rate of reproducibility and discuss open challenges that require further research and development.

References

[1]

D. E. Knuth, "Literate programming," The Computer Journal, vol. 27, no. 2, pp. 97--111, 1984.

Digital Library

[2]

H. Shen, "Interactive notebooks: Sharing the code," Nature News, vol. 515, no. 7525, p. 151, 2014.

[3]

A. Rule, A. Tabard, and J. D. Hollan, "Exploration and explanation in computational notebooks," in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ser. CHI '18. New York, NY, USA: ACM, 2018, pp. 32:1--32:12. {Online}. Available

Digital Library

[4]

F. Pérez and B. E. Granger, "Ipython: a system for interactive scientific computing," Computing in Science & Engineering, vol. 9, no. 3, pp. 21--29, 2007.

Digital Library

[5]

T. Kluyver, B. Ragan-Kelley, F. Pérez, B. E. Granger, M. Bussonnier, J. Frederic, K. Kelley, J. B. Hamrick, J. Grout, S. Corlay et al., "Jupyter notebooks ? a publishing format for reproducible computational workflows," in ELPUB, F. Loizides and B. Scmidt, Eds. Gttingen, Germany: IOS Press, 2016, pp. 87--90. {Online}. Available: https://eprints.soton.ac.uk/403913/

[6]

K. Pomogajko. (2015) Why I Don't Like Jupyter (FKA IPython Notebook). {Online}. Available: https://yihui.name/en/2018/09/notebook-war/

[7]

J. Grus, "I don't like notebooks." 2018, jupyterCon. {Online}. Available: https://conferences.oreilly.com/jupyter/jup-ny/public/schedule/detail/68282

[8]

A. Mueller. (2018) 5 reasons why jupyter notebooks suck. {Online}. Available: https://towardsdatascience.com/5-reasons-why-jupyter-notebooks-suck-4dc201e27086/

[9]

G. Wilson, D. A. Aruliah, C. T. Brown, N. P. Chue Hong, M. Davis, R. T. Guy, S. H. D. Haddock, K. D. Huff, I. M. Mitchell, M. D. Plumbley, B. Waugh, E. P. White, and P. Wilson, "Best practices for scientific computing," PLOS Biology, vol. 12, no. 1, pp. 1--7, 01 2014. {Online}. Available

[10]

W. L. Hürsch and C. V. Lopes, "Separation of concerns," Northeastern University, Tech. Rep., 1995.

[11]

G. J. Myers, C. Sandler, and T. Badgett, The art of software testing. John Wiley & Sons, 2011.

Digital Library

[12]

S. Horwitz and T. Reps, "The use of program dependence graphs in software engineering," in Proceedings of the 14th international conference on Software engineering. ACM, 1992, pp. 392--411.

Digital Library

[13]

M. B. Kery, M. Radensky, M. Arya, B. E. John, and B. A. Myers, "The story in the notebook: Exploratory data science using a literate programming tool," in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ser. CHI '18. New York, NY, USA: ACM, 2018, pp. 174:1--174:11. {Online}. Available

Digital Library

[14]

Neglectos. (2018) A preliminary analysis on the use of python notebooks. {Online}. Available: https://blog.bitergia.com/2018/04/02/a-preliminary-analysis-on-the-use-of-python-notebooks/

[15]

J. Freire, D. Koop, E. Santos, and C. Silva, "Provenance for computational tasks: A survey," Computing in Science & Engineering, vol. 10, no. 3, pp. 11--21, 2008.

Digital Library

[16]

V. Arnaoudova, M. Di Penta, and G. Antoniol, "Linguistic antipatterns: What they are and how developers perceive them," Empirical Software Engineering, vol. 21, no. 1, pp. 104--158, 2016.

Digital Library

[17]

Microsoft. (2018) Naming files, paths, and namespaces. Windows Dev Center. {Online}. Available: https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file

[18]

Tim and Doorknob. (2014) Is space not allowed in a filename? Unix & Linux. {Online}. Available: https://unix.stackexchange.com/q/148043

[19]

D. Lewine, POSIX programmers guide. "O'Reilly Media, Inc.", 1991.

[20]

P. Wiki. (2019) Python testing tools taxonomy. {Online}. Available: https://wiki.python.org/moin/PythonTestingToolsTaxonomy

[21]

T. Staley. (2017) Making git and jupyter notebooks play nice. {Online}. Available: http://timstaley.co.uk/posts/making-git-and-jupyter-notebooks-play-nice/

[22]

D. Koop and J. Patel, "Dataflow notebooks: encoding and tracking dependencies of cells," in 9th USENIX Workshop on the Theory and Practice of Provenance (TaPP 17). USENIX Association. Seattle, Washington: USENIX, 2017, pp. 1--7.

Digital Library

[23]

I. Anaconda. (2017) Conda documentation - managin environments. {Online}. Available: https://conda.io/docs/user-guide/tasks/manage-environments.html

[24]

Anaconda. (2018) Anaconda software distribution. {Online}. Available: https://www.anaconda.com

[25]

G. van Rossum, B. Warsaw, and N. Coghlan. (2001) Pep 8: style guide for python code. {Online}. Available: https://www.python.org/dev/peps/pep-0008/

[26]

N. Vavrová and V. Zaytsev, "Does python smell like java?" The Art, Science and Engineering of Programming (Programming), vol. 1, no. 2, pp. 11--1, 2017.

[27]

Udacity. (2017) Deep learning nanodegree foundation. {Online}. Available: https://github.com/udacity/deep-learning

[28]

D. Hook and D. Kelly, "Testing for trustworthiness in scientific software," in ICSE Workshop on Software Engineering for Computational Science and Engineering, SE-CSE 2009, Vancouver, BC, Canada, May 23, 2009. IEEE Computer Society, 2009, pp. 59--64. {Online}. Available

[29]

T. Burns and G. Ward. (2013) ipython-nose. {Online}. Available: https://github.com/taavi/ipython nose

[30]

J. F. Pimentel. (2016) ipython-unittest. {Online}. Available: https://github.com/JoaoFelipe/ipython-unittest

[31]

V. Garousi and B. Küçük, "Smells in software test code: A survey of knowledge in industry and academia," Journal of Systems and Software, vol. 138, pp. 52--81, 2018.

[32]

C. Collberg, T. Proebsting, G. Moraila, A. Shankaran, Z. Shi, and A. M. Warren, "Measuring reproducibility in computer systems research," Department of Computer Science, University of Arizona, Tech. Rep., 2014.

[33]

M. Tufano, F. Palomba, G. Bavota, M. Di Penta, R. Oliveto, A. De Lucia, and D. Poshyvanyk, "There and back again: Can you compile that snapshot?" Journal of Software: Evolution and Process, vol. 29, no. 4, p. e1838, 2017.

[34]

F. S. Chirigati, D. E. Shasha, and J. Freire, "Reprozip: Using provenance to support computational reproducibility." in TaPP, 2013.

[35]

N. Y. University. (2017) Making Jupyter Notebooks Reproducible with ReproZip. {Online}. Available: https://docs.reprozip.org/en/1.0.x/jupyter.html

[36]

E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian, "The promises and perils of mining github," in Proceedings of the 11th working conference on mining software repositories. ACM, 2014, pp. 92--101.

[37]

N. Munaiah, S. Kroh, C. Cabrey, and M. Nagappan, "Curating github for engineered software projects," Empirical Software Engineering, vol. 22, no. 6, pp. 3219--3253, 2017.

Digital Library

[38]

J. F. N. Pimentel, V. Braganholo, L. Murta, and J. Freire, "Collecting and analyzing provenance on interactive notebooks: when ipython meets noworkflow," in Workshop on the Theory and Practice of Provenance (TaPP). Edinburgh, Scotland: USENIX, 2015, pp. 155--167.

[39]

S. Samuel and B. König-Ries, "Provbook: Provenance-based semantic enrichment of interactive notebooks for reproducibility," in The 17th International Semantic Web Conference (ISWC), ser. ISWC. Monterey, California, USA: Springer, 2018, pp. 1--4.

Cited By

Allain KWeyde T(2024)JazzDAP: Collaborative Research Tools for Digital Jazz ArchivesProceedings of the 11th International Conference on Digital Libraries for Musicology10.1145/3660570.3660572(68-72)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3660570.3660572
Weber TEhe JMayer S(2024)Extending Jupyter with Multi-Paradigm EditorsProceedings of the ACM on Human-Computer Interaction10.1145/36602478:EICS(1-22)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3660247
Sato SNakamaru T(2024)Multiverse Notebook: Shifting Data Scientists to Time TravelersProceedings of the ACM on Programming Languages10.1145/36498388:OOPSLA1(754-783)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3649838
Show More Cited By

Recommendations

Assessing and restoring reproducibility of Jupyter notebooks
ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

Jupyter notebooks---documents that contain live code, equations, visualizations, and narrative text---now are among the most popular means to compute, present, discuss and disseminate scientific findings. In principle, Jupyter notebooks should easily ...
Understanding and improving the quality and reproducibility of Jupyter notebooks
Abstract
Jupyter Notebooks have been widely adopted by many different communities, both in science and industry. They support the creation of literate programming documents that combine code, text, and execution results with visualizations and other rich ...
ReproduceMeGit: A Visualization Tool for Analyzing Reproducibility of Jupyter Notebooks
Provenance and Annotation of Data and Processes
Abstract
Computational notebooks have gained widespread adoption among researchers from academia and industry as they support reproducible science. These notebooks allow users to combine code, text, and visualizations for easy sharing of experiments and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MSR '19: Proceedings of the 16th International Conference on Mining Software Repositories

May 2019

640 pages

General Chair:
Margaret-Anne Storey
University of Victoria, Canada
,
Program Chairs:
Bram Adams
MCIS, Polytechnique Montreal, Canada
,
Sonia Haiduc
Florida State University

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

IEEE Press

Publication History

Published: 26 May 2019

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '19

Sponsor:

SIGSOFT
IEEE-CS

ICSE '19: 41st International Conference on Software Engineering

May 26 - 27, 2019

Quebec, Montreal, Canada

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

57
Total Citations
View Citations
298
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)4

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Allain KWeyde T(2024)JazzDAP: Collaborative Research Tools for Digital Jazz ArchivesProceedings of the 11th International Conference on Digital Libraries for Musicology10.1145/3660570.3660572(68-72)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3660570.3660572
Weber TEhe JMayer S(2024)Extending Jupyter with Multi-Paradigm EditorsProceedings of the ACM on Human-Computer Interaction10.1145/36602478:EICS(1-22)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3660247
Sato SNakamaru T(2024)Multiverse Notebook: Shifting Data Scientists to Time TravelersProceedings of the ACM on Programming Languages10.1145/36498388:OOPSLA1(754-783)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3649838
Cabral RKalinowski MBaldassarre MVillamizar HEscovedo TLopes HBosch JLewis GCleland-Huang JMuccini H(2024)Investigating the Impact of SOLID Design Principles on Machine Learning Code UnderstandingProceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI10.1145/3644815.3644957(7-17)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3644815.3644957
Mostafavi Ghahfarokhi MAsgari AAbolnejadian MHeydarnoori ASpinellis DConstantinou EBacchelli A(2024)DistilKaggle: A Distilled Dataset of Kaggle Jupyter NotebooksProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644882(647-651)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644882
De Santana TNeto PDe Almeida EAhmed I(2024)Bug Analysis in Jupyter Notebook Projects: An Empirical StudyACM Transactions on Software Engineering and Methodology10.1145/364153933:4(1-34)Online publication date: 18-Apr-2024
https://dl.acm.org/doi/10.1145/3641539
Gilbertson CMundt MTeves JToribio SMilewicz R(2024)Towards Evidence-Based Software Quality Practices for Reproducibility: Practices and Aligned Software QualitiesProceedings of the 2nd ACM Conference on Reproducibility and Replicability10.1145/3641525.3663624(52-63)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3641525.3663624
Boufford NWonsil JPocock ASullivan JSeltzer MPasquier T(2024)Computational Experiment Comprehension using Provenance SummarizationProceedings of the 2nd ACM Conference on Reproducibility and Replicability10.1145/3641525.3663617(1-19)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3641525.3663617
Hüsing SSchulte CSparmann SBolte MStephenson BStone JBattestilli LRebelsky SShoop L(2024)Using Worked Examples for Engaging in Epistemic Programming ProjectsProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630961(443-449)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630961
Jayawardana YWannipurage DAbeysinghe EMarru S(2024)Enhancing Research Productivity: Seamless Integration of Personal Devices and HPC Resources with the Cybershuttle Notebook GatewayPractice and Experience in Advanced Research Computing 2024: Human Powered Computing10.1145/3626203.3670623(1-5)Online publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1145/3626203.3670623
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents