Dead Code Removal at Meta: Automatically Deleting Millions of Lines of Code and Petabytes of Deprecated Data
Authors: 
Will Shackleton, 
Katriel Cohn-Gordon, Peter C. Rigby, Rui Abreu, James Gill, 
Nachiappan Nagappan, Karim Nakad, Ioannis Papagiannis, Luke Petre, Giorgi Megreli, Patrick Riggs, James SaindonAuthors Info & Claims
Pages 1705 - 1715
Abstract
Software constantly evolves in response to user needs: new features are built, deployed, mature and grow old, and eventually their usage drops enough to merit switching them off. In any large codebase, this feature lifecycle can naturally lead to retaining unnecessary code and data. Removing these respects users’ privacy expectations, as well as helping engineers to work efficiently. In prior software engineering research, we have found little evidence of code deprecation or dead-code removal at industrial scale. We describe Systematic Code and Asset Removal Framework (SCARF), a product deprecation system to assist engineers working in large codebases. SCARF identifies unused code and data assets and safely removes them. It operates fully automatically, including committing code and dropping database tables. It also gathers developer input where it cannot take automated actions, leading to further removals. Dead code removal increases the quality and consistency of large codebases, aids with knowledge management and improves reliability. SCARF has had an important impact at Meta. In the last year alone, it has removed petabytes of data across 12.8 million distinct assets, and deleted over 104 million lines of code.
References
[1]
Luca Ardito, Riccardo Coppola, Luca Barbato, and Diego Verga. 2020. A Tool-Based Perspective on Software Code Maintainability Metrics: A Systematic Literature Review. Scientific Programming, 2020 (2020), Aug., 1–26. https://doi.org/10.1155/2020/8840389
[2]
I.D. Baxter and M. Mehlich. [n. d.]. Preprocessor conditional removal by simple partial evaluation. In Proceedings Eighth Working Conference on Reverse Engineering. IEEE Comput. Soc. https://doi.org/10.1109/wcre.2001.957833
[3]
Ira D. Baxter. 2002. DMS. In Proceedings of the International Workshop on Principles of Software Evolution. ACM. https://doi.org/10.1145/512035.512047
[4]
Nathan Bronson, Zach Amsden, George Cabrera, Prasad Chakka, Peter Dimov, Hui Ding, Jack Ferris, Anthony Giardullo, Sachin Kulkarni, and Harry Li. 2013. $TAO$: Facebook’s distributed data store for the social graph. In 2013 $USENIX$ Annual Technical Conference ($USENIX$$ATC$ 13). 49–60.
[5]
Nathan Bronson, Thomas Lento, and Janet L Wiener. 2015. Open data challenges at Facebook. In 2015 IEEE 31st international conference on data engineering. 1516–1519.
[6]
Michael D Brown and Santosh Pande. 2019. Carve: Practical security-focused software debloating using simple feature set mappings. In Proceedings of the 3rd ACM Workshop on Forming an Ecosystem Around Software Transformation. 1–7.
[7]
Yih-Fam Chen, Emden R Gansner, and Eleftherios Koutsofios. 1998. A C++ data model supporting reachability analysis and dead code detection. IEEE Transactions on Software Engineering, 24, 9 (1998), 682–694.
[8]
Siying Dong, Mark Callaghan, Leonidas Galanis, Dhruba Borthakur, Tony Savor, and Michael Strum. 2017. Optimizing Space Amplification in RocksDB. In CIDR. 3, 3.
[9]
Siying Dong, Andrew Kryczka, Yanqin Jin, and Michael Stumm. 2021. RocksDB: Evolution of Development Priorities in a Key-value Store Serving Large-scale Applications. ACM Transactions on Storage, 17, 4 (2021), Oct., 1–32. https://doi.org/10.1145/3483840
[10]
facebookincubator. [n. d.]. GitHub - facebookincubator/Glean: System for collecting, deriving and working with facts about source code. https://github.com/facebookincubator/glean
[11]
Amin Milani Fard and Ali Mesbah. 2013. Jsnose: Detecting javascript code smells. In 2013 IEEE 13th international working conference on Source Code Analysis and Manipulation (SCAM). 116–125.
[12]
Dror G. Feitelson, Eitan Frachtenberg, and Kent L. Beck. 2013. Development and Deployment at Facebook. IEEE Internet Computing, 17, 4 (2013), 8–17.
[13]
Stephen C Johnson. 1977. Lint, a C program checker. Bell Telephone Laboratories Murray Hill.
[14]
Guilherme Lacerda, Fabio Petrillo, Marcelo Pimenta, and Yann Gaël Guéhéneuc. 2020. Code smells and refactoring: A tertiary systematic review of challenges and observations. Journal of Systems and Software, 167 (2020), 110610.
[15]
Ivano Malavolta, Kishan Nirghin, Gian Luca Scoccia, Simone Romano, Salvatore Lombardi, Giuseppe Scanniello, and Patricia Lago. 2023. JavaScript Dead Code Identification, Elimination, and Empirical Assessment. IEEE Transactions on Software Engineering.
[16]
MariaDB Foundation. 2019. MariaDB. https://mariadb.org/
[17]
Simon Marlow. [n. d.]. Incremental indexing with Glean. https://glean.software/blog/incremental/
[18]
Phacility. 2011. Phabricator. https://www.phacility.com/phabricator/
[19]
Md Tajmilur Rahman, Louis-Philippe Querel, Peter C. Rigby, and Bram Adams. 2016. Feature Toggles: Practitioner Practices and a Case Study. In Proceedings of the 13th International Conference on Mining Software Repositories (MSR ’16). Association for Computing Machinery, New York, NY, USA. 201–211. isbn:9781450341868 https://doi.org/10.1145/2901739.2901745
[20]
Murali Krishna Ramanathan. 2020. Introducing Piranha: An Open Source Tool to Automatically Delete Stale Code. https://eng.uber.com/piranha/ Accessed: 2022-03-30
[21]
Murali Krishna Ramanathan, Lazaro Clapp, Rajkishore Barik, and Manu Sridharan. 2020. Piranha. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice. ACM. https://doi.org/10.1145/3377813.3381350
[22]
Nathan Rockenbach. 2022. AutoTransform. https://slack.engineering/autotransform-efficient-codebase-modification/
[23]
Mike Starr. [n. d.]. Dataswarm. https://www.youtube.com/watch?v=M0VCbhfQ3HQ
[24]
Liyin Tang, Vinod Venkataraman, and Charles Thayer. 2012. Facebook’s large scale monitoring system built on HBase. In Strata Conference, New York.
[25]
Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao, Prasad Chakka, Ning Zhang, Suresh Antony, Hao Liu, and Raghotham Murthy. 2010. Hive-a petabyte scale data warehouse using hadoop. In 2010 IEEE 26th international conference on data engineering (ICDE 2010). 996–1005.
[26]
Jeroen Vaelen. [n. d.]. Searching through Code at Scale. https://www.facebook.com/watch/?v=1911812842425144
[27]
David Wright and Paul De Hert. 2012. Privacy impact assessment. 6, Springer.
Index Terms
- Dead Code Removal at Meta: Automatically Deleting Millions of Lines of Code and Petabytes of Deprecated Data
Recommendations
Self-Modifying Code: A Provable Technique for Enhancing Program Obfuscation
This article describes how code obfuscation techniques aim to conceal the functionality of a program by mystifying the code so that it is unreadable or in an incomprehensible format. Since the objective of this article is to make a program obfuscated, ...
Beyond a Joke: Dead Code Elimination Can Delete Live Code
ICSE-NIER'24: Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging ResultsDead Code Elimination (DCE) is a fundamental compiler optimization technique that removes dead code (e.g., unreachable or reachable but whose results are unused) in the program to produce smaller or faster executables. However, since compiler ...
Tidy Up Your Source Code! Eliminating Wasteful Statements in Automatically Repaired Source Code
Product-Focused Software Process ImprovementAbstractAutomated program repair (APR) is a concept of automatically fixing bugs in source code to free developers from the burden of debugging. One of the issues facing search-based APR is that repaired code contains wasteful or meaningless statements ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2023
2215 pages
ISBN:9798400703270
DOI:10.1145/3611643
- General Chair:
- Satish Chandra,
- Program Chairs:
- Kelly Blincoe,
- Paolo Tonella
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 November 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ESEC/FSE '23
Sponsor:
ESEC/FSE '23: 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
December 3 - 9, 2023
CA, San Francisco, USA
Acceptance Rates
Overall Acceptance Rate 112 of 543 submissions, 21%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 272Total Downloads
- Downloads (Last 12 months)271
- Downloads (Last 6 weeks)10
Reflects downloads up to 12 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in