[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3475716.3475769acmconferencesArticle/Chapter ViewAbstractPublication PagesesemConference Proceedingsconference-collections
research-article

A comparative study of vulnerability reporting by software composition analysis tools

Published: 11 October 2021 Publication History

Abstract

Background: Modern software uses many third-party libraries and frameworks as dependencies. Known vulnerabilities in these dependencies are a potential security risk. Software composition analysis (SCA) tools, therefore, are being increasingly adopted by practitioners to keep track of vulnerable dependencies. Aim: The goal of this study is to understand the difference in vulnerability reporting by various SCA tools. Understanding if and how existing SCA tools differ in their analysis may help security practitioners to choose the right tooling and identify future research needs. Method: We present an in-depth case study by comparing the analysis reports of 9 industry-leading SCA tools on a large web application, OpenMRS, composed of Maven (Java) and npm (JavaScript) projects. Results: We find that the tools vary in their vulnerability reporting. The count of reported vulnerable dependencies ranges from 17 to 332 for Maven and from 32 to 239 for npm projects across the studied tools. Similarly, the count of unique known vulnerabilities reported by the tools ranges from 36 to 313 for Maven and from 45 to 234 for npm projects. Our manual analysis of the tools' results suggest that accuracy of the vulnerability database is a key differentiator for SCA tools. Conclusion: We recommend that practitioners should not rely on any single tool at the present, as that can result in missing known vulnerabilities. We point out two research directions in the SCA space: i) establishing frameworks and metrics to identify false positives for dependency vulnerabilities; and ii) building automation technologies for continuous monitoring of vulnerability data from open source package ecosystems.

References

[1]
About semantic versioning. https://docs.npmjs.com/about-semantic-versioning.
[2]
Common vulnerability scoring system. https://en.wikipedia.org/wiki/Common_Vulnerability_Scoring_System.
[3]
Cve states. https://cve.mitre.org/cve/identifiers/.
[4]
Eclipse steady 3.1.14 (incubator project). https://eclipse.github.io/steady/about/.
[5]
Github advisory database. https://github.com/advisories.
[6]
How does dependency-check work? https://jeremylong.github.io/DependencyCheck/general/internals.html.
[7]
Mitre cve datagbase. https://cve.mitre.org/.
[8]
National vulnerability database. https://nvd.nist.gov/vuln.
[9]
Npm security advisories. https://www.npmjs.com/advisories.
[10]
npm security advisories. https://www.npmjs.com/advisories.
[11]
Openmrs around the world. http://guide.openmrs.org/en/.
[12]
Openmrs reference application distribution. https://wiki.openmrs.org/display/docs/OpenMRS+Reference+Application+Distribution.
[13]
Openmrs sdk. https://wiki.openmrs.org/display/docs/OpenMRS+SDK.
[14]
Snyk open source security management. https://support.snyk.io/hc/en-us/articles/360000925438-What-does-Snyk-access-and- store-when- scanning-a-project-.
[15]
Snyk vulnerability db. https://snyk.io/vuln.
[16]
Sonatype oss index. https://ossindex.sonatype.org/.
[17]
Steady vulnerability dataset. https://github.com/SAP/project-kb.
[18]
Victims software vulnerability scanner. https://blog.victi.ms/.
[19]
Whitesource bolt for github. https://github.com/apps/whitesource-bolt-for-github.
[20]
Whitesource vulnerability database. https://www.whitesourcesoftware.com/vulnerability-database/.
[21]
0patch.com. Security patching is hard. https://0patch.com/files/SecurityPatchingIsHard_2017.pdf.
[22]
Sepehr Amir-Mohammadian, Stephen Chong, and Christian Skalka. Correct audit logging: Theory and practice. In International Conference on Principles of Security and Trust, pages 139--162. Springer, 2016.
[23]
Benji Catabi-Kalman. Why do organizations trust snyk to win the open source security battle? https://snyk.io/blog/why-snyk-wins-open-source-security-battle/.
[24]
Steven P Crain. Open source security assessment as a class project. Journal of Computing Sciences in Colleges, 32(6):41-53, 2017.
[25]
Beatriz Sainz de Abajo and Agustín Llamas Ballestero. Overview of the most important open source software: analysis of the benefits of openmrs, openemr, and vista. In Telemedicine and e-health services, policies, and applications: Advancements and developments, pages 315--346. IGI Global, 2012.
[26]
Alexandre Decan, Tom Mens, and Eleni Constantinou. On the impact of security vulnerabilities in the npm package dependency network. In Proceedings of the 15th International Conference on Mining Software Repositories, pages 181-191, 2018.
[27]
Alexandre Decan, Tom Mens, and Philippe Grosjean. An empirical comparison of dependency network evolution in seven software packaging ecosystems. Empirical Software Engineering, 24(1):381-416, 2019.
[28]
Aurelien M. Delaitre, Bertrand C. Stivalet, Paul E. Black, Vadim Okun, Terry S. Cohen, and Athos Ribeiro. SATE V Report: Ten years of static analysis tool expositions. Technical report, National Institute of Standards and Technology, 2018.
[29]
Ibrahim Haddad. An open guide to evaluating software composition analysis tools, 2020.
[30]
JI Hejderup. In dependencies we trust: How vulnerable are dependencies in software modules? 2015.
[31]
Riivo Kikas, Georgios Gousios, Marlon Dumas, and Dietmar Pfahl. Structure and evolution of package dependency networks. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), pages 102--112. IEEE, 2017.
[32]
Sean Kinzer. Using cpes for open-source vulnerabilities? think again. https://www.veracode.com/blog/managing-appsec/using-cpes-open-source-vulnerabilities-think-again.
[33]
Raula Gaikovina Kula, Daniel M German, Ali Ouni, Takashi Ishio, and Katsuro Inoue. Do developers update their library dependencies? Empirical Software Engineering, 23(1):384-417, 2018.
[34]
Josephine Lamp, Carlos E Rubio-Medrano, Ziming Zhao, and Gail-Joon Ahn. The danger of missing instructions: a systematic analysis of security requirements for mcps. In 2018 IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), pages 94--99. IEEE, 2018.
[35]
Tobias Lauinger, Abdelberi Chaabane, Sajjad Arshad, William Robertson, Christo Wilson, and Engin Kirda. Thou shalt not depend on me: Analysing the use of outdated javascript libraries on the web. arXiv preprint arXiv:1811.00918, 2018.
[36]
National Institute of Standards and Technoloy (NIST). Guide for conducting risk assessments, nist special publication 800--30. https://csrc.nist.gov/publications/detail/sp/800-30/rev-1/final, September 2012. [Online; accessed 7-Oct-2020].
[37]
Top OWASP. Top 10-2017 the ten most critical web application security risks. OWASP_Top_10-2017_% 28en, 29, 2020.
[38]
Ivan Pashchenko, Henrik Plate, Serena Elisa Ponta, Antonino Sabetta, and Fabio Massacci. Vuln4real: A methodology for counting actually vulnerable dependencies. IEEE Transactions on Software Engineering, 2020.
[39]
Ivan Pashchenko, Duc-Ly Vu, and Fabio Massacci. A qualitative study of dependency management and its security implications. Proc. of CCS, 20.
[40]
Henrik Plate, Serena Elisa Ponta, and Antonino Sabetta. Impact assessment for vulnerabilities in open-source software libraries. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 411--420. IEEE, 2015.
[41]
Serena Elisa Ponta, Henrik Plate, and Antonino Sabetta. Beyond metadata: Code-centric and usage-based analysis of known vulnerabilities in open-source software. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 449--460. IEEE, 2018.
[42]
Serena Elisa Ponta, Henrik Plate, and Antonino Sabetta. Detection, assessment and mitigation of vulnerabilities in open source dependencies. Empirical Software Engineering, pages 1-41, 2020.
[43]
Apache Maven Project. "introduction to the dependency mechanism". http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html.
[44]
Teri Radichel. Why patching software is hard: Technical challenges. https://www.darkreading.com/vulnerabilities-and-threats/why-patching-software-is-hard-technical-\challenges-/a/d-id/1330181.
[45]
Syed Zain Rizvi, Philip WL Fong, Jason Crampton, and James Sellwood. Relationship-based access control for openmrs. arXiv preprint arXiv:1503.06154, 2015.
[46]
Li Sui, Jens Dietrich, Amjed Tahir, and George Fourtounis. On the recall of static call graph construction in practice. ICSE, 2020.
[47]
Synopsys. 2021 open source security and risk analysis report. https://www.synopsys.com/software-integrity/resources/analyst-reports/open-source-security-risk-analysis.html, 2021.
[48]
Inger Anne Tøndel, Martin Gilje Jaatun, Daniela Soares Cruzes, and Laurie Williams. Collaborative security risk estimation in agile software development. Information & Computer Security, 2019.
[49]
Rodrigo Elizalde Zapata, Raula Gaikovina Kula, Bodin Chinthanet, Takashi Ishio, Kenichi Matsumoto, and Akinori Ihara. Towards smoother library migrations: A look at vulnerable dependency migrations at function level for npm javascript packages. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 559--563. IEEE, 2018.
[50]
Yaqin Zhou and Asankhaya Sharma. Automated identification of security issues from commit messages and bug reports. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pages 914-919, 2017.

Cited By

View all
  • (2024)How Does Code Optimization Impact Third-party Library Detection for Android Applications?Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695554(1919-1931)Online publication date: 27-Oct-2024
  • (2024)BinEq - A Benchmark of Compiled Java Programs to Assess Alternative BuildsProceedings of the 2024 Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses10.1145/3689944.3696162(15-25)Online publication date: 19-Nov-2024
  • (2024)SBOM Ouverture: What We Need and What We HaveProceedings of the 19th International Conference on Availability, Reliability and Security10.1145/3664476.3669975(1-9)Online publication date: 30-Jul-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
ESEM '21: Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)
October 2021
368 pages
ISBN:9781450386654
DOI:10.1145/3475716
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 October 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. case study
  2. dependency
  3. security tools
  4. software composition analysis
  5. supply chain security
  6. vulnerability

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ESEM '21
Sponsor:

Acceptance Rates

ESEM '21 Paper Acceptance Rate 24 of 124 submissions, 19%;
Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)543
  • Downloads (Last 6 weeks)39
Reflects downloads up to 26 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)How Does Code Optimization Impact Third-party Library Detection for Android Applications?Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695554(1919-1931)Online publication date: 27-Oct-2024
  • (2024)BinEq - A Benchmark of Compiled Java Programs to Assess Alternative BuildsProceedings of the 2024 Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses10.1145/3689944.3696162(15-25)Online publication date: 19-Nov-2024
  • (2024)SBOM Ouverture: What We Need and What We HaveProceedings of the 19th International Conference on Availability, Reliability and Security10.1145/3664476.3669975(1-9)Online publication date: 30-Jul-2024
  • (2024)On the Way to SBOMs: Investigating Design Issues and Solutions in PracticeACM Transactions on Software Engineering and Methodology10.1145/365444233:6(1-25)Online publication date: 27-Jun-2024
  • (2024)Vulnerability Root Cause Function Locating For Java VulnerabilitiesProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3641225(444-446)Online publication date: 14-Apr-2024
  • (2024)Enhancing Security through Modularization: A Counterfactual Analysis of Vulnerability Propagation and Detection Precision2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM)10.1109/SCAM63643.2024.00019(94-105)Online publication date: 7-Oct-2024
  • (2024)Catch the Butterfly: Peeking into the Terms and Conflicts Among SPDX Licenses2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00056(477-488)Online publication date: 12-Mar-2024
  • (2024)KPCA for Open Source Security: Analyzing and Assessing Component Vulnerabilities2024 5th International Conference on Information Science, Parallel and Distributed Systems (ISPDS)10.1109/ISPDS62779.2024.10667551(688-693)Online publication date: 31-May-2024
  • (2024)Are We There Yet? Filling the Gap Between Binary Similarity Analysis and Binary Software Composition Analysis2024 IEEE 9th European Symposium on Security and Privacy (EuroS&P)10.1109/EuroSP60621.2024.00034(506-523)Online publication date: 8-Jul-2024
  • (2024)Evaluating Python Static Code Analysis Tools Using FAIR PrinciplesIEEE Access10.1109/ACCESS.2024.350349312(173647-173659)Online publication date: 2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media