More Web Proxy on the site http://driver.im/

research-article

On the capability of static code analysis to detect security vulnerabilities

Authors:

Katerina Goseva-Popstojanova,

Andrei PerhinschiAuthors Info & Claims

Information and Software Technology, Volume 68, Issue C

Pages 18 - 33

https://doi.org/10.1016/j.infsof.2015.08.002

Published: 01 December 2015 Publication History

Abstract

Context: Static analysis of source code is a scalable method for discovery of software faults and security vulnerabilities. Techniques for static code analysis have matured in the last decade and many tools have been developed to support automatic detection.Objective: This research work is focused on empirical evaluation of the ability of static code analysis tools to detect security vulnerabilities with an objective to better understand their strengths and shortcomings.Method: We conducted an experiment which consisted of using the benchmarking test suite Juliet to evaluate three widely used commercial tools for static code analysis. Using design of experiments approach to conduct the analysis and evaluation and including statistical testing of the results are unique characteristics of this work. In addition to the controlled experiment, the empirical evaluation included case studies based on three open source programs.Results: Our experiment showed that 27% of C/C++ vulnerabilities and 11% of Java vulnerabilities were missed by all three tools. Some vulnerabilities were detected by only one or combination of two tools; 41% of C/C++ and 21% of Java vulnerabilities were detected by all three tools. More importantly, static code analysis tools did not show statistically significant difference in their ability to detect security vulnerabilities for both C/C++ and Java. Interestingly, all tools had median and mean of the per CWE recall values and overall recall across all CWEs close to or below 50%, which indicates comparable or worse performance than random guessing. While for C/C++ vulnerabilities one of the tools had better performance in terms of probability of false alarm than the other two tools, there was no statistically significant difference among tools' probability of false alarm for Java test cases.Conclusions: Despite recent advances in methods for static code analysis, the state-of-the-art tools are not very effective in detecting security vulnerabilities.

References

[1]

Cyberspace policy review: assuring a trusted and resilient information and communications infrastructure, 2009.

[2]

2014 Global report on the cost of cyber crime. Ponemon Institute research report, 2014.

[3]

Source code security analysis tool functional specification version 1.0, National Institute of Standards and Technology, Special Publication 500-268, 2007.

[4]

M. Zhivich, R.K. Cunningham, The real cost of software errors, IEEE Secur. Privacy, 7 (2009) 87-90.

Digital Library

[5]

G. McGraw, Addison-Wesley Professional, Boston, 2006.

[6]

J. Zheng, L. Williams, N. Nagappan, W. Snipes, J.P. Hudepohl, M.A. Vouk, On the value of static analysis for fault detection in software, IEEE Trans. Software Eng., 32 (2006) 240-253.

Digital Library

[7]

A. Austin, C. Holmgreen, L. Williams, A comparison of the efficiency and effectiveness of vulnerability discovery techniques, Inf. Software Technol., 55 (2013) 1279-1288.

[8]

B. Chess, G. McGraw, Static analysis for security, IEEE Secur. Privacy, 2 (2004) 76-79.

Digital Library

[9]

Common Weakness Enumeration, https://cwe.mitre.org/(accessed 21.12.14), 2013.

[10]

B. Chess, J. West, Addison-Wesley Software Security Series, Boston, 2007.

[11]

H. Do, S. Elbaum, G. Rothermel, Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact, Empirical Software Eng., 10 (2005) 405-435.

Digital Library

[12]

T. Boland, P.E. Black, The Juliet 1.1 C/C++ and Java test suite, IEEE Comput., 45 (2012) 88-90.

Digital Library

[13]

CAS Static Analysis Tool Study - Methodology, Center for Assured Software, National Security Agency, http://samate.nist.gov/docs/CAS (accessed 21.12.14), 2012.

[14]

V. Okun, A. Delaitre, P.E. Black, Report on the static analysis tool exposition (SATE) IV, NIST, 2013.

[15]

G. Díaz, J.R. Bermejo, Static analysis of source code security: assessment of tools against SAMATE tests, Inf. Software Technol., 55 (2013) 1462-1476.

Digital Library

[16]

K. Erno, Sticking to the facts II: scientific study of static analysis tools, Center for Assured Software, National Security Agency, 2012.

[17]

L.M.R. Velicheti, D.C. Feiock, M. Peiris, R. Raje, J.H. Hill, Towards modeling the behavior of static code analysis tools, ACM, 2014.

[18]

Static Analysis Tool Exposition, http://samate.nist.gov/SATE.html, (accessed 21.12.14), 2013.

[19]

M. Johns, M. Jodeit, W. Koeppl, M. Wimmer, ScanStud: evaluating static analysis tools, 2008.

[20]

P. Emanuelsson, U. Nilsson, A comparative study of industrial static analysis tools, Electron. Notes Theor. Comput. Sci., 217 (2008) 5-21.

Digital Library

[21]

D. Baca, K. Petersen, B. Carlsson, L. Lundberg, Static code analysis to detect software security vulnerabilities-does experience matter, IEEE, 2009.

[22]

D. Baca, B. Carlsson, K. Petersen, L. Lundberg, Improving software security with static automated code analysis in an industry setting, Software - Pract. Exp., 43 (2013) 259-279.

[23]

T. Hofer, École Polytechnique Fédérale de Lausanne, 2010.

[24]

N. Antunes, M. Vieira, Comparing the effectiveness of penetration testing and static code analysis on the detection of SQL injection vulnerabilities in Web services, IEEE, 2009.

[25]

C. Wohlin, P. Runeson, M. Höst, M.C. Ohlsson, B. Regnell, A. Wesslén, Springer-Verlag, New York, Dordrecht, London, 2012.

[26]

Juliet test suite, http://samate.nist.gov/SRD/testsuite.php (accessed 21.12.14), 2013.

[27]

P.E. Black, SAMATE and evaluating static analysis tools, Ada User J., 28 (2007) 184-188.

[28]

V. Okun, A. Delaitre, P.E. Black, Report on the Third Static Analysis Tool Exposition (SATE 2010), SP-500-283, US National Institute of Standards and Technology, 2011.

[29]

B. Kitchenham, L. Pickard, S.L. Pfleeger, Case studies for method and tool evaluation, IEEE Software, 12 (1995) 52-62.

Digital Library

Cited By

Esposito MFalaschi VFalessi D(2024)An Extensive Comparison of Static Application Security Testing ToolsProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661199(69-78)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661199
Charoenwet WThongtanunam PPham VTreude CChristakis MPradel M(2024)An Empirical Study of Static Analysis Tools for Secure Code ReviewProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680313(691-703)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680313
Li ZLiu ZWong WMa PWang S(2024)Evaluating C/C++ Vulnerability Detectability of Query-Based Static Application Security Testing ToolsIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2024.335478921:5(4600-4618)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1109/TDSC.2024.3354789
Show More Cited By

On the capability of static code analysis to detect security vulnerabilities
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

Hybrid analysis of executables to detect security vulnerabilities: security vulnerabilities
ISEC '09: Proceedings of the 2nd India software engineering conference

Detection of vulnerabilities in executables is one of the major challenges facing the software industry and is mainly due to the unavailability of the source code. In this work, we present a hybrid approach which is a combination of static and dynamic ...
Static analysis of source code security

ContextStatic analysis tools are used to discover security vulnerabilities in source code. They suffer from false negatives and false positives. A false positive is a reported vulnerability in a program that is not really a security problem. A false ...
Static Detection of User-specified Security Vulnerabilities in Client-side JavaScript
PLAS '16: Proceedings of the 2016 ACM Workshop on Programming Languages and Analysis for Security

Program defects tend to surface late in the development of programs, and they are hard to detect. Security vulnerabilities are particularly important defects to detect. They may cause sensitive information to be leaked or the system on which the program ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Information and Software Technology

Information and Software Technology Volume 68, Issue C

December 2015

98 pages

ISSN:0950-5849

Issue’s Table of Contents

Copyright © Elsevier B.V.

Publisher

Butterworth-Heinemann

United States

Publication History

Published: 01 December 2015

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Esposito MFalaschi VFalessi D(2024)An Extensive Comparison of Static Application Security Testing ToolsProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661199(69-78)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661199
Charoenwet WThongtanunam PPham VTreude CChristakis MPradel M(2024)An Empirical Study of Static Analysis Tools for Secure Code ReviewProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680313(691-703)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680313
Li ZLiu ZWong WMa PWang S(2024)Evaluating C/C++ Vulnerability Detectability of Query-Based Static Application Security Testing ToolsIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2024.335478921:5(4600-4618)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1109/TDSC.2024.3354789
Tian ZTian BLv JChen YChen L(2024)Enhancing vulnerability detection via AST decomposition and neural sub-tree encodingExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121865238:PBOnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121865
McLaughlin CLu Y(2024)Multi-class vulnerability prediction using value flow and graph neural networksNeural Computing and Applications10.1007/s00521-024-09819-336:25(15869-15891)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s00521-024-09819-3
Li KChen SFan LFeng RLiu HLiu CLiu YChen YChandra SBlincoe KTonella P(2023)Comparison and Evaluation on Static Application Security Testing (SAST) Tools for JavaProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616262(921-933)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616262
Lipp SBanescu SPretschner ARyu SSmaragdakis Y(2022)An empirical study on the effectiveness of static C code analyzers for vulnerability detectionProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3533767.3534380(544-555)Online publication date: 18-Jul-2022
https://dl.acm.org/doi/10.1145/3533767.3534380
Iovan MCruzes D(2022)Data-Driven Improvement of Static Application Security Testing Service: An Experience Report in VismaProduct-Focused Software Process Improvement10.1007/978-3-031-21388-5_11(157-170)Online publication date: 21-Nov-2022
https://dl.acm.org/doi/10.1007/978-3-031-21388-5_11
Vándor NMosolygó BHegelűs P(2022)Comparing ML-Based Predictions and Static Analyzer Tools for Vulnerability DetectionComputational Science and Its Applications – ICCSA 2022 Workshops10.1007/978-3-031-10542-5_7(92-105)Online publication date: 4-Jul-2022
https://dl.acm.org/doi/10.1007/978-3-031-10542-5_7
Siavvas MKehagias DTzovaras DGelenbe E(2021)A hierarchical model for quantifying software security based on static analysis alerts and software metricsSoftware Quality Journal10.1007/s11219-021-09555-029:2(431-507)Online publication date: 1-Jun-2021
https://dl.acm.org/doi/10.1007/s11219-021-09555-0
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents