More Web Proxy on the site http://driver.im/

research-article

Evaluating bug finders: test and measurement of static code analyzers

Authors:

Aurelien Delaitre,

Bertrand Stivalet,

Elizabeth Fong,

Vadim OkunAuthors Info & Claims

COUFLESS '15: Proceedings of the First International Workshop on Complex faUlts and Failures in LargE Software Systems

Pages 14 - 20

Published: 16 May 2015 Publication History

Abstract

Software static analysis is one of many options for finding bugs in software. Like compilers, static analyzers take a program as input. This paper covers tools that examine source codewithout executing itand output bug reports. Static analysis is a complex and generally undecidable problem. Most tools resort to approximation to overcome these obstacles and it sometimes leads to incorrect results. Therefore, tool effectiveness needs to be evaluated. Several characteristics of the tools should be examined. First, what types of bugs can they find? Second, what proportion of bugs do they report? Third, what percentage of findings is correct? These questions can be answered by one or more metrics. But to calculate these, we need test cases having certain characteristics: statistical significance, ground truth, and relevance. Test cases with all three attributes are out of reach, but we can use combinations of only two to calculate the metrics.

The results in this paper were collected during Static Analysis Tool Exposition (SATE) V, where participants ran 14 static analyzers on the test sets we provided and submitted their reports to us for analysis. Tools had considerably different support for most bug classes. Some tools discovered significantly more bugs than others or generated mostly accurate warnings, while others reported wrong findings more frequently. Using the metrics, an evaluator can compare candidates and select the tool that aligns best with his or her objectives. In addition, our results confirm that the bugs most commonly found by tools are among the most common and important bugs in software. We also observed that code complexity is a major hindrance for static analyzers and detailed which code constructs tools handle well and which impede their analysis.

References

[1]

D. Wheeler, and R. S. Moorthy, "State-of-the-art resources (SOAR) for software vulnerability detection, test, and evaluation," Institute of Defense Analyses IDA Paper P-5061, July 2014.

[2]

National Institute of Standards and Technology, "SAMATE - software assurance metrics and tool evaluation" Available from: http://samate.nist.gov

[3]

V. Okun, R. Gaucher, and P. E. Black, "Static analysis tool exposition (SATE) 2008, NIST Special Publication 500--279, June 2009.

[4]

V. Okun, A. Delaitre, and P. E. Black, "The second static analysis tool exposition (SATE) 2009," NIST Special Publication 500--287, June 2010.

[5]

V. Okun, A. Delaitre, and P. E. Black, "Report on the third static analysis tool exposition (SATE) 2010," NIST Special Publication 500--283, October 2011.

[6]

V. Okun, A. Delaitre, and P. E. Black, "Report on the static analysis tool exposition (SATE) IV," NIST Special Publication 500--297, January 2013, http://dx.doi.org/10.6028/NIST.SP.500--297

[7]

A. Delaitre et al., "SATE V report," NIST Special Publication, unpublished.

[8]

A. Delaitre, V. Okun, and E. Fong, "Of massive static analysis data," Proceeding of Software Security and Reliability (SERE 2013), June 2013.

Digital Library

[9]

G. Diaz and J. R. Bermejo, "Static analysis of source code security: assessment of tools against SAMATE tests," Information and software technology 55 (2013) 1462-1476. (Also available at http://dx.doi.org/10.10.16/j.infsof.2013.02.005)

Digital Library

[10]

Software Assurance Reference Dataset (SARD) project web site. http://samate.nist.gov/SARD

[11]

J. A. Kupsch and B. P. Miller, "Manual vs automated vulnerability assessment: A case study," First international workshop on managing insider security threats (MIST 2009), West Lafayette, IN, June 2009.

[12]

C. Willis, "CAS static analysis tool study overview," In proc. eleventh annual high confidence software and systems conference, page 86, National Security Agency, 2011, http://hcss.csp.org

[13]

Common Weakness Enumeration (CWE). The MITRE corporation, http://cwe.mitre.org

[14]

Paul E. Black, "Static analyzers: seat belts for your code," May-June 2012, IEEE Security & Privacy, 10(3):48--52.

Digital Library

[15]

Paul E. Black, "Counting bugs is harder than you think," in 11th IEEE Int'l Working conference on source Code Analysis and Manipulation (SCAM2011), Williamsburg, VA, September 2011.

Digital Library

[16]

"CAS static analysis tool study methodology," Center for Assured Software, NSA, 2011, http://samate.nist.gov/docs/CAS_2011_SA_Tool_Method.pdf

[17]

Common Vulnerabilities and Exposures (CVE). The MITRE corporation, http://cve.mitre.org

[18]

"Juliet test suite v1.2 user guide," Appendix A, Center for Assured Software, NSA, December 2012, http://samate.nist.gov/SARD/testsuite.php

[19]

Gerard J. Holzmann, "Conquering complexity," Computer 40 (12): 111--113, Dec. 2007.

Digital Library

[20]

2011 CWE/SANS Top 25 Most Dangerous Software Errors, http://cwe.mitre.org/top25/

Cited By

Flynn LSnavely WSvoboda DVanHoudnos NQin RBurns JZubrow DStoddard RMarce-Santurio GSentilles SBoehm BTrubiani CFranch XKoziolek A(2018)Prioritizing alerts from multiple static analysis tools, using classification modelsProceedings of the 1st International Workshop on Software Qualities and Their Dependencies10.1145/3194095.3194100(13-20)Online publication date: 28-May-2018
https://dl.acm.org/doi/10.1145/3194095.3194100
Reynolds ZJayanth AKoc UPorter ARaje RHill JShukla RSen SBishop JBreitman K(2017)Identifying and documenting false positive patterns generated by static code analysis toolsProceedings of the 4th International Workshop on Software Engineering Research and Industrial Practice10.1109/SER-IP.2017..20(55-61)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.1109/SER-IP.2017..20
Pewny JHolz TSchwab SRobertson WBalzarotti D(2016)EvilCoderProceedings of the 32nd Annual Conference on Computer Security Applications10.1145/2991079.2991103(214-225)Online publication date: 5-Dec-2016
https://dl.acm.org/doi/10.1145/2991079.2991103

Index Terms

Evaluating bug finders: test and measurement of static code analyzers

Recommendations

Evaluating Bug Finders -- Test and Measurement of Static Code Analyzers
COUFLESS '15: Proceedings of the 2015 IEEE/ACM 1st International Workshop on Complex faUlts and Failures in LargE Software Systems

Software static analysis is one of many options for finding bugs in software. Like compilers, static analyzers take a program as input. This paper covers tools that examine source code -- without executing it -- and output bug reports. Static analysis ...
Empirical Evaluation of Hunk Metrics as Bug Predictors
IWSM '09 /Mensura '09: Proceedings of the International Conferences on Software Process and Product Measurement

Reducing the number of bugs is a crucial issue during software development and maintenance. Software process and product metrics are good indicators of software complexity. These metrics have been used to build bug predictor models to help developers ...
Find bugs in static bug finders
ICPC '22: Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension

Static bug finders (also known as static code analyzers, e.g., Find-Bugs, SonarQube) have been widely-adopted by developers to find bugs in real-world software projects. They leverage predefined heuristic static analysis rules to scan source code or ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

COUFLESS '15: Proceedings of the First International Workshop on Complex faUlts and Failures in LargE Software Systems

May 2015

87 pages

Program Chairs:
Mark Grechanik
University of Illionois at Chicago
,
Javier Alonso
Universidad de León, Spain
,
Allen P. Nikora
Jet Propulsory Laboratory

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\DATC: IEEE Computer Society
TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

IEEE Press

Publication History

Published: 16 May 2015

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '15

Sponsor:

ACM
SIGSOFT
IEEE-CS\DATC
TCSE

ICSE '15: 37th International Conference on Software Engineering

May 16 - 24, 2015

Florence, Italy

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
136
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Flynn LSnavely WSvoboda DVanHoudnos NQin RBurns JZubrow DStoddard RMarce-Santurio GSentilles SBoehm BTrubiani CFranch XKoziolek A(2018)Prioritizing alerts from multiple static analysis tools, using classification modelsProceedings of the 1st International Workshop on Software Qualities and Their Dependencies10.1145/3194095.3194100(13-20)Online publication date: 28-May-2018
https://dl.acm.org/doi/10.1145/3194095.3194100
Reynolds ZJayanth AKoc UPorter ARaje RHill JShukla RSen SBishop JBreitman K(2017)Identifying and documenting false positive patterns generated by static code analysis toolsProceedings of the 4th International Workshop on Software Engineering Research and Industrial Practice10.1109/SER-IP.2017..20(55-61)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.1109/SER-IP.2017..20
Pewny JHolz TSchwab SRobertson WBalzarotti D(2016)EvilCoderProceedings of the 32nd Annual Conference on Computer Security Applications10.1145/2991079.2991103(214-225)Online publication date: 5-Dec-2016
https://dl.acm.org/doi/10.1145/2991079.2991103

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents