More Web Proxy on the site http://driver.im/

research-article

Can we use software bug reports to identify vulnerability discovery strategies?

Authors:

Farzana Ahamed Bhuiyan,

Akond RahmanAuthors Info & Claims

HotSoS '20: Proceedings of the 7th Symposium on Hot Topics in the Science of Security

Article No.: 7, Pages 1 - 10

https://doi.org/10.1145/3384217.3385618

Published: 21 September 2020 Publication History

Abstract

Daily horror stories related to software vulnerabilities necessitates the understanding of how vulnerabilities are discovered. Identification of data sources that can be leveraged to understand how vulnerabilities are discovered could aid cybersecurity researchers to characterize exploitation of vulnerabilities. The goal of the paper is to help cybersecurity researchers in characterizing vulnerabilities by conducting an empirical study of software bug reports. We apply qualitative analysis on 729, 908, and 5336 open source software (OSS) bug reports respectively, collected from Gentoo, LibreOffice, and Mozilla to investigate if bug reports include vulnerability discovery strategies i.e. sequences of computation and/or cognitive activities that an attacker performs to discover vulnerabilities, where the vulnerability is indexed by a credible source, such as the National Vulnerability Database (NVD). We evaluate two approaches namely, text feature-based approach and regular expression-based approach to automatically identify bug reports that include vulnerability discovery strategies.

We observe the Gentoo, LibreOffice, and Mozilla bug reports to include vulnerability discovery strategies. Using text feature-based prediction models, we observe the highest prediction performance for the Mozilla dataset with a recall of 0.78. Using the regular expression-based approach we observe recall to be 0.83 for the same dataset. Findings from our paper provide the groundwork for cybersecurity researchers to use OSS bug reports as a data source for advancing the science of vulnerabilities.

References

[1]

N. Ali, A. Sabané, Y. Guéhéneuc, and G. Antoniol. 2012. Improving Bug Location Using Binary Class Relationships. In 2012 IEEE 12th International Working Conference on Source Code Analysis and Manipulation. 174--183.

Digital Library

[2]

Leo Breiman. 2001. Random Forests. Machine Learning 45, 1 (2001), 5--32.

Digital Library

[3]

Leo Breiman et al. 1984. Classification and Regression Trees (1st ed.). Chapman & Hall, New York. 358 pages. http://www.crcpress.com/catalog/C4841.htm

[4]

Bugzilla. 2017. Web content can open local files by hooking drag and drop to outside of content. https://bugzilla.mozilla.org/show_bug.cgi?id=1379842. [Online; accessed 21-December-2019].

[5]

Oscar Chaparro, Carlos Bernal-Cardenas, Jing Lu, Kevin Moran, Andrian Marcus, Massimiliano Di Penta, Denys Poshyvanyk, and Vincent Ng. 2019. Assessing the Quality of the Steps to Reproduce in Bug Reports. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019). ACM, New York, NY, USA, 86--96.

Digital Library

[6]

O. Chaparro, J. M. Florez, and A. Marcus. 2016. On the Vocabulary Agreement in Software Issue Descriptions. In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME). 448--452.

[7]

Oscar Chaparro, Juan Manuel Florez, and Andrian Marcus. 2019. Using bug descriptions to reformulate queries during text-retrieval-based bug localization. Empirical Software Engineering 24, 5 (01 Oct 2019), 2947--3007.

[8]

Oscar Chaparro, Jing Lu, Fiorella Zampetti, Laura Moreno, Massimiliano Di Penta, Andrian Marcus, Gabriele Bavota, and Vincent Ng. 2017. Detecting Missing Information in Bug Descriptions. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017). ACM, New York, NY, USA, 396--407.

Digital Library

[9]

Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 (1960), 37--46. arXiv:http://dx.doi.org/10.1177/001316446002000104

[10]

Fabrice Colas and Pavel Brazdil. 2006. Comparison of SVM and Some Older Classification Algorithms in Text Classification Tasks. In Artificial Intelligence in Theory and Practice, Max Bramer (Ed.). Springer US, Boston, MA, 169--178.

[11]

National Vulnerability Database. 2019. NVD-Vulnerabiliites. https://nvd.nist.gov/vuln. [Online; accessed 22-August-2019].

[12]

Tezcan Dilshener, Michel Wermelinger, and Yijun Yu. 2016. Locating Bugs Without Looking Back. In Proceedings of the 13th International Conference on Mining Software Repositories (MSR '16). ACM, New York, NY, USA, 286--290.

Digital Library

[13]

PC Doctor Inc. 2019. PC Diagnostic & System Information Solutions Pre-installed on PC/Android Systems. https://www.pc-doctor.com/solutions/oems. [Online; accessed 14-Nov-2019].

[14]

DZone. 2019. Millions of Dell PCs Vulnerable to Flaw in Third-Party Component. https://threatpost.com/millions-of-dell-pcs-vulnerable-to-flaw-in-third-party-component/145833/. [Online; accessed 08-Aug-2019].

[15]

Mona Erfani Joorabchi, Mehdi Mirzaaghaei, and Ali Mesbah. 2014. Works for Me! Characterizing Non-reproducible Bug Reports. In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR 2014). ACM, New York, NY, USA, 62--71.

Digital Library

[16]

David Freedman. 2005. Statistical Models : Theory and Practice. Cambridge University Press.

[17]

Wei Fu, Tim Menzies, and Xipeng Shen. 2016. Tuning for software analytics: Is it really necessary? Information and Software Technology 76 (2016), 135 -- 146.

Digital Library

[18]

Takafumi Fukushima, Yasutaka Kamei, Shane McIntosh, Kazuhiro Yamashita, and Naoyasu Ubayashi. 2014. An Empirical Study of Just-in-time Defect Prediction Using Cross-project Models. In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR 2014). ACM, New York, NY, USA, 172--181.

Digital Library

[19]

Baljinder Ghotra, Shane McIntosh, and Ahmed E. Hassan. 2015. Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models. In Proceedings of the 37th International Conference on Software Engineering - Volume 1 (ICSE '15). IEEE Press, Piscataway, NJ, USA, 789--800. http://dl.acm.org/citation.cfm?id=2818754.2818850

Digital Library

[20]

T. Hall, S. Beecham, D. Bowes, D. Gray, and S. Counsell. 2012. A Systematic Literature Review on Fault Prediction Performance in Software Engineering. IEEE Transactions on Software Engineering 38, 6 (Nov 2012), 1276--1304.

Digital Library

[21]

Daniel Jurafsky and James H. Martin. 2009. Speech and Language Processing (2Nd Edition). Prentice-Hall, Inc., Upper Saddle River, NJ, USA.

Digital Library

[22]

J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159--174. http://www.jstor.org/stable/2529310

[23]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature 521, 7553 (2015), 436--444.

[24]

David D Lewis and William A Gale. 1994. A sequential algorithm for training text classifiers. In SIGIR'94. Springer, 3--12.

[25]

Tim Menzies, Alex Dekhtyar, Justin Distefano, and Jeremy Greenwald. 2007. Problems with Precision: A Response to "Comments on 'Data Mining Static Code Attributes to Learn Defect Predictors'". IEEE Trans. Softw. Eng. 33, 9 (Sept. 2007), 637--640.

Digital Library

[26]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 3111--3119. http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf

Digital Library

[27]

Vikramjit Mitra, Chia-Jiu Wang, and Satarupa Banerjee. 2007. Text Classification: A Least Square Support Vector Machine Approach. Appl. Soft Comput. 7, 3 (June 2007), 908--914.

Digital Library

[28]

N. Munaiah, A. Rahman, J. Pelletier, L. Williams, and A. Meneely. 2019. Characterizing Attacker Behavior in a Cybersecurity Penetration Testing Competition. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1--6.

[29]

Department of Homeland Security. 2019. Information Technology Sector. https://www.dhs.gov/cisa/information-technology-sector. [Online; accessed 15-Nov-2019].

[30]

Science of Security and Privacy. 2010. Cyber Security - Is Science Possible? https://cps-vo.org/node/624. [Online; accessed 24-December-2019].

[31]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12 (Nov. 2011), 2825--2830. http://dl.acm.org/citation.cfm?id=1953048.2078195

Digital Library

[32]

F. Peters, T. T. Tun, Y. Yu, and B. Nuseibeh. 2019. Text Filtering and Ranking for Security Bug Report Prediction. IEEE Transactions on Software Engineering 45, 6 (June 2019), 615--631.

[33]

Foyzur Rahman and Premkumar Devanbu. 2013. How, and Why, Process Metrics Are Better. In Proceedings of the 2013 International Conference on Software Engineering (ICSE '13). IEEE Press, Piscataway, NJ, USA, 432--441. http://dl.acm.org/citation.cfm?id=2486788.2486846

[34]

Pankaj Setia, Balaji Rajagopalan, Vallabh Sambamurthy, and Roger Calantone. 2012. How peripheral developers contribute to open-source software development. Information Systems Research 23, 1 (2012), 144--163.

Digital Library

[35]

J. D. Strate and P. A. Laplante. 2013. A Literature Review of Research in Software Defect Reporting. IEEE Transactions on Reliability 62, 2 (June 2013), 444--454.

[36]

Pang-Ning Tan, Michael Steinbach, and Vipin Kumar. 2005. Introduction to Data Mining, (First Edition). Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.

Digital Library

[37]

Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2016. Automated Parameter Optimization of Classification Techniques for Defect Prediction Models. In Proceedings of the 38th International Conference on Software Engineering (ICSE '16). ACM, New York, NY, USA, 321--332.

Digital Library

[38]

Hanna M Wallach. 2006. Topic modeling: beyond bag-of-words. In Proceedings of the 23rd international conference on Machine learning. ACM, 977--984.

Digital Library

[39]

Bernard Widrow and Michael A Lehr. 1990. 30 years of adaptive neural networks: perceptron, madaline, and backpropagation. Proc. IEEE 78, 9 (1990), 1415--1442.

[40]

Yu Zhao, Kye Miller, Tingting Yu, Wei Zheng, and Minchao Pu. 2019. Automatically Extracting Bug Reproducing Steps from Android Bug Reports. In Reuse in the Big Data Era, Xin Peng, Apostolos Ampatzoglou, and Tanmay Bhowmik (Eds.). Springer International Publishing, Cham, 100--111.

[41]

Thomas Zimmermann, Nachiappan Nagappan, Philip J. Guo, and Brendan Murphy. 2012. Characterizing and Predicting Which Bugs Get Reopened. In Proceedings of the 34th International Conference on Software Engineering (ICSE '12). IEEE Press, Piscataway, NJ, USA, 1074--1083. http://dl.acm.org/citation.cfm?id=2337223.2337363

Digital Library

[42]

T. Zimmermann, R. Premraj, N. Bettenburg, S. Just, A. Schroter, and C. Weiss. 2010. What Makes a Good Bug Report? IEEE Transactions on Software Engineering 36, 5 (Sep. 2010), 618--643.

Digital Library

Cited By

Wang WLi ZYou FZhao R(2023)Vulnerability Report Analysis and Vulnerability Reproduction for Web ApplicationsDependable Software Engineering. Theories, Tools, and Applications10.1007/978-981-99-8664-4_16(279-297)Online publication date: 15-Dec-2023
https://doi.org/10.1007/978-981-99-8664-4_16
Tsutsui TShiraishi YMorii M(2021)Systemization of Vulnerability Information by Ontology for Impact Analysis2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C)10.1109/QRS-C55045.2021.00167(1126-1134)Online publication date: Dec-2021
https://doi.org/10.1109/QRS-C55045.2021.00167
Zheng WZhang MTang HCai YChen XWu XAbdallah Semasaba A(2021)Automatically Identifying Bug Reports with Tactical Vulnerabilities by Deep Feature Learning2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE52982.2021.00043(333-344)Online publication date: Oct-2021
https://doi.org/10.1109/ISSRE52982.2021.00043

Index Terms

Can we use software bug reports to identify vulnerability discovery strategies?
1. Security and privacy
  1. Software and application security
    1. Software security engineering

Recommendations

Vulnerability discovery strategies used in software projects
ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

Malicious users can exploit undiscovered software vulnerabilities i.e., undiscovered weaknesses in software, to cause serious consequences, such as large-scale data breaches. A systematic approach that synthesizes strategies used by security testers can ...
Automated identification of security issues from commit messages and bug reports
ESEC/FSE 2017: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering

The number of vulnerabilities in open source libraries is increasing rapidly. However, the majority of them do not go through public disclosure. These unidentified vulnerabilities put developers' products at risk of being hacked since they are ...
Software Vulnerability Discovery Techniques: A Survey
MINES '12: Proceedings of the 2012 Fourth International Conference on Multimedia Information Networking and Security

Software vulnerabilities are the root cause of computer security problem. How people can quickly discover vulnerabilities existing in a certain software has always been the focus of information security field. This paper has done research on software ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

HotSoS '20: Proceedings of the 7th Symposium on Hot Topics in the Science of Security

September 2020

189 pages

ISBN:9781450375610

DOI:10.1145/3384217

General Chair:
Perry Alexander
The University of Kansas
,
Program Chairs:
Drew Davidson
The University of Kansas
,
Baek-Young Choi
University of Missouri, Kansas City

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

HotSoS '20

HotSoS '20: Hot Topics in the Science of Security

September 21 - 23, 2020

Kansas, Lawrence

Acceptance Rates

Overall Acceptance Rate 34 of 60 submissions, 57%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
135
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)2

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang WLi ZYou FZhao R(2023)Vulnerability Report Analysis and Vulnerability Reproduction for Web ApplicationsDependable Software Engineering. Theories, Tools, and Applications10.1007/978-981-99-8664-4_16(279-297)Online publication date: 15-Dec-2023
https://doi.org/10.1007/978-981-99-8664-4_16
Tsutsui TShiraishi YMorii M(2021)Systemization of Vulnerability Information by Ontology for Impact Analysis2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C)10.1109/QRS-C55045.2021.00167(1126-1134)Online publication date: Dec-2021
https://doi.org/10.1109/QRS-C55045.2021.00167
Zheng WZhang MTang HCai YChen XWu XAbdallah Semasaba A(2021)Automatically Identifying Bug Reports with Tactical Vulnerabilities by Deep Feature Learning2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE52982.2021.00043(333-344)Online publication date: Oct-2021
https://doi.org/10.1109/ISSRE52982.2021.00043

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents