More Web Proxy on the site http://driver.im/

research-article

Are datasets for information retrieval-based bug localization techniques trustworthy?: Impact analysis of bug types on IRBL

Authors:

Eunseok LeeAuthors Info & Claims

Empirical Software Engineering, Volume 26, Issue 3

https://doi.org/10.1007/s10664-021-09946-8

Published: 01 May 2021 Publication History

Abstract

Various evaluation datasets are used to evaluate the performance of information retrieval-based bug localization (IRBL) techniques. To accurately evaluate the IRBL and furthermore improve the performance, it is strongly required to analyze the validity of these datasets in advance. To this end, we surveyed 50 previous studies, collected 41,754 bug reports, and found out critical problems that affect the validity of results of performance evaluation. They are in both the ground truth and the search space. These problems arise from using different bug types without clearly distinguishing them. We divided the bugs into production- and test-related bugs. Based on this distinction, we investigate and analyze the impact of the bug type on IRBL performance evaluation. Approximately 18.6% of the bug reports were linked to non-buggy files as the ground truth. Up to 58.5% of the source files in the search space introduced noise into the localization of a specific bug type. From the experiments, we validated that the average precision changed in approximately 90% of the bug reports linked with an incorrect ground truth; we determined that specifying a suitable search space changed the average precision in at least half of the bug reports. Further, we showed that these problems can alter the relative ranks of the IRBL techniques. Our large-scale analysis demonstrated that a significant amount of noise occurs, which can compromise the evaluation results. An important finding of this study is that it is essential to consider the bug types to improve the accuracy of the performance evaluation.

References

[1]

Ali N, Sabane A, Gueheneuc Y G, Antoniol G (2012) Improving bug location using binary class relationships. In: Proceedings of the international working conference on source code analysis and manipulation (SCAM). IEEE, pp 174–183

[2]

Almhana R, Mkaouer W, Kessentini M, Ouni A (2016) Recommending relevant classes for bug reports using multi-objective search. In: Proceedings of the international conference on automated software engineering (ASE). ACM, pp 286–295

[3]

Beck K (2003) Test-driven development: by example. Addison-Wesley Professional

[4]

Beizer B (2003) Software testing techniques. Dreamtech

[5]

Catolino G, Palomba F, Zaidman A, Ferrucci F (2019) Not all bugs are the same: Understanding, characterizing, and classifying bug types. Journal of Systems and Software (JSS)

[6]

Chaparro O, Florez J M, Marcus A (2017) Using observed behavior to reformulate queries during text retrieval-based bug localization. In: Proceedings of the international conference on software maintenance and evolution (ICSME). IEEE, pp 376–387

[7]

Chaparro O, Florez J M, Marcus A (2019) Using bug descriptions to reformulate queries during text-retrieval-based bug localization. Empirical Software Engineering (ESE), pp 1–61

[8]

Charette RN Why software fails [software failure] Spectrum 2005 42 9 42-49

[9]

Corder G W, Foreman DI (2014) Nonparametric statistics: A step-by-step approach. John Wiley & Sons

[10]

Dallmeier V, Zimmermann T (2007) Extraction of bug localization benchmarks from history. In: Proceedings of the international conference on automated software engineering (ASE). ACM, pp 433–436

[11]

Davies S, Roper M (2013) Bug localisation through diverse sources of information. In: Proceedings of the international symposium on software reliability engineering workshops (ISSREW). IEEE, pp 126–131

[12]

Davies S, Roper M, Wood M (2012) Using bug report similarity to enhance bug localisation. In: Proceedings of the working conference on the reverse engineering (WCRE). IEEE, pp 125–134

[13]

Dillman DA, Smyth JD, Christian LM (2014) Internet, phone, mail, and mixed-mode surveys: the tailored design method. John Wiley & Sons

[14]

Dilshener T, Wermelinger M, Yu Y (2016) Locating bugs without looking back. In: Proceedings of the international conference on mining software repositories (MSR). ACM, pp 286–290

[15]

Dit B, Revelle M, Gethers M, and Poshyvanyk D Feature location in source code: a taxonomy and survey Journal of Software: Evolution and Process 2013 25 1 53-95

[16]

Garnier M, Ferreira I, and Garcia A On the influence of program constructs on bug localization effectiveness J Softw Eng Res Dev 2017 5 1 6

[17]

Garousi V and Küçük B Smells in software test code: A survey of knowledge in industry and academia Journal of Systems and Software (JSS) 2018 138 52-81

[18]

Garousi V, Kucuk B, Felderer M (2018) What we know about smells in software test code. Software

[19]

Grottke M and Trivedi KS A classification of software faults J Reliab Eng Assoc Japan 2005 27 7 425-438

[20]

Grottke M, Trivedi K S (2007) Fighting bugs: Remove, retry, replicate, and rejuvenate. Computer 40(2)

[21]

Grottke M, Nikora A P, Trivedi KS (2010) An empirical investigation of fault types in space mission system software. In: Proceedings of the international conference on dependable systems and networks (DSN). IEEE/IFIP, pp 447–456

[22]

Haiduc S, Bavota G, Marcus A, Oliveto R, De Lucia A, Menzies T (2013) Automatic query reformulations for text retrieval in software engineering. In: Proceedings of the international conference on software engineering (ICSE). IEEE Press, pp 842–851

[23]

Herzig K, Zeller A (2013) The impact of tangled code changes. In: Proceedings of the working conference on mining software repositories (MSR). IEEE, pp 121–130

[24]

Herzig K, Just S, Zeller A (2013) It’s not a bug, it’s a feature: how misclassification impacts bug prediction. In: Proceedings of the international conference on software engineering (ICSE). IEEE Press, pp 392–401

[25]

Hill E, Rao S, Kak A (2012) On the use of stemming for concern location and bug localization in java. In: Proceedings of the international working conference on source code analysis and manipulation (SCAM). IEEE, pp 184–193

[26]

Huo X, Thung F, Li M, Lo D, Shi ST (2019) Deep transfer bug localization. IEEE Transactions on software engineering

[27]

Khatiwada S, Tushev M, and Mahmoud A Just enough semantics: an information theoretic approach for ir-based software bug localization Information and Software Technology (IST) 2018 93 45-57

[28]

Kim D, Zeller A, Tao Y, and Kim S Where should we fix this bug?: A two-phase recommendation model Trans Softw Eng (TSE) 2013 99 1 1

[29]

Kim M, Lee E (2018) Are information retrieval-based bug localization techniques trustworthy?. In: Proceedings of the international conference on software engineering: Companion Proceeedings (ICSE-C), ACM, ICSE ’18, pp 248–249

[30]

Kim M, Lee E (2019) A novel approach to automatic query reformulation for ir-based bug localization. In: Proceedings of the symposium on applied computing (SAC). ACM, pp 1752–1759

[31]

Kim M, Lee E (2020) Manq: Many-objective optimization-based automatic query reduction for ir-based bug localization. Information and Software Technology, pp 106334

[32]

Kochhar P S, Tian Y, Lo D (2014) Potential biases in bug localization: Do they matter?. In: Proceedings of the international conference on automated software engineering (ASE). ACM, pp 803–814

[33]

Kochhar P S, Xia X, Lo D, Li S (2016) Practitioners’ expectations on automated fault localization. In: Proceedings of the international symposium on software testing and analysis (ISSTA). ACM, pp 165–176

[34]

Koyuncu A, Bissyandé TF, Kim D, Liu K, Klein J, Monperrus M, Traon YL (2019) D&c: A divide-and-conquer approach to ir-based bug localization. arXiv:190202703

[35]

Labuschagne A, Inozemtseva L, Holmes R (2017) Measuring the cost of regression testing in practice: a study of java projects using continuous integration. In: Proceedings of the joint meeting on foundations of software engineering (FSE). ACM, pp 821–830

[36]

Lam A N, Nguyen A T, Nguyen H A, Nguyen TN (2015) Combining deep learning with information retrieval to localize buggy files for bug reports (n). In: Proceedings of the international conference on automated software engineering (ASE). IEEE, pp 476–481

[37]

Lam A N, Nguyen A T, Nguyen H A, Nguyen TN (2017) Bug localization with combination of deep learning and information retrieval. In: Proceedings of the international conference on program comprehension (ICPC). IEEE, pp 218–229

[38]

Larson RR Introduction to information retrieval J. Am. Soc. Inf. Sci. Technol. 2010 61 4 852-853

[39]

Lawrie D, Binkley D (2018) On the value of bug reports for retrieval-based bug localization. In: Proceedings of the international conference on software maintenance and evolution (ICSME). IEEE, pp 524–528

[40]

Le T D B, Oentaryo R J, Lo D (2015) Information retrieval and spectrum based bug localization: better together. In: Proceedings of the joint meeting on foundations of software engineering (FSE). ACM, pp 579–590

[41]

Lee J, Kim D, Bissyandé T F, Jung W, Le Traon Y (2018) Bench4bl: Reproducibility study on the performance of ir-based bug localization. In: Proceedings of the international symposium on software testing and analysis (ISSTA), ACM, ISSTA 2018, pp 61–72

[42]

Liang H, Sun L, Wang M, and Yang Y Deep learning with customized abstract syntax tree for bug localization IEEE Access 2019 7 116309-116320

[43]

Lu S, Li Z, Qin F, Tan L, Zhou P, Zhou Y (2005) Bugbench: Benchmarks for evaluating bug detection tools. In: Proceedings of the workshop on the evaluation of software defect detection tools

[44]

Lukins SK, Kraft NA, and Etzkorn LH Bug localization using latent dirichlet allocation Information and Software Technology (IST) 2010 52 9 972-990

[45]

Maalej W, Nabil H (2015) Bug report, feature request, or simply praise? on automatically classifying app reviews. In: Proceedings of the international requirements engineering conference (RE). IEEE, pp 116–125

[46]

Marsavina C, Romano D, Zaidman A (2014 ) Studying fine-grained co-evolution patterns of production and test code. In: Proceedings of the international working conference on source code analysis and manipulation (SCAM). IEEE, pp 195–204

[47]

Meszaros G (2007) xUnit test patterns: Refactoring test code. Pearson Education

[48]

Moreno L, Treadway J J, Marcus A, Shen W (2014) On the use of stack traces to improve text retrieval-based bug localization. In: Proceedings of the international conference on software maintenance and evolution (ICSME). IEEE, pp 151–160

[49]

Moreno L, Bavota G, Haiduc S, Di Penta M, Oliveto R, Russo B, Marcus A (2015) Query-based configuration of text retrieval solutions for software engineering tasks. In: Proceedings of the joint meeting on foundations of software engineering (FSE). ACM, pp 567–578

[50]

Nguyen H A, Nguyen A T, Nguyen TN (2013) Filtering noise in mixed-purpose fixing commits to improve defect prediction and localization. In: Proceedings of the international symposium on software reliability engineering (ISSRE). IEEE, pp 138–147

[51]

Palomba F, Zaidman A (2019) The smell of fear: on the relation between test smells and flaky tests. Empirical Software Engineering (ESE), pp 1–40

[52]

Palomba F, Zaidman A, De Lucia A (2018) Automatic test smell detection using information retrieval techniques. In: Proceedings of the international conference on software maintenance and evolution (ICSME). IEEE, pp 311–322

[53]

Poshyvanyk D, Gueheneuc Y G, Marcus A, Antoniol G, Rajlich V (2007) Feature location using probabilistic ranking of methods based on execution scenarios and information retrieval. Trans Softw Eng (TSE) 33(6)

[54]

Rahman M M, Roy CK (2018) Improving ir-based bug localization with context-aware query reformulation. In: Proceedings of the joint meeting on European software engineering conference and symposium on the foundations of software engineering (ESEC/FSE). ACM, pp 621–632

[55]

Rao S, Medeiros H, Kak A (2013) An incremental update framework for efficient retrieval from software libraries for bug localization. In: Proceedings of the working conference on reverse engineering (WCRE). IEEE, pp 62–71

[56]

Rath M and Mäder P Structured information in bug report descriptions—influence on ir-based bug localization and developers Softw. Qual. J. 2019 27 3 1315-1337

[57]

Saha R K, Lease M, Khurshid S, Perry DE (2013) Improving bug localization using structured information retrieval. In: Proceedings of the international conference on automated software engineering (ASE). IEEE, pp 345–355

[58]

Saha R K, Lawall J, Khurshid S, Perry DE (2014) On the effectiveness of information retrieval based bug localization for c programs. In: Proceedings of the international conference on software maintenance and evolution (ICSME). IEEE, pp 161–170

[59]

Shi Z, Keung J, Bennin KE, and Zhang X Comparing learning to rank techniques in hybrid bug localization Appl. Soft Comput. 2018 62 636-648

[60]

Sisman B, Kak AC (2012) Incorporating version histories in information retrieval based bug localization. In: Proceedings of the working conference on mining software repositories (MSR). IEEE, pp 50–59

[61]

Sisman B, Kak AC (2013) Assisting code search with automatic query reformulation for bug localization. In: Proceedings of the working conference on mining software repositories (MSR). IEEE Press, pp 309–318

[62]

Sisman B, Akbar SA, and Kak AC Exploiting spatial code proximity and order for improved source code retrieval for bug localization J Softw Evol Process 2017 29 1 e1805

[63]

Sun X, Zhou W, Li B, Ni Z, and Lu J Bug localization for version issues with defect patterns IEEE Access 2019 7 18811-18820

[64]

Tan L, Liu C, Li Z, Wang X, Zhou Y, and Zhai C Bug characteristics in open source software Empirical Software Engineering (ESE) 2014 19 6 1665-1705

[65]

Tantithamthavorn C, Teekavanich R, Ihara A, Matsumoto K (2013) Mining a change history to quickly identify bug locations: A case study of the eclipse project. In: Proceedings of the international symposium on software reliability engineering workshops (ISSREW). IEEE, pp 108–113

[66]

Tantithamthavorn C, Abebe SL, Hassan AE, Ihara A, and Matsumoto K The impact of ir-based classifier configuration on the performance and the effort of method-level bug localization Information and Software Technology (IST) 2018 102 160-174

[67]

Thomas SW, Nagappan M, Blostein D, and Hassan AE The impact of classifier configuration and classifier combination on bug localization Trans Softw Eng (TSE) 2013 39 10 1427-1443

[68]

Vahabzadeh A, Fard A M, Mesbah A (2015) An empirical study of bugs in test code. In: Proceedings of the international conference on software maintenance and evolution (ICSME). IEEE, pp 101–110

[69]

Vahabzadeh Sefiddarbon A (2016) A study of bugs in test code and a test model for analyzing tests. PhD thesis, University of British Columbia

[70]

Wang B, Xu L, Yan M, Liu C, Liu L (2020a) Multi-dimension convolutional neural network for bug localization. IEEE Transactions on Services Computing

[71]

Wang S, Lo D (2014) Version history, similar report, and structure: Putting them together for improved bug localization. In: Proceedings of the international conference on program comprehension (ICPC). ACM, pp 53–63

[72]

Wang S and Lo D Amalgam+: Composing rich information sources for accurate bug localization J Softw Evol Process 2016 28 10 921-942

[73]

Wang S, Lo D, Lawall J (2014) Compositional vector space models for improved bug localization. In: Proceedings of the international conference on software maintenance and evolution (ICSME). IEEE, pp 171–180

[74]

Wang Y, Yao Y, Tong H, Huo X, Li M, Xu F, Lu J (2020b) Enhancing supervised bug localization with metadata and stack-trace. Knowledge and Information Systems, pp 1–24

[75]

Wen M, Wu R, Cheung S C (2016) Locus: Locating bugs from software changes. In: Proceedings of the international conference on automated software engineering (ASE). IEEE, pp 262–273

[76]

Wong C P, Xiong Y, Zhang H, Hao D, Zhang L, Mei H (2014) Boosting bug-report-oriented fault localization with segmentation and stack-trace analysis. In: Proceedings of the international conference on software maintenance and evolution (ICSME). IEEE, pp 181–190

[77]

Xiao Y, Keung J, Bennin KE, and Mi Q Machine translation-based bug localization technique for bridging lexical gap Information and Software Technology (IST) 2018 99 58-61

[78]

Xiao Y, Keung J, Bennin KE, and Mi Q Improving bug localization with word embedding and enhanced convolutional neural networks Information and Software Technology (IST) 2019 105 17-29

[79]

Yang G, Min K, Lee B (2020) Applying deep learning algorithm to automatic bug localization and repair. In: Proceedings of the 35th Annual ACM symposium on applied computing, pp 1634–1641

[80]

Ye X, Bunescu R, Liu C (2014) Learning to rank relevant files for bug reports using domain knowledge. In: Proceedings of the international symposium on foundations of software engineering (FSE). ACM, pp 689–699

[81]

Ye X, Bunescu R, and Liu C Mapping bug reports to relevant files: A ranking model, a fine-grained benchmark, and feature evaluation Trans Softw Eng (TSE) 2016 42 4 379-402

[82]

Ye X, Shen H, Ma X, Bunescu R, Liu C (2016b) From word embeddings to document similarities for improved information retrieval in software engineering. In: Proceedings of the international conference on software engineering (ICSE). ACM, pp 404–415

[83]

Youm K C, Ahn J, Kim J, Lee E (2015) Bug localization based on code change histories and bug reports. In: Proceedings of the Asia-Pacific software engineering conference (APSEC). IEEE, pp 190–197

[84]

Youm KC, Ahn J, and Lee E Improved bug localization based on code change histories and bug reports Information and Software Technology (IST) 2017 82 177-192

[85]

Zhang W, Li Z, Wang Q, Li J (2019) Finelocator: a novel approach to method-level fine-grained bug localization by query expansion. Information and Software Technology (IST)

[86]

Zhao F, Tang Y, Yang Y, Lu H, Zhou Y, Xu B (2015) Is learning-to-rank cost-effective in recommending relevant files for bug localization?. In: Proceedings of the international conference on software quality, reliability and security (QRS). IEEE, pp 298–303

[87]

Zhou J, Zhang H, Lo D (2012) Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports. In: Proceedings of the international conference on software engineering (ICSE). IEEE, pp 14–24

[88]

Zimmermann T, Premraj R, Bettenburg N, Just S, Schroter A, and Weiss C What makes a good bug report? Trans Softw Eng (TSE) 2010 36 5 618-643

Cited By

Zhang THan DVinayakarao VIrsan IXu BThung FLo DJiang L(2023)Duplicate Bug Report Detection: How Far Are We?ACM Transactions on Software Engineering and Methodology10.1145/357604232:4(1-32)Online publication date: 27-May-2023
https://dl.acm.org/doi/10.1145/3576042
Hirsch THofer BBissyandé TKlein JBird CSarro F(2023)The MAP Metric in Information Retrieval Fault LocalizationProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE56229.2023.00041(1480-1491)Online publication date: 11-Nov-2023
https://dl.acm.org/doi/10.1109/ASE56229.2023.00041
Campos DMartins LMachado I(2022)An empirical study on the influence of developers’ experience on software test code qualityProceedings of the XXI Brazilian Symposium on Software Quality10.1145/3571473.3571481(1-10)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3571473.3571481
Show More Cited By

Index Terms

Are datasets for information retrieval-based bug localization techniques trustworthy?: Impact analysis of bug types on IRBL

Index terms have been assigned to the content through auto-classification.

Recommendations

Are information retrieval-based bug localization techniques trustworthy?
ICSE '18: Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings

Information retrieval-based bug localization techniques are evaluated using datasets with an oracle. However, datasets can contain non-buggy files, which affect the reliability of these techniques. To investigate the impact of non-buggy files, we show ...
How does the first buggy file work well for iterative IR-based bug localization?
SAC '22: Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing

Information-retrieval-based bug localization (IRBL) generates a ranked list of source files by using bug reports as queries and provides the list to developers to reduce debugging costs. The IRBL performance is typically evaluated by batch because the ...
Where should the bugs be fixed? - more accurate information retrieval-based bug localization based on bug reports
ICSE '12: Proceedings of the 34th International Conference on Software Engineering

For a large and evolving software system, the project team could receive a large number of bug reports. Locating the source code files that need to be changed in order to fix the bugs is a challenging task. Once a bug report is received, it is ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 26, Issue 3

May 2021

967 pages

ISSN:1382-3256

Issue’s Table of Contents

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC part of Springer Nature 2021.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 May 2021

Accepted: 22 January 2021

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang THan DVinayakarao VIrsan IXu BThung FLo DJiang L(2023)Duplicate Bug Report Detection: How Far Are We?ACM Transactions on Software Engineering and Methodology10.1145/357604232:4(1-32)Online publication date: 27-May-2023
https://dl.acm.org/doi/10.1145/3576042
Hirsch THofer BBissyandé TKlein JBird CSarro F(2023)The MAP Metric in Information Retrieval Fault LocalizationProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE56229.2023.00041(1480-1491)Online publication date: 11-Nov-2023
https://dl.acm.org/doi/10.1109/ASE56229.2023.00041
Campos DMartins LMachado I(2022)An empirical study on the influence of developers’ experience on software test code qualityProceedings of the XXI Brazilian Symposium on Software Quality10.1145/3571473.3571481(1-10)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3571473.3571481
Nagy NAbdalkareem RLo DMcIntosh SNovielli N(2022)On the co-occurrence of refactoring of test and source codeProceedings of the 19th International Conference on Mining Software Repositories10.1145/3524842.3528529(122-126)Online publication date: 23-May-2022
https://dl.acm.org/doi/10.1145/3524842.3528529
Kim MKim YLee EDwyer M(2022)Deep learning-based production and test bug report classification using source filesProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings10.1145/3510454.3528646(343-344)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510454.3528646
Kim MKim YLee EHong JBures MPark JCerny T(2022)How does the first buggy file work well for iterative IR-based bug localization?Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing10.1145/3477314.3507034(1509-1516)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3477314.3507034
Liang HHang DLi X(2022)Modeling function-level interactions for file-level bug localizationEmpirical Software Engineering10.1007/s10664-022-10237-z27:7Online publication date: 1-Oct-2022
https://dl.acm.org/doi/10.1007/s10664-022-10237-z

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents