More Web Proxy on the site http://driver.im/

research-article

Silent Compiler Bug De-duplication via Three-Dimensional Analysis

Authors:

Jun SunAuthors Info & Claims

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 677 - 689

https://doi.org/10.1145/3597926.3598087

Published: 13 July 2023 Publication History

Abstract

Compiler testing is an important task for assuring the quality of compilers, but investigating test failures is very time-consuming. This is because many test failures are caused by the same compiler bug (known as bug duplication problem). In particular, this problem becomes much more challenging on silent compiler bugs (also called wrong code bugs), since these bugs can provide little information (unlike crash bugs that can produce error messages) for bug de-duplication. In this work, we propose a novel technique (called D³) to solve the duplication problem on silent compiler bugs. Its key insight is to characterize the silent bugs from the testing process and identify three-dimensional information (i.e., test program, optimizations, and test execution) for bug de-duplication. However, there are huge amount of bug-irrelevant details on the three dimensions, D³ then systematically conducts causal analysis to identify bug-causal features from each of the three dimensions for more accurate bug de-duplication. Finally, D³ ranks the test failures that are more likely to be caused by different silent bugs higher by measuring the distance among test failures based on the three-dimensional bug-causal features. Our experimental results on four datasets (including duplicate bugs of both GCC and LLVM) demonstrate the significant superiority of D³ over the two state-of-the-art compiler bug de-duplication techniques, achieving the average improvement of 19.36% and 51.43% in identifying unique silent compiler bugs when analyzing the same number of test failures.

References

[1]

2020. Clang Static Analyzer. https://gcc.gnu.org/onlinedocs/gcc/Gcov.html

[2]

2022. D3 homepage. https://github.com/chenyangyc/D3

[3]

2022. GCC. https://gcc.gnu.org

[4]

2022. LLVM. https://llvm.org

[5]

2022. tree-sitter. https://github.com/tree-sitter/tree-sitter

[6]

Rui Abreu, Peter Zoeteweij, and Arjan JC Van Gemund. 2007. On the accuracy of spectrum-based fault localization. In Testing: Academic and industrial conference practice and research techniques-MUTATION (TAICPART-MUTATION 2007). 89–98.

Digital Library

[7]

Anahita Alipour, Abram Hindle, and Eleni Stroulia. 2013. A contextual approach towards more accurate duplicate bug report detection. In 2013 10th Working Conference on Mining Software Repositories (MSR). 183–192.

[8]

Mohammad Amin Alipour, Alex Groce, Rahul Gopinath, and Arpit Christi. 2016. Generating focused random tests using directed swarm testing. In Proceedings of the 25th International Symposium on Software Testing and Analysis. 70–81.

Digital Library

[9]

Cyrille Artho. 2011. Iterative delta debugging. International Journal on Software Tools for Technology Transfer, 13, 3 (2011), 223–246.

Digital Library

[10]

Scotty Bauer, Cuoq Pascal, and Regehr John. [n. d.]. Deniable Backdoors Using Compiler Bugs. International Journal of PoC||GTFO.

[11]

Junjie Chen, Yanwei Bai, Dan Hao, Yingfei Xiong, Hongyu Zhang, and Bing Xie. 2017. Learning to prioritize test programs for compiler testing. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). 700–711.

Digital Library

[12]

Junjie Chen, Yanwei Bai, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. 2016. Test case prioritization for compilers: A text-vector based approach. In 2016 IEEE international conference on software testing, verification and validation (ICST). 266–277.

[13]

Junjie Chen, Jiaqi Han, Peiyi Sun, Lingming Zhang, Dan Hao, and Lu Zhang. 2019. Compiler bug isolation via effective witness test program generation. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 223–234.

Digital Library

[14]

Junjie Chen, Wenxiang Hu, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. 2016. An empirical comparison of compiler testing techniques. In Proceedings of the 38th International Conference on Software Engineering. 180–190.

Digital Library

[15]

Junjie Chen, Haoyang Ma, and Lingming Zhang. 2020. Enhanced compiler bug isolation via memoized search. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 78–89.

Digital Library

[16]

Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A survey of compiler testing. ACM Computing Surveys (CSUR), 53, 1 (2020), 1–36.

Digital Library

[17]

Junjie Chen and Chenyao Suo. 2022. Boosting Compiler Testing via Compiler Optimization Exploration. ACM Transactions on Software Engineering and Methodology.

[18]

Junjie Chen, Guancheng Wang, Dan Hao, Yingfei Xiong, Hongyu Zhang, and Lu Zhang. 2019. History-guided configuration diversification for compiler test-program generation. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 305–316.

Digital Library

[19]

M.Y. Chen, E. Kiciman, E. Fratkin, A. Fox, and E. Brewer. 2002. Pinpoint: problem determination in large, dynamic Internet services. In Proceedings International Conference on Dependable Systems and Networks. 595–604. https://doi.org/10.1109/DSN.2002.1029005

[20]

Yang Chen, Alex Groce, Chaoqiang Zhang, Weng-Keen Wong, Xiaoli Fern, Eric Eide, and John Regehr. 2013. Taming compiler fuzzers. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation. 197–208.

Digital Library

[21]

Zhichao Chen, Junjie Chen, Weijing Wang, Jianyi Zhou, Meng Wang, Xiang Chen, Shan Zhou, and Jianmin Wang. [n. d.]. Exploring Better Black-Box Test Case Prioritization via Log Analysis. ACM Transactions on Software Engineering and Methodology.

[22]

Alastair F Donaldson, Hugues Evrard, and Paul Thomson. 2020. Putting randomized compiler testing into production (experience report). In 34th European Conference on Object-Oriented Programming (ECOOP 2020).

[23]

Alastair F Donaldson, Paul Thomson, Vasyl Teliman, Stefano Milizia, André Perez Maselco, and Antoni Karpiński. 2021. Test-case reduction and deduplication almost for free with transformation-based compiler testing. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation. 1017–1032.

Digital Library

[24]

Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, and Martin Monperrus. 2014. Fine-grained and accurate source code differencing. In Proceedings of the 29th ACM/IEEE international conference on Automated software engineering. 313–324.

Digital Library

[25]

Teofilo F Gonzalez. 1985. Clustering to minimize the maximum intercluster distance. Theoretical computer science, 38 (1985), 293–306.

[26]

Dan Han, Chenlei Zhang, Xiaochao Fan, Abram Hindle, Kenny Wong, and Eleni Stroulia. 2012. Understanding android fragmentation with topic analysis of vendor-specific bugs. In 2012 19th Working Conference on Reverse Engineering. 83–92.

Digital Library

[27]

Foyzul Hassan and Xiaoyin Wang. 2018. Hirebuild: An automatic approach to history-driven repair of build scripts. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). 1078–1089.

Digital Library

[28]

Josie Holmes and Alex Groce. 2018. Causal distance-metric-based assistance for debugging after compiler fuzzing. In 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE). 166–177.

[29]

Jiajun Jiang, Ran Wang, Yingfei Xiong, Xiangping Chen, and Lu Zhang. 2019. Combining spectrum-based fault localization and statistical debugging: An empirical study. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 502–514.

Digital Library

[30]

James A Jones and Mary Jean Harrold. 2005. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering. 273–282.

Digital Library

[31]

Ajmeera Kiran and D Vasumathi. 2020. Data mining: min–max normalization based data perturbation technique for privacy preservation. In Proceedings of the Third International Conference on Computational Intelligence and Informatics. 723–734.

[32]

Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. ACM Sigplan Notices, 49, 6 (2014), 216–226.

Digital Library

[33]

Suyoung Lee, HyungSeok Han, Sang Kil Cha, and Sooel Son. 2020. Montage: A Neural Network Language Model-Guided JavaScript Engine Fuzzer. In 29th USENIX Security Symposium (USENIX Security 20). 2613–2630.

[34]

J. Lerch and M. Mezini. 2013. Finding Duplicates of Your Yet Unwritten Bug Report. In Software Maintenance and Reengineering (CSMR), 2013 17th European Conference on.

[35]

Zheng Li, Mark Harman, and Robert M Hierons. 2007. Search algorithms for regression test case prioritization. IEEE Transactions on software engineering, 33, 4 (2007), 225–237.

Digital Library

[36]

Meng-Jie Lin, Cheng-Zen Yang, Chao-Yuan Lee, and Chun-Chang Chen. 2016. Enhancements for duplication detection in bug reports with manifold correlation features. Journal of Systems and Software, 121 (2016), 223–233.

Digital Library

[37]

Fernando Lourenco, Victor Lobo, and Fernando Bacao. 2004. Binary-based similarity measures for categorical data and their application in Self-Organizing Maps.

[38]

Ghassan Misherghi and Zhendong Su. 2006. HDD: hierarchical delta debugging. In Proceedings of the 28th international conference on Software engineering. 142–151.

Digital Library

[39]

Lee Naish, Hua Jie Lee, and Kotagiri Ramamohanarao. 2011. A Model for Spectra-Based Software Diagnosis. ACM Trans. Softw. Eng. Methodol., 20, 3 (2011), 32 pages. https://doi.org/10.1145/2000791.2000795

Digital Library

[40]

A. T. Nguyen, T. T. Nguyen, T. N. Nguyen, D. Lo, and C. Sun. 2012. Duplicate bug report detection with a combination of information retrieval and topic modeling.

[41]

José R Pérez-Agüera, Javier Arroyo, Jane Greenberg, Joaquin Perez Iglesias, and Victor Fresno. 2010. Using BM25F for semantic search. In Proceedings of the 3rd international semantic search workshop. 1–8.

Digital Library

[42]

John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. 2012. Test-case reduction for C compiler bugs. In Proceedings of the 33rd ACM SIGPLAN conference on Programming Language Design and Implementation. 335–346.

Digital Library

[43]

Gregg Rothermel, Roland H Untch, Chengyun Chu, and Mary Jean Harrold. 1999. Test case prioritization: An empirical study. In Proceedings IEEE International Conference on Software Maintenance-1999 (ICSM’99).’Software Maintenance for Business Change’(Cat. No. 99CB36360). 179–188.

[44]

Gregg Rothermel, Roland H. Untch, Chengyun Chu, and Mary Jean Harrold. 2001. Prioritizing test cases for regression testing. IEEE Transactions on software engineering, 27, 10 (2001), 929–948.

Digital Library

[45]

Hanan Samet. 1977. Toward automatic debugging of compilers. In Proceedings of the 5th international joint conference on Artificial intelligence-Volume 1. 379–379.

Digital Library

[46]

Jeongju Sohn and Shin Yoo. 2017. Fluccs: Using code and change metrics to improve fault localization. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. 273–283.

Digital Library

[47]

Chengnian Sun, Vu Le, Qirun Zhang, and Zhendong Su. 2016. Toward understanding compiler bugs in GCC and LLVM. In Proceedings of the 25th International Symposium on Software Testing and Analysis. 294–305.

Digital Library

[48]

Chengnian Sun, David Lo, Siau-Cheng Khoo, and Jing Jiang. 2011. Towards more accurate retrieval of duplicate bug reports. In 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011). 253–262.

Digital Library

[49]

C. Sun, D. Lo, X. Wang, J. Jing, and S. C. Khoo. 2010. A discriminative model approach for accurate duplicate bug report retrieval. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1, ICSE 2010, Cape Town, South Africa, 1-8 May 2010.

[50]

Xiaoyin Wang, Lu Zhang, Tao Xie, John Anvik, and Jiasu Sun. 2008. An approach to detecting duplicate bug reports using natural language and execution information. In 30th International Conference on Software Engineering (ICSE 2008), Leipzig, Germany, May 10-18, 2008.

Digital Library

[51]

Ming Wen, Junjie Chen, Rongxin Wu, Dan Hao, and Shing-Chi Cheung. 2018. Context-aware patch generation for better automated program repair. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). 1–11.

Digital Library

[52]

W Eric Wong, Vidroha Debroy, Ruizhi Gao, and Yihao Li. 2013. The DStar method for effective software fault localization. IEEE Transactions on Reliability, 63, 1 (2013), 290–308.

[53]

Robert F Woolson. 2007. Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials, 1–3.

[54]

Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation. 283–294.

Digital Library

[55]

Jerrold H Zar. 2005. Spearman rank correlation. Encyclopedia of biostatistics, 7 (2005).

[56]

Andreas Zeller. 1999. Yesterday, my program worked. Today, it does not. Why? ACM SIGSOFT Software engineering notes, 24, 6 (1999), 253–267.

[57]

Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and isolating failure-inducing input. IEEE Transactions on Software Engineering, 28, 2 (2002), 183–200.

Digital Library

[58]

Qirun Zhang, Chengnian Sun, and Zhendong Su. 2017. Skeletal program enumeration for rigorous compiler testing. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation. 347–361.

Digital Library

[59]

Jie Zou, Ling Xu, Mengning Yang, Xiaohong Zhang, Jun Zeng, and Sachio Hirokawa. 2016. Automated duplicate bug report detection using multi-factor analysis. IEICE TRANSACTIONS on Information and Systems, 99, 7 (2016), 1762–1775.

Cited By

Liu YZhu MDong JYu JHao DFilkov VRay BZhou M(2024)Compiler Bug Isolation via Enhanced Test Program MutationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695074(819-830)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695074
Yang CChen JJiang JHuang Y(2024)Dependency-Aware Code NaturalnessProceedings of the ACM on Programming Languages10.1145/36897948:OOPSLA2(2355-2377)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689794
Liu SLan JDu XLi JLu WJiang JDu XChristakis MPradel M(2024)Testing Graph Database Systems with Graph-State Persistence OracleProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680311(666-677)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680311

Index Terms

Silent Compiler Bug De-duplication via Three-Dimensional Analysis
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Toward understanding compiler bugs in GCC and LLVM
ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and Analysis

Compilers are critical, widely-used complex software. Bugs in them have significant impact, and can cause serious damage when they silently miscompile a safety-critical application. An in-depth understanding of compiler bugs can help detect and fix ...
A contextual approach towards more accurate duplicate bug report detection and ranking

The issue-tracking systems used by software projects contain issues, bugs, or tickets written by a wide variety of bug reporters, with different levels of training and knowledge about the system under development. Typically, reporters lack the skills ...
Enhanced compiler bug isolation via memoized search
ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

Compiler bugs can be disastrous since they could affect all the software systems built on the buggy compilers. Meanwhile, diagnosing compiler bugs is extremely challenging since usually limited debugging information is available and a large number of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2023

1554 pages

ISBN:9798400702211

DOI:10.1145/3597926

General Chair:
René Just
University of Washington, USA
,
Program Chair:
Gordon Fraser
University of Passau, Germany

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Conference

ISSTA '23

Sponsor:

SIGSOFT

ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 17 - 21, 2023

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
258
Total Downloads

Downloads (Last 12 months)120
Downloads (Last 6 weeks)4

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu YZhu MDong JYu JHao DFilkov VRay BZhou M(2024)Compiler Bug Isolation via Enhanced Test Program MutationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695074(819-830)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695074
Yang CChen JJiang JHuang Y(2024)Dependency-Aware Code NaturalnessProceedings of the ACM on Programming Languages10.1145/36897948:OOPSLA2(2355-2377)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689794
Liu SLan JDu XLi JLu WJiang JDu XChristakis MPradel M(2024)Testing Graph Database Systems with Graph-State Persistence OracleProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680311(666-677)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680311

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents