More Web Proxy on the site http://driver.im/

research-article

Closer to the edge: testing compilers more thoroughly by being less conservative about undefined behaviour

Authors:

Karine Even-Mendoza,

Cristian Cadar,

Alastair F. DonaldsonAuthors Info & Claims

ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

Pages 1219 - 1223

https://doi.org/10.1145/3324884.3418933

Published: 27 January 2021 Publication History

Abstract

Randomised compiler testing techniques require a means of generating programs that are free from undefined behaviour (UB) in order to reliably reveal miscompilation bugs. Existing program generators such as Csmith heavily restrict the form of generated programs in order to achieve UB-freedom. We hypothesise that the idiomatic nature of such programs limits the test coverage they can offer. Our idea is to generate less restricted programs that are still UB-free---programs that get closer to the edge of UB, but that do not quite cross the edge. We present preliminary support for our idea via a prototype tool, CsmithEdge, which uses simple dynamic analysis to determine where Csmith has been too conservative in its use of safe math wrappers that guarantee UB-freedom for arithmetic operations. By eliminating redundant wrappers, CsmithEdge was able to discover two new miscompilation bugs in GCC that could not be found via intensive testing using regular Csmith, and to achieve substantial differences in code coverage on GCC compared with regular Csmith.

References

[1]

Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering (TSE) 41, 5 (2015).

[2]

Csmith Homepage. Date Accessed March 09, 2020. https://embed.cs.utah.edu/csmith/.

[3]

CsmithEdge. Date Accessed July 27, 2020. https://srg.doc.ic.ac.uk/projects/CsmithEdge/.

[4]

Alastair F. Donaldson, Hugues Evrard, Andrei Lascu, and Paul Thomson. 2017. Automated Testing of Graphics Shader Compilers. In Proc. of the ACM on Programming Languages (OOPSLA'17).

Digital Library

[5]

Alastair F. Donaldson, Hugues Evrard, and Paul Thomson. 2020. Putting Randomized Compiler Testing into Production. In Proc. of the 34th European Conference on Object-Oriented Programming (ECOOP'20).

[6]

International Organization for Standardization. 2018. ISO/IEC 9899:2018: Programming Languages---C.

[7]

Frama-C. Date Accessed July 29, 2020. https://frama-c.com.

[8]

GCC Bugzilla. Date Accessed April 28, 2020. Bug 94809. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94809.

[9]

GCC Bugzilla. Date Accessed February 15, 2020. Bug 93744. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93744.\

[10]

GitHub. 2018. Git Repository of Yarpgen. https://github.com/intel/yarpgen.

[11]

GitHub. Date Accessed April 13, 2020. Git Repository of Csmith. https://github.com/csmith-project/csmith.git.

[12]

GitHub. Date Accessed July 13, 2020. Git repository of CsmithEdge. https://github.com/karineek/CsmithEdge.git.

[13]

GitHub. Date Accessed May 15, 2020. Git Repository of gfauto. https://github.com/google/graphicsfuzz.git.

[14]

Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler Validation via Equivalence Modulo Inputs. In Proc. of the Conference on Programing Language Design and Implementation (PLDI'14).

Digital Library

[15]

Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-core compiler fuzzing. In Proc. of the Conference on Programing Language Design and Implementation (PLDI'15).

[16]

Michael Marcozzi, Qiyi Tang, Alastair Donaldson, and Cristian Cadar. 2019. Compiler Fuzzing: How Much Does It Matter?. In Proc. of the ACM on Programming Languages (OOPSLA'19).

Digital Library

[17]

Kazuhiro Nakamura and Nagisa Ishiura. 2016. Random testing of C compilers based on test program generation by equivalence transformation. In 2016 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS).

[18]

Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitry Vyukov. 2012. AddressSanitizer: A Fast Address Sanity Checker. In Proc. of the 2012 USENIX Annual Technical Conference (USENIX ATC'12).

[19]

Xi Wang, Nickolai Zeldovich, Frans Kaashoek, and Armando Solar-Lezama. 2013. Towards Optimization-Safe Systems: Analyzing the Impact of Undefined Behavior. In Proc. of the 24th ACM Symposium on Operating Systems Principles (SOSP'13).

Digital Library

[20]

Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and Understanding Bugs in C Compilers. In Proc. of the Conference on Programing Language Design and Implementation (PLDI'11).

Digital Library

Cited By

Gao FChen HZhou YWang KFilkov VRay BZhou M(2024)Shoot Yourself in the Foot — Efficient Code Causes Inefficiency in Compiler OptimizationsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695548(1846-1857)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695548
Theodoridis TSu Z(2024)Refined Input, Degraded Output: The Counterintuitive World of Compiler BehaviorProceedings of the ACM on Programming Languages10.1145/36564048:PLDI(671-691)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656404
Li STheodoridis TSu Z(2024)Boosting Compiler Testing by Injecting Real-World CodeProceedings of the ACM on Programming Languages10.1145/36563868:PLDI(223-245)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656386
Show More Cited By

Index Terms

Closer to the edge: testing compilers more thoroughly by being less conservative about undefined behaviour
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
  2. Software notations and tools
    1. Compilers

Recommendations

Compiler fuzzing: how much does it matter?

Despite much recent interest in randomised testing (fuzzing) of compilers, the practical impact of fuzzer-found compiler bugs on real-world applications has barely been assessed. We present the first quantitative and qualitative study of the tangible ...
CsmithEdge: more effective compiler testing by handling undefined behaviour less conservatively
Abstract
Compiler fuzzing techniques require a means of generating programs that are free from undefined behaviour (UB) to reliably reveal miscompilation bugs. Existing program generators such as Csmith achieve UB-freedom by heavily restricting the form of ...
GrayC: Greybox Fuzzing of Compilers and Analysers for C
ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

Fuzzing of compilers and code analysers has led to a large number of bugs being found and fixed in widely-used frameworks such as LLVM, GCC and Frama-C. Most such fuzzing techniques have taken a blackbox approach, with compilers and code analysers ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

December 2020

1449 pages

ISBN:9781450367684

DOI:10.1145/3324884

General Chair:
John Grundy,
Program Chairs:
Claire Le Goues,
David Lo

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 January 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

EPSRC

Conference

ASE '20

Sponsor:

ASE '20: 35th IEEE/ACM International Conference on Automated Software Engineering

December 21 - 25, 2020

Virtual Event, Australia

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
144
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)1

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gao FChen HZhou YWang KFilkov VRay BZhou M(2024)Shoot Yourself in the Foot — Efficient Code Causes Inefficiency in Compiler OptimizationsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695548(1846-1857)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695548
Theodoridis TSu Z(2024)Refined Input, Degraded Output: The Counterintuitive World of Compiler BehaviorProceedings of the ACM on Programming Languages10.1145/36564048:PLDI(671-691)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656404
Li STheodoridis TSu Z(2024)Boosting Compiler Testing by Injecting Real-World CodeProceedings of the ACM on Programming Languages10.1145/36563868:PLDI(223-245)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656386
Sotiropoulos TChaliasos SSu Z(2024)API-Driven Program Synthesis for Testing Static Typing ImplementationsProceedings of the ACM on Programming Languages10.1145/36329048:POPL(1850-1881)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3632904
Felici RPozzi LFuria C(2024)HyperPUT: generating synthetic faulty programs to challenge bug-finding toolsEmpirical Software Engineering10.1007/s10664-023-10430-829:2Online publication date: 15-Jan-2024
https://dl.acm.org/doi/10.1007/s10664-023-10430-8
Livinskii VBabokin DRegehr J(2023)Fuzzing Loop Optimizations in Compilers for C++ and Data-Parallel LanguagesProceedings of the ACM on Programming Languages10.1145/35912957:PLDI(1826-1847)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591295
Lecoeur BMohsin HDonaldson A(2023)Program Reconditioning: Avoiding Undefined Behaviour When Finding and Reducing Compiler BugsProceedings of the ACM on Programming Languages10.1145/35912947:PLDI(1801-1825)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591294
Chen JSuo CJiang JChen PLi X(2023)Compiler Test-Program Generation via Memoized Configuration Search2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00172(2035-2047)Online publication date: May-2023
https://doi.org/10.1109/ICSE48619.2023.00172
Chaliasos SSotiropoulos TSpinellis DGervais ALivshits BMitropoulos DJhala RDillig I(2022)Finding typing compiler bugsProceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation10.1145/3519939.3523427(183-198)Online publication date: 9-Jun-2022
https://dl.acm.org/doi/10.1145/3519939.3523427
Even-Mendoza KCadar CDonaldson A(2022)CsmithEdge: more effective compiler testing by handling undefined behaviour less conservativelyEmpirical Software Engineering10.1007/s10664-022-10146-127:6Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10664-022-10146-1

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents