More Web Proxy on the site http://driver.im/

research-article

Open access

RustSmith: Random Differential Compiler Testing for Rust

Authors:

Alastair F. DonaldsonAuthors Info & Claims

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 1483 - 1486

https://doi.org/10.1145/3597926.3604919

Published: 13 July 2023 Publication History

Abstract

We present RustSmith, the first Rust randomised program generator for end-to-end testing of Rust compilers. RustSmith generates programs that conform to the advanced type system of Rust, respecting rules related to borrowing and lifetimes, and that are guaranteed to yield a well-defined result. This makes RustSmith suitable for differential testing between compilers or across optimisation levels. By applying RustSmith to a series of versions of the official Rust compiler, rustc, we show that it can detect insidious historical bugs that evaded detection for some time. We have also used RustSmith to find previously-unknown bugs in an alternative Rust compiler implementation, mrustc. In a controlled experiment, we assess statement and mutation coverage achieved by RustSmith vs. the rustc optimisation test suite.

References

[1]

[n. d.]. libFuzzer – a library for coverage-guided fuzz testing. https://llvm.org/docs/LibFuzzer.html

[2]

[n. d.]. Mutabah’s Rust Compiler. https://github.com/thepowersgang/mrustc

[3]

[n. d.]. Rust Fuzzing Trophy Case. https://github.com/rust-fuzz/trophy-case

[4]

[n. d.]. The const propagator cannot trace references. https://github.com/rust-lang/rust/pull/73613

[5]

Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A survey of compiler testing. Comput. Surveys, 53, 1 (2020), 1–36.

[6]

T.Y. Chen, S.C. Cheung, and S.M. Yiu. 1998. Metamorphic Testing: a New Approach for Generating Next Test Cases. Department of Computer Science, The Hong Kong University of Science and Technology.

[7]

Kyle Dewey, Jared Roesch, and Ben Hardekopf. 2015. Fuzzing the Rust typechecker using CLP. In ASE 2015. 482–493.

Digital Library

[8]

Alastair F Donaldson, Hugues Evrard, Andrei Lascu, and Paul Thomson. 2017. Automated testing of graphics shader compilers. Proceedings of the ACM on Programming Languages, 1, OOPSLA (2017), 1–29.

Digital Library

[9]

Andrea Fioraldi, Dominik Maier, Heiko Eiß feldt, and Marc Heuse. 2020. AFL++ combining incremental steps of fuzzing research. In Proceedings of the 14th USENIX Conference on Offensive Technologies. 10–10.

[10]

Yann Herklotz and John Wickerson. 2020. Finding and Understanding Bugs in FPGA Synthesis Tools. In FPGA 2020. ACM, 277–287. https://doi.org/10.1145/3373087.3375310

Digital Library

[11]

Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler Validation via Equivalence modulo Inputs. In PLDI 2014. ACM, 216–226. isbn:9781450327848 https://doi.org/10.1145/2594291.2594334

Digital Library

[12]

Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F Donaldson. 2015. Many-core compiler fuzzing. In PLDI 2015. ACM.

Digital Library

[13]

Donough Liu. 2022. Infinite recursion optimized away with opt-level=z. https://github.com/rust-lang/rust/issues/97428

[14]

Nuno P Lopes, Juneyoung Lee, Chung-Kil Hur, Zhengyang Liu, and John Regehr. 2021. Alive2: bounded translation validation for LLVM. In PLDI 2021. 65–79.

Digital Library

[15]

Michaël Marcozzi, Qiyi Tang, Alastair F Donaldson, and Cristian Cadar. 2019. Compiler fuzzing: How much does it matter? Proceedings of the ACM on Programming Languages, 3, OOPSLA (2019), 1–29.

Digital Library

[16]

William M McKeeman. 1998. Differential testing for software. Digital Technical Journal, 10, 1 (1998), 100–107.

[17]

Martin Pool. 2023. cargo-mutants GitHub repository. https://github.com/sourcefrog/cargo-mutants

[18]

John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. 2012. Test-case reduction for C compiler bugs. In PLDI 2012. 335–346.

Digital Library

[19]

Mayank Sharma. 2022. Bug thrown with nested return. https://github.com/thepowersgang/mrustc/issues/275

[20]

Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and Understanding Bugs in C Compilers. In PLDI 2011. ACM, 283–294. isbn:9781450306638 https://doi.org/10.1145/1993498.1993532

Digital Library

Cited By

Yang WGao CLiu XLi YXue YFilkov VRay BZhou M(2024)Rust-twins: Automatic Rust Compiler Testing through Program Mutation and Dual Macros GenerationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695059(631-642)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695059
Wang QJung R(2024)Rustlantis: Randomized Differential Testing of the Rust CompilerProceedings of the ACM on Programming Languages10.1145/36897808:OOPSLA2(1955-1981)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689780
Suo CChen JLiu SJiang JZhao YWang JChristakis MPradel M(2024)Fuzzing MLIR Compiler Infrastructure via Operation Dependency AnalysisProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680360(1287-1299)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680360
Show More Cited By

Index Terms

RustSmith: Random Differential Compiler Testing for Rust
1. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Maintaining software
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Compilers

Recommendations

Differential testing: a new approach to change detection
ESEC-FSE companion '07: The 6th Joint Meeting on European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering: companion papers

Regression testing, as it's commonly practiced, is unsound due to inconsistent test repair and test addition. This paper presents a new technique, differential testing, that alleviates the test repair problem and detects more changes than regression ...
Rustlantis: Randomized Differential Testing of the Rust Compiler

Compilers are at the core of all computer architecture. Their middle-end and back-end are full of subtle code that is easy to get wrong. At the same time, the consequences of compiler bugs can be severe. Therefore, it is important that we develop ...
Differential testing: a new approach to change detection
ESEC-FSE '07: Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering

Regression testing, as it's commonly practiced, is unsound due to inconsistent test repair and test addition. This paper presents a new technique, differential testing, that alleviates the test repair problem and detects more changes than regression ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2023

1554 pages

ISBN:9798400702211

DOI:10.1145/3597926

General Chair:
René Just
University of Washington, USA
,
Program Chair:
Gordon Fraser
University of Passau, Germany

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution 4.0 International License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '23

Sponsor:

SIGSOFT

ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 17 - 21, 2023

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
710
Total Downloads

Downloads (Last 12 months)482
Downloads (Last 6 weeks)49

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang WGao CLiu XLi YXue YFilkov VRay BZhou M(2024)Rust-twins: Automatic Rust Compiler Testing through Program Mutation and Dual Macros GenerationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695059(631-642)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695059
Wang QJung R(2024)Rustlantis: Randomized Differential Testing of the Rust CompilerProceedings of the ACM on Programming Languages10.1145/36897808:OOPSLA2(1955-1981)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689780
Suo CChen JLiu SJiang JZhao YWang JChristakis MPradel M(2024)Fuzzing MLIR Compiler Infrastructure via Operation Dependency AnalysisProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680360(1287-1299)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680360
Donaldson ASheth DTristan JUsher A(2024)Randomised Testing of the Compiler for a Verification-Aware Programming Language2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00044(407-418)Online publication date: 27-May-2024
https://doi.org/10.1109/ICST60714.2024.00044

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents