Understanding and Detecting Annotation-Induced Faults of Static Analyzers
Article No.: 33, Pages 722 - 744
Abstract
Static analyzers can reason about the properties and behaviors of programs and detect various issues without executing them. Hence, they should extract the necessary information to understand the analyzed program well. Annotation has been a widely used feature for different purposes in Java since the introduction of Java 5. Annotations can change program structures and convey semantics information without awareness of static analyzers, consequently leading to imprecise analysis results. This paper presents the first comprehensive study of annotation-induced faults (AIF) by analyzing 246 issues in six open-source and popular static analyzers (i.e., PMD, SpotBugs, CheckStyle, Infer, SonarQube, and Soot). We analyzed the issues' root causes, symptoms, and fix strategies and derived ten findings and some practical guidelines for detecting and repairing annotation-induced faults. Moreover, we developed an automated testing framework called AnnaTester based on three metamorphic relations originating from the findings. AnnaTester generated new tests based on the official test suites of static analyzers and unveiled 43 new faults, 20 of which have been fixed. The results confirm the value of our study and its findings.
References
[1]
adangel. 2019. NoPackage: False Negative for enums. https://github.com/pmd/pmd/issues/1782
[2]
Dima Alhadidi, Amine Boukhtouta, Nadia Belblidia, Mourad Debbabi, and Prabir Bhattacharya. 2009. The dataflow pointcut: a formal and practical framework. In Proceedings of the 8th ACM international conference on Aspect-oriented software development. 15–26.
[3]
Cláudio A Araújo, Marcio E Delamaro, José C Maldonado, and Auri MR Vincenzi. 2016. Correlating automatic static analysis and mutation testing: towards incremental strategies. Journal of Software Engineering Research and Development, 4, 1 (2016), 1–32.
[4]
Nathaniel Ayewah and William Pugh. 2010. The Google FindBugs fixit. 241–252. https://doi.org/10.1145/1831708.1831738
[5]
Nathaniel Ayewah, William Pugh, J David Morgenthaler, John Penix, and YuQian Zhou. 2007. Evaluating static analysis defect warnings on production software. In Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering. 1–8.
[6]
Belle. 2022. A false positive about rule RSPEC-2095. https://community.sonarsource.com/t/a-false-positive-about-rule-rspec-2095/67536
[7]
Belle-PL. 2022. Parse error on array type annotations. https://github.com/pmd/pmd/issues/4152##issuecomment-1277447394
[8]
Cristian Cadar and Alastair F Donaldson. 2016. Analysing the program analyser. In Proceedings of the 38th International Conference on Software Engineering Companion. 765–768.
[9]
G Ann Campbell and Patroklos P Papapetrou. 2013. SonarQube in action. Manning Publications Co.
[10]
Walter Cazzola and Edoardo Vacchi. 2014. @ Java: Bringing a richer annotation model to Java. Computer Languages, Systems & Structures, 40, 1 (2014), 2–18.
[11]
CheckStyle. 2022. Checkstyle. https://checkstyle.sourceforge.io/
[12]
Haicheng Chen, Wensheng Dou, Yanyan Jiang, and Feng Qin. 2019. Understanding exception-related bugs in large-scale cloud systems. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 339–351.
[13]
Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, TH Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys (CSUR), 51, 1 (2018), 1–27.
[14]
Maria Christakis and Christian Bird. 2016. What developers want and need from program analysis: an empirical study. In Proceedings of the 31st IEEE/ACM international conference on automated software engineering. 332–343.
[15]
Pascal Cuoq, Benjamin Monate, Anne Pacalet, Virgile Prevosto, John Regehr, Boris Yakobowski, and Xuejun Yang. 2012. Testing static analyzers with randomly generated programs. In NASA Formal Methods Symposium. 120–125.
[16]
Krysztof Czarnecki and Ulrich Eisenecker. 2000. Generative Programming: Methods, Tools, and Applications. Addison-Wesley, Boston, MA. isbn:978-0-201-30977-5
[17]
Lisa Nguyen Quang Do, James R. Wright, and Karim Ali. 2022. Why Do Software Developers Use Static Analysis Tools? A User-Centered Study of Developer Needs and Motivations. IEEE Transactions on Software Engineering, 48, 3 (2022), 835–847. https://doi.org/10.1109/TSE.2020.3004525
[18]
fluxroot. 2019. False-positive with Lombok and inner classes. https://github.com/pmd/pmd/issues/1641
[19]
Michael Gumowski. 2015. Annotations should be handled in all cases allowed by java 8. https://sonarsource.atlassian.net/browse/SONARJAVA-1167
[20]
Michael Gumowski. 2016. Classes annotated with lombok’s @EqualsAndHashCode should be ignored. https://sonarsource.atlassian.net/browse/SONARJAVA-1513
[21]
Michael Gumowski. 2017. Annotations on type in a fully qualified name are not resolved. https://sonarsource.atlassian.net/browse/SONARJAVA-2205
[22]
Michael Gumowski. 2017. FP on S1128 (UselessImportCheck) when annotation is used in fully qualified name. https://sonarsource.atlassian.net/browse/SONARJAVA-2083
[23]
Michael Gumowski. 2019. Annotations are not always resolved when used in parameterized types. https://sonarsource.atlassian.net/browse/SONARJAVA-3045
[24]
Michael Gumowski. 2019. Generating starting states make analysis crash (OutOfMemory) when too many annotated parameters. https://sonarsource.atlassian.net/browse/SONARJAVA-3108
[25]
A. Habib and M. Pradel. 2018. How Many of All Bugs Do We Find? A Study of Static Bug Detectors. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). 317–328. https://doi.org/10.1145/3238147.3238213
[26]
Infer. 2022. Infer Static Analyzer. https://fbinfer.com/
[27]
Quentin Jaquier. 2019. Support Java 11 Generated annotation. https://sonarsource.atlassian.net/browse/SONARJAVA-3174
[28]
Quentin Jaquier. 2020. Consistently support Nullable/CheckForNull/Nonnull annotations in rules. https://sonarsource.atlassian.net/browse/SONARJAVA-3536
[29]
Quentin Jaquier. 2023. S5122: ClassCastException when annotation is defined with an identifier. https://sonarsource.atlassian.net/browse/SONARJAVA-3438
[30]
Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why don’t software developers use static analysis tools to find bugs? In 2013 35th International Conference on Software Engineering (ICSE). 672–681.
[31]
Josh Juneau. 2022. JSR 308 Explained: Java Type Annotations. https://www.oracle.com/technical-resources/articles/java/ma14-architect-annotations.html
[32]
Anant Kharkar, Roshanak Zilouchian Moghaddam, Matthew Jin, Xiaoyu Liu, Xin Shi, Colin Clement, and Neel Sundaresan. 2022. Learning to reduce false positives in analytic bug detectors. In Proceedings of the 44th International Conference on Software Engineering. 1307–1316.
[33]
Dong Jae Kim, Nikolaos Tsantalis, Tse-Hsun Chen, and Jinqiu Yang. 2021. Studying Test Annotation Maintenance in the Wild. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 62–73. https://doi.org/10.1109/ICSE43902.2021.00019
[34]
Christian Klinger, Maria Christakis, and Valentin Wüstholz. 2019. Differentially testing soundness and precision of program analyzers. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 239–250.
[35]
konrad jamrozik. 2013. Regression: ERROR/dalvikvm(1854): Invalid type descriptor: ’dalvik.annotation.EnclosingClass’. https://github.com/soot-oss/soot/issues/123
[36]
krzyk. 2015. SuppressWarnings should support CamelCase. https://github.com/checkstyle/checkstyle/issues/2202
[37]
Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. ACM Sigplan Notices, 49, 6 (2014), 216–226.
[38]
lgoldstein. 2017. EmptyBlock: NPE on annotation declaration. https://github.com/checkstyle/checkstyle/issues/4472
[39]
Junjie Li and Jinqiu Yang. 2024. Tracking the Evolution of Static Code Warnings: the State-of-the-Art and a Better Approach. IEEE Transactions on Software Engineering.
[40]
Stephan Lipp, Sebastian Banescu, and Alexander Pretschner. 2022. An empirical study on the effectiveness of static C code analyzers for vulnerability detection. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 544–555.
[41]
Dongmiao Liu. 2022. SONARJAVA-73 add more lombok’s used annotations for UnusedPrivateFieldCheck. https://github.com/SonarSource/sonar-java/pull/102##issuecomment-87545890
[42]
Han Liu, Sen Chen, Ruitao Feng, Chengwei Liu, Kaixuan Li, Zhengzi Xu, Liming Nie, Yang Liu, and Yixiang Chen. 2023. A Comprehensive Study on Quality Assurance Tools for Java. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2023). Association for Computing Machinery, New York, NY, USA. 285–297. isbn:9798400702211 https://doi.org/10.1145/3597926.3598056
[43]
Yi Liu, Yadong Yan, Chaofeng Sha, Xin Peng, Bihuan Chen, and Chong Wang. 2022. DeepAnna: Deep Learning based Java Annotation Recommendation and Misuse Detection. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 685–696.
[44]
LynnBroe. 2021. UnnecessaryConstructor: false-positive with @Inject. https://github.com/pmd/pmd/issues/4487
[45]
Muhammad Numair Mansur, Maria Christakis, and Valentin Wüstholz. 2021. Metamorphic testing of Datalog engines. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 639–650.
[46]
marcelmore. 2021. UnusedPrivateMethod violation for disabled class in 6.23. https://github.com/pmd/pmd/issues/2454
[47]
martin. 2020. UnusedPrivateField cannot override ignored annotations property. https://github.com/pmd/pmd/issues/2876
[48]
mjustin. 2020. Exception when using SuppressWarningsHolder with @SuppressWarnings as an annotation property. https://github.com/checkstyle/checkstyle/issues/7522
[49]
Austin Mordahl, Zenong Zhang, Dakota Soles, and Shiyi Wei. 2023. ECSTATIC: An Extensible Framework for Testing and Debugging Configurable Static Analysis. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 550–562. https://doi.org/10.1109/ICSE48619.2023.00056
[50]
msridhar. 2017. Eradicate not reading annotations from class file. https://github.com/facebook/infer/issues/559
[51]
Marharyta Nedzelska. 2021. FP in S3077 when volatile is used with @Immutable and @ThreadSafe annotations. https://sonarsource.atlassian.net/browse/SONARJAVA-3804
[52]
nrmancuso. 2020. Compact Constructor AST is missing annotations. https://github.com/checkstyle/checkstyle/issues/8734
[53]
Batyr Nuryyev, Ajay Kumar Jha, Sarah Nadi, Yee-Kang Chang, Emily Jiang, and Vijay Sundaresan. 2022. Mining Annotation Usage Rules: A Case Study with MicroProfile. In 2022 38th International Conference on Software Maintenance and Evolution, IEEE.
[54]
Batyr Nuryyev, Ajay Kumar Jha, Sarah Nadi, Yee-Kang Chang, Emily Jiang, and Vijay Sundaresan. 2022. Mining Annotation Usage Rules: A Case Study with MicroProfile. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). 553–562.
[55]
Fernando Rodriguez Olivera. 2017. FindBugs JSR305. https://mvnrepository.com/artifact/com.google.code.findbugs/jsr305
[56]
Fernando Rodriguez Olivera. 2023. MvnRepository. https://mvnrepository.com/
[57]
oowekyala. 2022. How PMD Works. https://docs.pmd-code.org/pmd-doc-6.53.0/pmd_devdocs_how_pmd_works.html
[58]
Chris Parnin, Christian Bird, and Emerson Murphy-Hill. 2013. Adoption and use of Java generics. Empirical Software Engineering, 18, 6 (2013), 1047–1089.
[59]
Nicolas Peru. 2015. Annotation on array type should be properly handled. https://sonarsource.atlassian.net/browse/SONARJAVA-1420
[60]
Pedro Pinheiro, José Carlos Viana, Márcio Ribeiro, Leo Fernandes, Fabiano Ferrari, Rohit Gheyi, and Baldoino Fonseca. 2020. Mutating code annotations: An empirical evaluation on Java and C# programs. Science of Computer Programming, 191 (2020), 102418.
[61]
PMD. 2022. PMD An extensible cross-language static code analyzer. https://pmd.github.io/
[62]
William Pugh. 2022. JSR 305: Annotations for Software Defect Detection. https://jcp.org/en/jsr/detail?id=305
[63]
rnveach. 2016. Java 8 Grammar: annotations on varargs parameters. https://github.com/checkstyle/checkstyle/issues/3238
[64]
rnveach. 2021. AtclauseOrder: Falsely ignores method with annotation. https://github.com/checkstyle/checkstyle/issues/9941
[65]
robtimus. 2021. OperatorWrap with token ASSIGN too strict for annotations. https://github.com/checkstyle/checkstyle/issues/10945
[66]
Henrique Rocha and Marco Tulio Valente. 2011. How Annotations are Used in Java: An Empirical Study. In SEKE. 426–431.
[67]
Qingchao Shen, Haoyang Ma, Junjie Chen, Yongqiang Tian, Shing-Chi Cheung, and Xiang Chen. 2021. A comprehensive study of deep learning compiler bugs. In Proceedings of the 29th ACM Joint meeting on european software engineering conference and symposium on the foundations of software engineering. 968–980.
[68]
SonarQube. 2022. SonarQube Code Quality and Code Security. https://www.sonarqube.org/
[69]
Soot. 2023. Soot - A framework for analyzing and transforming Java and Android applications. http://soot-oss.github.io/soot/
[70]
SpotBugs. 2022. SpotBugs: Find bugs in Java Programs. https://spotbugs.github.io/
[71]
Chengnian Sun, Vu Le, Qirun Zhang, and Zhendong Su. 2016. Toward understanding compiler bugs in GCC and LLVM. In Proceedings of the 25th international symposium on software testing and analysis. 294–305.
[72]
Xiaobing Sun, Tianchi Zhou, Gengjie Li, Jiajun Hu, Hui Yang, and Bin Li. 2017. An empirical study on real bugs for machine learning programs. In 2017 24th Asia-Pacific Software Engineering Conference (APSEC). 348–357.
[73]
Jubi Taneja, Zhengyang Liu, and John Regehr. 2020. Testing static analyses for precision and soundness. In Proceedings of the 18th ACM/IEEE International Symposium on Code Generation and Optimization. 81–93.
[74]
Daniel Tang, Ales Plsek, and Jan Vitek. 2010. Static checking of safety critical Java annotations. In Proceedings of the 8th International Workshop on Java Technologies for Real-Time and Embedded Systems. 148–154.
[75]
Ferdian Thung, Shaowei Wang, David Lo, and Lingxiao Jiang. 2012. An empirical study of bugs in machine learning systems. In 2012 IEEE 23rd International Symposium on Software Reliability Engineering. 271–280.
[76]
David A Tomassi and Cindy Rubio-González. 2021. On the real-world effectiveness of static bug detectors at finding null pointer exceptions. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 292–303.
[77]
Susana M Vieira, Uzay Kaymak, and João MC Sousa. 2010. Cohen’s kappa coefficient as a performance measure for feature selection. In International conference on fuzzy systems. 1–8.
[78]
Jules Villard. 2020. Infer workflow. https://fbinfer.com/docs/infer-workflow
[79]
vovkss. 2018. [java] Processing error (ClassCastException) if a TYPE_USE annotation is used on a base class in the ëxtends" clause. https://github.com/pmd/pmd/issues/1369
[80]
Zhiyuan Wan, David Lo, Xin Xia, and Liang Cai. 2017. Bug characteristics in blockchain systems: a large-scale empirical study. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). 413–424.
[81]
Junjie Wang, Yuchao Huang, Song Wang, and Qing Wang. 2022. Find Bugs in Static Bug Finders. In 2022 IEEE/ACM 30th International Conference on Program Comprehension (ICPC). 516–527. https://doi.org/10.1145/3377811.3380380
[82]
Zhongxing Yu, Chenggang Bai, Lionel Seinturier, and Martin Monperrus. 2021. Characterizing the Usage, Evolution and Impact of Java Annotations in Practice. IEEE Transactions on Software Engineering, 47, 5 (2021), 969–986. https://doi.org/10.1109/TSE.2019.2910516
[83]
Huaien Zhang, Yu Pei, Junjie Chen, and Shin Hwei Tan. 2023. Statfier: Automated Testing of Static Analyzers via Semantic-preserving Program Transformations. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering.
[84]
Ru Zhang, Wencong Xiao, Hongyu Zhang, Yu Liu, Haoxiang Lin, and Mao Yang. 2020. An empirical study on program failures of deep learning jobs. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 1159–1170.
[85]
Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018. An empirical study on TensorFlow program bugs. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. 129–140.
Index Terms
- Understanding and Detecting Annotation-Induced Faults of Static Analyzers
Recommendations
Characterizing and Detecting Program Representation Faults of Static Analysis Frameworks
ISSTA 2024: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and AnalysisStatic analysis frameworks (SAFs) such as Soot and WALA have been a fundamental support in today’s software analysis. They usually adopt various analysis techniques to transform programs into different representations which imply specific properties, ...
Finding and Understanding Defects in Static Analyzers by Constructing Automated Oracles
Static analyzers are playing crucial roles in helping find programming mistakes and security vulnerabilities. The correctness of their analysis results is crucial for the usability in practice. Otherwise, the potential defects in these analyzers (, ...
Statfier: Automated Testing of Static Analyzers via Semantic-Preserving Program Transformations
ESEC/FSE 2023: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software EngineeringStatic analyzers reason about the behaviors of programs without executing them and report issues when they violate pre-defined desirable properties. One of the key limitations of static analyzers is their tendency to produce inaccurate and incomplete ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2024
2770 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 July 2024
Published in PACMSE Volume 1, Issue FSE
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 194Total Downloads
- Downloads (Last 12 months)194
- Downloads (Last 6 weeks)61
Reflects downloads up to 11 Dec 2024
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in