More Web Proxy on the site http://driver.im/

short-paper

Public Access

NLBSE'22 tool competition

Authors:

Oscar Chaparro,

Andrea Di Sorbo,

Sebastiano PanichellaAuthors Info & Claims

NLBSE '22: Proceedings of the 1st International Workshop on Natural Language-based Software Engineering

Pages 25 - 28

https://doi.org/10.1145/3528588.3528664

Published: 01 February 2023 Publication History

Abstract

We report on the organization and results of the first edition of the Tool Competition from the International Workshop on Natural Language-based Software Engineering (NLBSE'22). This year, five teams submitted multiple classification models to automatically classify issue reports as bugs, enhancements, or questions. Most of them are based on BERT (Bidirectional Encoder Representations from Transformers) and were fine-tuned and evaluated on a benchmark dataset of 800k issue reports. The goal of the competition was to improve the classification performance of a baseline model based on fastText. This report provides details of the competition, including its rules, the teams and contestant models, and the ranking of models based on their average classification performance across the issue types.

References

[1]

Shikhar Bharadwaj and Tushar Kadam. Github issue classification using bert-style models. In Proceedings of The 1st International Workshop on Natural Language-based Software Engineering (NLBSE'22), page (to appear), 2022.

[2]

Giuseppe Colavito, Filippo Lanubile, and Nicole Novielli. Issue report classification using pre-trained language models. In Proceedings of The 1st International Workshop on Natural Language-based Software Engineering (NLBSE'22), page (to appear), 2022.

[3]

Maliheh Izadi. Catiss: An intelligent tool for categorizing issues reports using transformers. In Proceedings of The 1st International Workshop on Natural Language-based Software Engineering (NLBSE'22), page (to appear), 2022.

[4]

Mohammed Latif Siddiq and Joanna C.S. Santos. Bert-based github issue report classification. In Proceedings of The 1st International Workshop on Natural Language-based Software Engineering (NLBSE'22), page (to appear), 2022.

[5]

Alexander Trautsch and Steffen Herbold. Predicting issue types with sebert. In Proceedings of The 1st International Workshop on Natural Language-based Software Engineering (NLBSE'22), page (to appear), 2022.

[6]

Rafael Kallis, Oscar Chaparro, Andrea Di Sorbo, and Sebastiano Panichella. Nlbse'22 tool competition. In Proceedings of The 1st International Workshop on Natural Language-based Software Engineering (NLBSE'22), 2022.

[7]

Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759, 2016.

[8]

Rafael Kallis, Andrea Di Sorbo, Gerardo Canfora, and Sebastiano Panichella. Ticket tagger: Machine learning driven issue classification. In Proceedings of IEEE International Conference on Software Maintenance and Evolution (ICSME'19), pages 406--409, 2019.

[9]

Rafael Kallis, Andrea Di Sorbo, Gerardo Canfora, and Sebastiano Panichella. Predicting issue types on github. Science of Computer Programming, 205:102598, 2021.

[10]

Ilya Grigorik. The github archive. https://www.gharchive.org/, 2022.

[11]

Cloud Google. Bigquery - google cloud platform. https://cloud.google.com/bigquery/, 2022.

[12]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

[13]

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.

[14]

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020.

[15]

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.

[16]

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019.

[17]

Julian von der Mosel, Alexander Trautsch, and Steffen Herbold. On the validity of pre-trained transformers for natural language processing in the software engineering domain. arXiv preprint arXiv:2109.04738, 2021.

[18]

Sebastiano Panichella, Andrea Di Sorbo, Emitza Guzman, Corrado Aaron Visaggio, Gerardo Canfora, and Harald C. Gall. How can i improve my app? classifying user reviews for software maintenance and evolution. In International Conference on Software Maintenance and Evolution, pages 281--290. IEEE, 2015.

Digital Library

[19]

Andrea Di Sorbo, Sebastiano Panichella, Carol V. Alexandru, Junji Shimagaki, Corrado Aaron Visaggio, Gerardo Canfora, and Harald C. Gall. What would users change in my app? summarizing app reviews for recommending software changes. In International Symposium on Foundations of Software Engineering, pages 499--510. ACM, 2016.

Digital Library

[20]

Andrea Di Sorbo, Giovanni Grano, Corrado Aaron Visaggio, and Sebastiano Panichella. Investigating the criticality of user-reported issues through their relations with app rating. J. Softw. Evol. Process., 33(3), 2021.

Digital Library

[21]

Sebastiano Panichella. Summarization techniques for code, change, testing, and user feedback (invited paper). In Cyrille Artho and Rudolf Ramler, editors, 2018 IEEE Workshop on Validation, Analysis and Evolution of Software Tests, VST@SANER 2018, Campobasso, Italy, March 20, 2018, pages 1--5. IEEE, 2018.

[22]

Pooja Rani, Sebastiano Panichella, Manuel Leuenberger, Andrea Di Sorbo, and Oscar Nierstrasz. How to identify class comment types? A multi-language approach for class comment classification. J. Syst. Softw., 181:111047, 2021.

Digital Library

[23]

Yang Song and Oscar Chaparro. Bee: A tool for structuring and analyzing bug reports. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE'20), pages 1551--1555, 2020.

Digital Library

[24]

Oscar Chaparro, Carlos Bernal-Cárdenas, Jing Lu, Kevin Moran, Andrian Marcus, Massimiliano Di Penta, Denys Poshyvanyk, and Vincent Ng. Assessing the quality of the steps to reproduce in bug reports. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE'19), pages 86--96, 2019.

Digital Library

[25]

Oscar Chaparro, Jing Lu, Fiorella Zampetti, Laura Moreno, Massimiliano Di Penta, Andrian Marcus, Gabriele Bavota, and Vincent Ng. Detecting missing information in bug descriptions. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE'17), pages 396--407, 2017.

Digital Library

Cited By

Rejithkumar GAnish PGhaisas SIzadi MDi Sorbo APanichella S(2024)Text-To-Text Generation for Issue Report ClassificationProceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering10.1145/3643787.3648042(53-56)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643787.3648042
Kallis RColavito GAl-Kaswan APascarella LChaparro ORani PIzadi MDi Sorbo APanichella S(2024)The NLBSE'24 Tool CompetitionProceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering10.1145/3643787.3648038(33-40)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643787.3648038
Colavito GLanubile FNovielli NQuaranta L(2024)Impact of data quality for automatic issue classification using pre-trained language modelsJournal of Systems and Software10.1016/j.jss.2023.111838210:COnline publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111838
Show More Cited By

Recommendations

The NLBSE'24 Tool Competition
NLBSE '24: Proceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering

We report on the organization and results of the tool competition of the third International Workshop on Natural Language-based Software Engineering (NLBSE'24). As in prior editions, we organized the competition on automated issue report classification, ...
Supermarket Competition: The Case of Every Day Low Pricing

Every Day Low Pricing EDLP strategy has proved to be a successful innovation resulting in higher profits to supermarkets adopting it in competition with Promotional Pricing PROMO. Conventional wisdom attributes this success either to lower costs or to ...
Consumer preferences, cannibalization, and competition: evidence from the personal computer industry

Understanding the degree of cannibalization and competition in online and offline markets is important to firms' product line designs. However, few empirical studies have measured both effects simultaneously or have examined the factors that determine ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

NLBSE '22: Proceedings of the 1st International Workshop on Natural Language-based Software Engineering

May 2022

87 pages

ISBN:9781450393430

DOI:10.1145/3528588

Conference Chairs:
Andrea Di Sorbo
University of Sannio, Benevento, Italy
,
Sebastiano Panichella
Zurich University of Applied Sciences, Zurich, Switzerland

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 February 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Short-paper

Funding Sources

Conference

ICSE '22

Sponsor:

SIGSOFT

ICSE '22: 44th International Conference on Software Engineering

May 21, 2022

Pennsylvania, Pittsburgh

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
184
Total Downloads

Downloads (Last 12 months)124
Downloads (Last 6 weeks)13

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rejithkumar GAnish PGhaisas SIzadi MDi Sorbo APanichella S(2024)Text-To-Text Generation for Issue Report ClassificationProceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering10.1145/3643787.3648042(53-56)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643787.3648042
Kallis RColavito GAl-Kaswan APascarella LChaparro ORani PIzadi MDi Sorbo APanichella S(2024)The NLBSE'24 Tool CompetitionProceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering10.1145/3643787.3648038(33-40)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643787.3648038
Colavito GLanubile FNovielli NQuaranta L(2024)Impact of data quality for automatic issue classification using pre-trained language modelsJournal of Systems and Software10.1016/j.jss.2023.111838210:COnline publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111838
Aktas ECakmak EInan MYilmaz C(2024)Improving the quality of software issue report descriptions in Turkish: An industrial case study at SofttechEmpirical Software Engineering10.1007/s10664-023-10434-429:2Online publication date: 12-Feb-2024
https://dl.acm.org/doi/10.1007/s10664-023-10434-4
Colavito GLanubile FNovielli N(2023)Few-Shot Learning for Issue Report Classification2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering (NLBSE)10.1109/NLBSE59153.2023.00011(16-19)Online publication date: May-2023
https://doi.org/10.1109/NLBSE59153.2023.00011
Laiq M(2023)An Intelligent Tool for Classifying Issue Reports2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering (NLBSE)10.1109/NLBSE59153.2023.00010(13-15)Online publication date: May-2023
https://doi.org/10.1109/NLBSE59153.2023.00010
Nikeghbal NHossein Kargaran AHeydarnoori ASchütze H(2023)GIRT-Data: Sampling GitHub Issue Report Templates2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)10.1109/MSR59073.2023.00026(104-108)Online publication date: May-2023
https://doi.org/10.1109/MSR59073.2023.00026
Colavito GLanubile FNovielli NSorbo APanichella S(2022)Issue report classification using pre-trained language modelsProceedings of the 1st International Workshop on Natural Language-based Software Engineering10.1145/3528588.3528659(29-32)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3528588.3528659

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents