short-paper

Open access

Assessing AI-Based Code Assistants in Method Generation Tasks

Authors:

Vincenzo Corso,

Leonardo Mariani,

Daniela Micucci,

Oliviero RiganelliAuthors Info & Claims

ICSE-Companion '24: Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings

Pages 380 - 381

https://doi.org/10.1145/3639478.3643122

Published: 23 May 2024 Publication History

PDF eReader

Abstract

AI-based code assistants are increasingly popular as a means to enhance productivity and improve code quality. This study compares four AI-based code assistants, GitHub Copilot, Tabnine, ChatGPT, and Google Bard, in method generation tasks, assessing their ability to produce accurate, correct, and efficient code. Results show that code assistants are useful, with complementary capabilities, although they rarely generate ready-to-use correct code.

References

[1]

Vincenzo Corso, Leonardo Mariani, Daniela Micucci, and Oliviero Riganelli. 2024. Generating Java Methods: An Empirical Assessment of Four AI-Based Code Assistants. In Proceedings of the International Conference on Program Comprehension.

Google Scholar

[2]

Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023. GitHub Copilot AI pair programmer: Asset or Liability? Journal of Systems and Software 203 (2023).

Google Scholar

[3]

GitHub. 2023. Copilot. https://github.com/features/copilot.

Google Scholar

[4]

Google. 2023. Bard. https://bard.google.com.

Google Scholar

[5]

Microsoft. 2020. CodeXGLUE. https://shorturl.at/gwxIL.

Google Scholar

[6]

OpenAI. 2023. ChatGPT. https://openai.com/chatgpt.

Google Scholar

[7]

PyPI. 2023. python-Levenshtein 0.21.1. https://shorturl.at/iKVW6.

Google Scholar

[8]

Tabnine. 2023. Tabnine. https://www.tabnine.com.

Google Scholar

[9]

Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the Conference on Human Factors in Computing Systems.

Google Scholar

[10]

Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. 2022. Assessing the Quality of GitHub Copilot's Code Generation. In Proceedings of the International Conference on Predictive Models and Data Analytics in Software Engineering.

Digital Library

Google Scholar

Index Terms

Assessing AI-Based Code Assistants in Method Generation Tasks
1. Software and its engineering
  1. Software notations and tools
    1. Development frameworks and environments
      1. Integrated and visual development environments

Recommendations

Generating Java Methods: An Empirical Assessment of Four AI-Based Code Assistants
ICPC '24: Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension

AI-based code assistants are promising tools that can facilitate and speed up code development. They exploit machine learning algorithms and natural language processing to interact with developers, suggesting code snippets (e.g., method implementations) ...
Assessing the quality of GitHub copilot’s code generation
PROMISE 2022: Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering

The introduction of GitHub’s new code generation tool, GitHub Copilot, seems to be the first well-established instance of an AI pair-programmer. GitHub Copilot has access to a large number of open-source projects, enabling it to utilize more extensive ...
Demystifying code snippets in code reviews: a study of the OpenStack and Qt communities and a practitioner survey
Abstract
Code review is widely known as one of the best practices for software quality assurance in software development. In a typical code review process, reviewers check the code committed by developers to ensure the quality of the code, during which ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ICSE-Companion '24: Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings

April 2024

531 pages

ISBN:9798400705021

DOI:10.1145/3639478

Co-chairs:
Ana Paiva,
Rui Abreu,
Program Co-chairs:
Abhik Roychoudhury,
Margaret Storey

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs International 4.0 License.

In-Cooperation

Faculty of Engineering of University of Porto

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 May 2024

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Ministero dell'Università e della Ricerca

Conference

ICSE-Companion '24

Sponsor:

SIGSOFT

ICSE-Companion '24: 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings

April 14 - 20, 2024

Lisbon, Portugal

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
190
Total Downloads

Downloads (Last 12 months)190
Downloads (Last 6 weeks)35

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Index Terms

Recommendations

Generating Java Methods: An Empirical Assessment of Four AI-Based Code Assistants

Assessing the quality of GitHub copilot’s code generation

Demystifying code snippets in code reviews: a study of the OpenStack and Qt communities and a practitioner survey