research-article

Open access

Navigating the Complexity of Generative AI Adoption in Software Engineering—RCR Report

Author:

Daniel RussoAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology, Volume 33, Issue 8

Article No.: 221, Pages 1 - 5

https://doi.org/10.1145/3680471

Published: 03 December 2024 Publication History

PDF eReader

Abstract

This Replicated Computational Results (RCR) report complements the study “Navigating the Complexity of Generative AI Adoption in Software Engineering,” which examines the factors influencing the integration of AI tools in software engineering practices. Employing a mixed-methods approach grounded in the Technology Acceptance Model, Diffusion of Innovation Theory, and Social Cognitive Theory, the study introduces the Human-AI Collaboration and Adaptation Framework (HACAF), validated through PLS-SEM analysis. The replication package detailed herein includes survey instruments, raw data, and analysis scripts essential for reproducing the study's findings. By providing these artifacts, the RCR report aims to support transparency, enable replication, and encourage further research on effective AI tool adoption strategies in software engineering.

1 Overview

This article investigates the adoption of Generative AI tools within the software engineering domain, emphasizing the influencing factors at individual, technological, and organizational levels. Conducted through a convergent mixed-methods approach, the study spans an extensive examination of how software engineers perceive and integrate AI tools into their workflows. The research leverages established theoretical frameworks, including the Technology Acceptance Model (TAM), Diffusion of Innovation Theory (DOI), and Social Cognitive Theory (SCT), to dissect the multifaceted nature of AI adoption.

Our findings suggest that compatibility with existing development workflows is a critical driver for the adoption of Generative AI tools, challenging traditional beliefs about technology acceptance. While perceived usefulness and social influences are commonly highlighted in the literature as significant adoption factors, our study reveals a nuanced reality where these elements play a less pronounced role in the early stages of Generative AI integration into software engineering practices.

The article introduces the Human-AI Collaboration and Adaptation Framework (HACAF) as a novel lens to understand and navigate the complexities of AI tool adoption. Validated through Partial Least Squares-Structural Equation Modeling (PLS-SEM), the HACAF model provides a comprehensive understanding of the dynamics at play, including personal innovativeness and the impact of social factors.

These insights not only contribute to the theoretical discourse on AI tool adoption but also offer practical implications for tool designers and organizational leaders aiming to foster effective integration of AI technologies in software engineering. As the domain continues to evolve, understanding these adoption dynamics becomes crucial for developing strategies that enhance innovation and productivity. For an in-depth analysis, we refer the reader to the article [1].

1.1 Data Generation and Statistical Analysis

The data for this study was obtained through a comprehensive multi-phase approach:

Data Generation.

—

Survey Design and Distribution: A questionnaire survey was designed based on the TAM, DOI, and SCT. Participants were recruited via Prolific Academic, targeting software engineers with specific eligibility criteria.

—

Validation Survey: A second survey using validated instruments was conducted to confirm the theoretical model.

Statistical Analysis.

—

PLS-SEM: PLS-SEM was employed to validate the theoretical model, handling complex models with multiple constructs and indicators.

—

Overfitting Analysis: The dataset was split into training and validation sets, and cross-validation techniques were applied to ensure model robustness and generalizability.

1.2 Artifacts

The artifacts associated with the study on the adoption of Generative AI tools in software engineering are publicly accessible on Zenodo and comprise essential components for reproducing and extending the research findings. These components are systematically outlined in Table 1 for easy reference.

Table 1.

Artifact	Location (DOI)	Description
Survey Instruments	https://doi.org/10.5281/zenodo.12205686	Includes the two questionnaires for the survey studies.
PLS-SEM Analysis Files	https://doi.org/10.5281/zenodo.12205686	Contains files for PLS-SEM analysis for the empirical validation of the HACAF model.
Overfitting Analysis	https://doi.org/10.5281/zenodo.12205686	Features analysis focused on overfitting issues within the model, ensuring the robustness and reliability of the findings.

Table 1. Overview of Artifacts, Location, and Description

The artifacts include comprehensive datasets from surveys and interviews that underpin the development and validation of the HACAF. The inclusion of overfitting analysis highlights the meticulous approach taken to ensure the integrity and reliability of the model validation process.

PLS-SEM analysis files, along with detailed scripts for conducting both quantitative and qualitative analyses, form the backbone of the study's empirical validation. These scripts not only facilitate the reproduction of the study's core findings but also enable scholars and practitioners to probe deeper into the data, potentially uncovering new insights into the adoption of AI tools in software engineering.

For researchers interested in the methodological rigor of model validation, the overfitting analysis artifact offers valuable insights into the techniques and considerations employed to minimize bias and enhance the validity of the study's conclusions.

To effectively utilize these artifacts, it is recommended to place all relevant files and scripts in the same working directory and adhere to the guidelines specified within each component's documentation.

1.3 Prerequisites and Requirements

To engage with the study's replication package effectively, certain technical requirements must be met:

—

SmartPLS (version 4.0.9.5 or above) is essential for conducting the PLS-SEM analysis as it was the specific software used in the original study for model validation and hypothesis testing.

—

A software environment capable of executing R or Python scripts is required for additional data analyses included in the package.

—

PDF and Excel file readers are necessary to access the survey instruments and the PLS-SEM analysis results, respectively.

Participants in the original study were selected based on their expertise in software engineering and familiarity with Generative AI tools, ensuring the collected data robustly represents professionals’ views on AI tool adoption.

To replicate the study or further explore its findings, the following steps are recommended:

(1)

Familiarize yourself with the survey and interview materials to grasp the context and scope of the data collection.

(2)

Review the PLS-SEM results and raw data files to understand the empirical basis of the study's conclusions.

(3)

Ensure the installation of SmartPLS (version 4.0.9.5 or above), along with R or Python, to execute the analysis scripts.

(4)

Organize all relevant materials in a single directory to streamline the analysis process.

(5)

Execute the analysis scripts to replicate the study's findings or to pursue new analytical directions.

1.4 Steps to Reproduce

For researchers interested in replicating or extending the study's analysis on the adoption of Generative AI tools in software engineering, the following methodology is outlined:

(1)

Secure a participant pool reflective of the original study's targeted demographic of software engineers experienced in Generative AI technologies.

(2)

Follow the data collection protocols detailed in the replication package for survey distribution and interviews, maintaining ethical standards and participant confidentiality.

(3)

Utilize SmartPLS (4.0.9.5 or above) or similar statistical packages for PLS-SEM analysis to validate the HACAF and to test related hypotheses.

Detailed instructions for data analysis:

(1)

Install SmartPLS (4.0.9.5 or above) along with the necessary R or Python environments for executing supplementary analysis scripts.

(2)

Download and organize the survey data, interview transcripts, and any other relevant files as outlined in the replication package.

(3)

Follow the guidelines within the analysis scripts and SmartPLS projects to conduct the analysis, adjusting parameters as needed to explore specific research inquiries.

These steps ensure a comprehensive approach to replicating the study's methodology and findings, offering a solid foundation for further investigation into Generative AI tool adoption among software engineers.

2 Artifact Instructions

2.1 Preparing the Artifact

To ensure the artifact is ready for replication and further analysis, adhere to the steps outlined below:

(1)

Download the Replication Package:

—

Access the Zenodo repository via the provided link: https://doi.org/10.5281/zenodo.12205686.

—

Download the complete replication package, which encompasses survey data, interview transcripts, PLS-SEM analysis files, and analysis scripts.

(2)

Organize the Files:

—

On your computer, create a folder specifically for the replication package, e.g., “GenerativeAIAdoptionStudy”.

—

Unpack the downloaded files into this folder, ensuring a logical organization for easy access.

(3)

Software Installation:

—

Install SmartPLS (version 4.0.9.5 or above) from https://www.smartpls.com/ to perform the PLS-SEM analysis as done in the original study.

—

Install R from https://cran.r-project.org/ and, optionally, RStudio from https://www.rstudio.com/ for running analysis scripts not related to PLS-SEM.

(4)

Review the Data and Analysis Methods:

—

Examine the survey data and interview transcripts to understand the scope and depth of the qualitative and quantitative insights gathered.

—

Assess the PLS-SEM analysis files to familiarize yourself with the structural model and the analytical process.

(5)

Understand the Analysis Scripts:

—

Review any provided R or Python scripts to comprehend the steps taken for additional analyses, including data preparation and preliminary analyses.

(6)

Configure the Analysis Environment:

—

Ensure that SmartPLS, R, and Python are correctly installed and functioning on your system.

—

Place all analysis files, including the SmartPLS project files and any scripts, in the designated project folder to streamline the analysis process.

Completing these steps will prepare you to replicate the original study's findings or to conduct further analysis.

Acknowledgment

ChatGPT-4 has been used to ensure linguistic accuracy and enhance the readability of this article.

Reference

[1]

Daniel Russo. 2024. Navigating the complexity of generative AI adoption in software engineering. ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 1–50.

Digital Library

Google Scholar

Cited By

View all

Russo D(2024)Navigating the Complexity of Generative AI Adoption in Software Engineering—RCR ReportACM Transactions on Software Engineering and Methodology10.1145/368047133:8(1-5)Online publication date: 23-Jul-2024
https://dl.acm.org/doi/10.1145/3680471
de Moor Avan Deursen AIzadi MAdams BZimmermann TOzkaya ILin DZhang J(2024)A Transformer-Based Approach for Smart Invocation of Automatic Code CompletionProceedings of the 1st ACM International Conference on AI-Powered Software10.1145/3664646.3664760(28-37)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3664646.3664760
Başer MBüyükbeşe T(2024)Exploring the Intention of Travel Agencies to Adopt Chatbots: Integrating TOE and MGBJournal of Quality Assurance in Hospitality & Tourism10.1080/1528008X.2024.2386588(1-35)Online publication date: Aug-2024
https://doi.org/10.1080/1528008X.2024.2386588

Index Terms

Navigating the Complexity of Generative AI Adoption in Software Engineering—RCR Report
1. Social and professional topics
  1. Professional topics
    1. Computing industry
    2. Management of computing and information systems
      1. Project and people management

Recommendations

Navigating the Complexity of Generative AI Adoption in Software Engineering
This article explores the adoption of Generative Artificial Intelligence (AI) tools within the domain of software engineering, focusing on the influencing factors at the individual, technological, and social levels. We applied a convergent mixed-methods ...
Can GPT-4 Replicate Empirical Software Engineering Research?

Empirical software engineering research on production systems has brought forth a better understanding of the software engineering process for practitioners and researchers alike. However, only a small subset of production systems is studied, limiting ...
Evidence-Based Software Engineering for Practitioners

Software engineers might make incorrect decisions about adopting new techniques if they donýt consider scientific evidence about the techniquesý efficacy. Procedures used for evidence-based medicine can also apply to software engineering. Such evidence-...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 33, Issue 8

November 2024

972 pages

EISSN:1557-7392

DOI:10.1145/3613733

Editor:
Mauro Pezze
Universita della Svizzera italiana

Issue’s Table of Contents

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 December 2024

Online AM: 23 July 2024

Accepted: 26 June 2024

Revised: 21 June 2024

Received: 10 March 2024

Published in TOSEM Volume 33, Issue 8

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
248
Total Downloads

Downloads (Last 12 months)248
Downloads (Last 6 weeks)97

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Russo D(2024)Navigating the Complexity of Generative AI Adoption in Software Engineering—RCR ReportACM Transactions on Software Engineering and Methodology10.1145/368047133:8(1-5)Online publication date: 23-Jul-2024
https://dl.acm.org/doi/10.1145/3680471
de Moor Avan Deursen AIzadi MAdams BZimmermann TOzkaya ILin DZhang J(2024)A Transformer-Based Approach for Smart Invocation of Automatic Code CompletionProceedings of the 1st ACM International Conference on AI-Powered Software10.1145/3664646.3664760(28-37)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3664646.3664760
Başer MBüyükbeşe T(2024)Exploring the Intention of Travel Agencies to Adopt Chatbots: Integrating TOE and MGBJournal of Quality Assurance in Hospitality & Tourism10.1080/1528008X.2024.2386588(1-35)Online publication date: Aug-2024
https://doi.org/10.1080/1528008X.2024.2386588

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

1 Overview

1.1 Data Generation and Statistical Analysis

1.2 Artifacts

1.3 Prerequisites and Requirements

1.4 Steps to Reproduce

2 Artifact Instructions

2.1 Preparing the Artifact

Acknowledgment

Reference

Cited By

Index Terms

Recommendations