[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3632620.3671112acmconferencesArticle/Chapter ViewAbstractPublication PagesicerConference Proceedingsconference-collections
research-article

An Investigation of the Drivers of Novice Programmers' Intentions to Use Web Search and GenAI

Published: 12 August 2024 Publication History

Abstract

External help resources are frequently used by novice programmers solving classwork in undergraduate computing courses. Traditionally, these tools consisted of web resources such as tutorial websites and Q&A forums. With the rise of Generative AI (GenAI), there has been increasing concern and research about how external resources should be used in the classroom. However, little work has directly contrasted student beliefs and perceptions of web resources with GenAI, has grounded these beliefs in prior psychological theory, and has investigated how demographic factors and student backgrounds influence these beliefs and intentions. We administered a vignette-style survey across two courses required for a CS major at an R1 University, a freshman (n = 152) and senior capstone course (n = 44). Students responded to likert questions aiming to measure behavioral factors related to these tools, such as intention to use, perceived attitudes, peer perceptions, and their own perceived tool competency. We primarily investigate the results of an introductory course, finding that novices have a wide range of opinions on both resources, but overall find them slightly useful and have a tendency to prefer web-search. We compare this with seniors, who have more positive perceptions of these tools, and discuss possible reasons and implications for this difference. We constructed two path models to investigate which factors strongly influence novices’ intention to use resources and find the primary factor to be their general attitudes in how these tools will result in a positive or negative outcome (e.g. perceived benefits, justifiability). We also measure the effects of student background on intention to use these resources. Finally, we discuss implications and suggestions on how instructors can use this information to approach, address, and influence resource usage in their classrooms.

References

[1]
Icek Ajzen. 1991. The theory of planned behavior. Organizational behavior and human decision processes 50, 2 (1991), 179–211.
[2]
Mareh Fakhir Al-sammarraie. 2017. An Empirical Investigation of Collaborative Web Search Tool on Novice ’ s Query Behavior. (2017).
[3]
Sara Amani, Lance White, Trini Balart, Laksha Arora, Dr. Kristi J. Shryock, Dr. Kelly Brumbelow, and Dr. Karan L. Watson. 2023. Generative AI Perceptions: A Survey to Measure the Perceptions of Faculty, Staff, and Students on Generative AI Tools in Academia. arxiv:2304.14415 [cs.HC]
[4]
Gina R Bai, Joshua Kayani, and Kathryn T Stolee. 2020. How graduate computing students search when using an unfamiliar programming language. In Proceedings of the 28th International Conference on Program Comprehension. 160–171.
[5]
Albert Bandura. 2001. Social Cognitive Theory of Mass Communication. (2001), 265–299.
[6]
Albert Bandura. 2012. Social Cognitive Theory. SAGE Publications Ltd, 349–374. https://doi.org/10.4135/9781446249215.n18
[7]
Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2023. Grounded Copilot: How Programmers Interact with Code-Generating Models. Proceedings of the ACM on Programming Languages 7, OOPSLA1 (2023), 1–27. https://doi.org/10.1145/3586030 arxiv:2206.15000
[8]
Joel Brandt, Philip J Guo, Joel Lewenstein, Mira Dontcheva, and Scott R Klemmer. 2009. Two studies of opportunistic programming: interleaving web foraging, learning, and writing code. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1589–1598.
[9]
Cecilia Ka Yuk Chan and Louisa H. Y. Tsi. 2023. The AI Revolution in Education: Will AI Replace or Assist Teachers in Higher Education?arxiv:2305.01185 [cs.CY]
[10]
Cecilia Ka Yuk Chan and Wenxin Zhou. 2023. An expectancy value theory (EVT) based instrument for measuring student perceptions of generative AI. Smart Learning Environments 10, 1 (Dec. 2023). https://doi.org/10.1186/s40561-023-00284-4
[11]
Prashanth Chandrasekar. 2023. Announcing OverflowAI. https://stackoverflow.blog/2023/07/27/announcing-overflowai/
[12]
Preetha Chatterjee, Minji Kong, and Lori Pollock. 2020. Finding help with programming errors: An exploratory study of novice software engineers’ focus in stack overflow posts. Journal of Systems and Software 159 (2020), 110454. https://doi.org/10.1016/j.jss.2019.110454
[13]
Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2024. Prompt Problems: A New Programming Exercise for the Generative AI Era. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (Portland, OR, USA) (SIGCSE 2024). Association for Computing Machinery, New York, NY, USA, 296–302. https://doi.org/10.1145/3626252.3630909
[14]
Chris Dewberry and Duncan JR Jackson. 2018. An application of the theory of planned behavior to student retention. Journal of Vocational Behavior 107 (2018), 100–110.
[15]
Adamantios Diamantopoulos, Marko Sarstedt, Christoph Fuchs, Petra Wilczynski, and Sebastian Kaiser. 2012. Guidelines for choosing between multi-item and single-item scales for construct measurement: a predictive validity perspective. Journal of the Academy of Marketing Science 40, 3 (Feb. 2012), 434–449. https://doi.org/10.1007/s11747-011-0300-3
[16]
Augie Doebling and Ayaan M. Kazerouni. 2021. Patterns of Academic Help-Seeking in Undergraduate Computing Students. ACM International Conference Proceeding Series (2021). https://doi.org/10.1145/3488042.3488052
[17]
Jayne Everson, F. Megumi Kivuva, and Amy J. Ko. 2022. "A Key to Reducing Inequities in Like, AI, is by Reducing Inequities Everywhere First": Emerging Critical Consciousness in a Co-Constructed Secondary CS Classroom. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1 (Providence, RI, USA) (SIGCSE 2022). Association for Computing Machinery, New York, NY, USA, 209–215. https://doi.org/10.1145/3478431.3499395
[18]
James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Proceedings of the 24th Australasian Computing Education Conference (Virtual Event, Australia) (ACE ’22). Association for Computing Machinery, New York, NY, USA, 10–19. https://doi.org/10.1145/3511861.3511863
[19]
Denae Ford, Justin Smith, Philip J. Guo, and Chris Parnin. 2016. Paradise unplugged: identifying barriers for female participation on stack overflow. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (Seattle, WA, USA) (FSE 2016). Association for Computing Machinery, New York, NY, USA, 846–857. https://doi.org/10.1145/2950290.2950331
[20]
Claes Fornell and David F. Larcker. 1981. Evaluating Structural Equation Models with Unobservable Variables and Measurement Error. Journal of Marketing Research 18, 1 (Feb. 1981), 39. https://doi.org/10.2307/3151312
[21]
Sabrina Gado, Regina Kempen, Katharina Lingelbach, and Tanja Bipp. 2021. Artificial intelligence in psychology: How can we enable psychology students to accept and use artificial intelligence?Psychology Learning & Teaching 21, 1 (Aug. 2021), 37–56. https://doi.org/10.1177/14757257211037149
[22]
Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md. Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen Ahmed. 2023. Bias and Fairness in Large Language Models: A Survey. ArXiv abs/2309.00770 (2023). https://doi.org/10.48550/arXiv.2309.00770
[23]
Jamie Gorson and Eleanor O’Rourke. 2020. Why do CS1 Students Think They’re Bad at Programming?: Investigating Self-efficacy and Self-assessments at Three Universities. ICER 2020 - Proceedings of the 2020 ACM Conference on International Computing Education Research (2020), 170–181. https://doi.org/10.1145/3372782.3406273
[24]
Jr. Hair, Joseph F., G. Tomas M. Hult, Christian M. Ringle, and Marko Sarstedt. 2022. A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM) (3 ed.). SAGE, Los Angeles. https://lccn.loc.gov/2021004786 Includes bibliographical references and index.
[25]
Jörg Henseler, Christian M. Ringle, and Marko Sarstedt. 2014. A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science 43, 1 (Aug. 2014), 115–135. https://doi.org/10.1007/s11747-014-0403-8
[26]
Muntasir Hoq, Yang Shi, Juho Leinonen, Damilola Babalola, Collin Lynch, Thomas Price, and Bita Akram. 2024. Detecting ChatGPT-Generated Code Submissions in a CS1 Course Using Machine Learning Models. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (Portland, OR, USA,) (SIGCSE 2024). Association for Computing Machinery, New York, NY, USA, 526–532. https://doi.org/10.1145/3626252.3630826
[27]
Irene Hou, Sophia Mettille, Owen Man, Zhuo Li, Cynthia Zastudil, and Stephen MacNeil. 2024. The Effects of Generative AI on Computing Students’ Help-Seeking Preferences. In Proceedings of the 26th Australasian Computing Education Conference (Sydney, NSW, Australia) (ACE ’24). Association for Computing Machinery, New York, NY, USA, 39–48. https://doi.org/10.1145/3636243.3636248
[28]
Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. Vol. 1. Association for Computing Machinery. https://doi.org/10.1145/3544548.3580919 arxiv:2302.07427
[29]
Shao-Heng Ko and Kristin Stephens-Martinez. 2023. What Drives Students to Office Hours: Individual Differences and Similarities. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON>, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 959–965. https://doi.org/10.1145/3545945.3569777
[30]
Sam Lau and Philip J Guo. 2023. From "Ban It Till We Understand It" to "Resistance is Futile": How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as ChatGPT and GitHub Copilot. Vol. 1. Association for Computing Machinery. https://doi.org/10.1145/3568813.3600138
[31]
Annie Li, Madeline Endres, and Westley Weimer. 2022. Debugging with Stack Overflow: Web Search Behavior in Novice and Expert Programmers. (2022).
[32]
Hongwei Li, Zhenchang Xing, Xin Peng, and Wenyun Zhao. 2013. What help do developers seek, when and how?. In 2013 20th working conference on reverse engineering (WCRE). IEEE, 142–151.
[33]
Michael Liut, Anna Ly, Jessica Jia-Ni Xu, Justice Banson, Paul Vrbik, and Caroline D. Hardin. 2024. “I Didn’t Know”: Examining Student Understanding of Academic Dishonesty in Computer Science. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1(SIGCSE 2024). ACM. https://doi.org/10.1145/3626252.3630753
[34]
Lauri Malmi, Judy Sheard, Päivi Kinnunen, Simon, and Jane Sinclair. 2019. Computing Education Theories: What Are They and How Are They Used?. In Proceedings of the 2019 ACM Conference on International Computing Education Research(ICER ’19). ACM. https://doi.org/10.1145/3291279.3339409
[35]
Silvia Muller, Monica Babes-Vroman, Mary Emenike, and Thu D Nguyen. 2020. Exploring Novice Programmers’ Homework Practices: Initial Observations of Information Seeking Behaviors. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. 333–339.
[36]
Marek Muszyński. 2023. Attention checks and how to use them: Review and practical recommendations. Ask: Research and Methods 32, 1 (2023), 3–38. https://doi.org/10.18061/ask.v32i1.0001
[37]
National Center for Science and Engineering Statistics (NCSES). 2023. Diversity and STEM: Women, Minorities, and Persons with Disabilities 2023. Special Report NSF 23-315. National Science Foundation, Alexandria, VA. https://ncses.nsf.gov/wmpd
[38]
Greg L. Nelson and Amy J. Ko. 2018. On Use of Theory in Computing Education Research. In Proceedings of the 2018 ACM Conference on International Computing Education Research(ICER ’18). ACM. https://doi.org/10.1145/3230977.3230992
[39]
Sharon Nelson-Le Gall. 1981. Help-seeking: An understudied problem-solving skill in children. Developmental review 1, 3 (1981), 224–246.
[40]
James Prather, Paul Denny, Juho Leinonen, Brett A. Becker, Ibrahim Albluwi, Michael E. Caspersen, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew Luxton-Reilly, Stephen MacNeil, Andrew Petersen, Raymond Pettit, Brent N. Reeves, and Jaromir Savelka. 2023. Transformed by Transformers: Navigating the AI Coding Revolution for Computing Education: An ITiCSE Working Group Conducted by Humans. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 561–562. https://doi.org/10.1145/3587103.3594206
[41]
James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023. "It’s Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers. Proceedings of ACM Transactions on Computer-Human Interaction (TOCHI) 1, 1 (2023), 1–26. https://doi.org/11.1111/1111111.1111111 arxiv:2304.02491
[42]
Brent Reeves, Sami Sarsa, James Prather, Paul Denny, Brett A. Becker, Arto Hellas, Bailey Kimmel, Garrett Powell, and Juho Leinonen. 2023. Evaluating the Performance of Code Generation Models for Solving Parsons Problems With Small Prompt Variations. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 299–305. https://doi.org/10.1145/3587102.3588805
[43]
Christian M. Ringle, Sven Wende, and Jan-Michael Becker. 2024. SmartPLS 4. SmartPLS, Bönningstedt. https://www.smartpls.com
[44]
Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence?Section 2 (2022), 1–27. arxiv:2208.06213http://arxiv.org/abs/2208.06213
[45]
Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (Lugano and Virtual Event, Switzerland) (ICER ’22). Association for Computing Machinery, New York, NY, USA, 27–43. https://doi.org/10.1145/3501385.3543957
[46]
Marko Sarstedt and P. Wilczynski. 2009. More for Less? A Comparison of Single-item and Multi-item Measures. Die Betriebswirtschaft 69 (01 2009), 211–227.
[47]
Jaromir Savelka, Arav Agarwal, Marshall An, Chris Bogart, and Majd Sakr. 2023. Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. iii (2023). https://doi.org/10.1145/3568813.3600142 arxiv:2306.10073
[48]
James Skripchuk, Neil Bennett, Jeffrey Zheng, Eric Li, and Thomas Price. 2023. Analysis of Novices’ Web-Based Help-Seeking Behavior While Programming. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 945–951. https://doi.org/10.1145/3545945.3569852
[49]
Manuel Sojer, Oliver Alexy, Sven Kleinknecht, and Joachim Henkel. 2014. Understanding the Drivers of Unethical Programming Behavior: The Inappropriate Reuse of Internet-Accessible Code. Journal of Management Information Systems 31, 3 (2014), 287–325. https://doi.org/10.1080/07421222.2014.995563
[50]
Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. Conference on Human Factors in Computing Systems - Proceedings (2022). https://doi.org/10.1145/3491101.3519665
[51]
David Wong-Aitken, Diana Cukierman, and Parmit K. Chilana. 2022. "It Depends on Whether or Not I’m Lucky": How Students in an Introductory Programming Course Discover, Select, and Assess the Utility of Web-Based Resources. Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE 1 (2022), 512–518. https://doi.org/10.1145/3502718.3524751
[52]
Xin Xia, Lingfeng Bao, David Lo, Pavneet Singh Kochhar, Ahmed E. Hassan, and Zhenchang Xing. 2017. What do developers search for on the web?Empirical Software Engineering 22, 6 (2017), 3149–3185. https://doi.org/10.1007/s10664-017-9514-4

Index Terms

  1. An Investigation of the Drivers of Novice Programmers' Intentions to Use Web Search and GenAI

      Recommendations

      Comments

      Please enable JavaScript to view thecomments powered by Disqus.

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      ICER '24: Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1
      August 2024
      539 pages
      ISBN:9798400704758
      DOI:10.1145/3632620
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 12 August 2024

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. CS Education
      2. GenAI
      3. Help-seeking
      4. student perspectives
      5. web-search

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Funding Sources

      Conference

      ICER 2024
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 189 of 803 submissions, 24%

      Upcoming Conference

      ICER 2025
      ACM Conference on International Computing Education Research
      August 3 - 6, 2025
      Charlottesville , VA , USA

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 209
        Total Downloads
      • Downloads (Last 12 months)209
      • Downloads (Last 6 weeks)24
      Reflects downloads up to 05 Feb 2025

      Other Metrics

      Citations

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media