More Web Proxy on the site http://driver.im/

short-paper

Open access

Kattis vs ChatGPT: Assessment and Evaluation of Programming Tasks in the Age of Artificial Intelligence

Authors:

Jacqueline Wong,

Olga VibergAuthors Info & Claims

LAK '24: Proceedings of the 14th Learning Analytics and Knowledge Conference

Pages 821 - 827

https://doi.org/10.1145/3636555.3636882

Published: 18 March 2024 Publication History

All formats PDF

Abstract

AI-powered education technologies can support students and teachers in computer science education. However, with the recent developments in generative AI, and especially the increasingly emerging popularity of ChatGPT, the effectiveness of using large language models for solving programming tasks has been underexplored. The present study examines ChatGPT’s ability to generate code solutions at different difficulty levels for introductory programming courses. We conducted an experiment where ChatGPT was tested on 127 randomly selected programming problems provided by Kattis, an automatic software grading tool for computer science programs, often used in higher education. The results showed that ChatGPT independently could solve 19 out of 127 programming tasks generated and assessed by Kattis. Further, ChatGPT was found to be able to generate accurate code solutions for simple problems but encountered difficulties with more complex programming tasks. The results contribute to the ongoing debate on the utility of AI-powered tools in programming education.

Supplemental Material

PDF File

Appendix: Presenting examples of programming tasks, along with their solutions, and explaining the Kattis rating process.

Download
2.00 MB

References

[1]

Yazid AlBadarin, Markku Tukiainen, Mohammed Saqr, and Nicholas Pope. 2023. A Systematic Literature Review of Empirical Research on ChatGPT in Education. Available at SSRN 4562771 (2023).

[2]

Ram B Basnet, Tenzin Doleck, David John Lemay, and Paul Bazelais. 2018. Exploring computer science students’ continuance intentions to use Kattis. Education and Information Technologies 23 (2018), 1145–1158.

Digital Library

[3]

Brett A Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023. Programming is hard-or at least it used to be: Educational opportunities and challenges of ai code generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 500–506.

Digital Library

[4]

Cecilia Ka Yuk Chan and Wenjie Hu. 2023. Students’ Voices on Generative AI: Perceptions, Benefits, and Challenges in Higher Education. arXiv preprint arXiv:2305.00290 (2023).

[5]

Hayden Cheers, Yuqing Lin, and Shamus P Smith. 2021. Academic source code plagiarism detection by measuring program behavioral similarity. IEEE Access 9 (2021), 50391–50412.

[6]

Tyne Crow, Andrew Luxton-Reilly, and Burkhard Wuensche. 2018. Intelligent tutoring systems for programming education: a systematic review. In Proceedings of the 20th Australasian Computing Education Conference. 53–62.

Digital Library

[7]

Yun Dai, Ang Liu, and Cher Ping Lim. 2023. Reconceptualizing ChatGPT and generative AI as a student-driven innovation in higher education. (2023).

[8]

Emma Enström, Gunnar Kreitz, Fredrik Niemelä, and Viggo Kann. 2010. Testdriven utbildning–strukturerad formativ examination. In NU2010, Dialog för lärande. Stockholm, Swe. 13-15 oktober 2010.

[9]

Borja Fernandez-Gauna, Naiara Rojo, and Manuel Graña. 2023. Automatic feedback and assessment of team-coding assignments in a DevOps context. International Journal of Educational Technology in Higher Education 20, 1 (2023), 17.

[10]

Tim Fütterer, Christian Fischer, Anastasiia Alekseeva, Xiaobin Chen, Tamara Tate, Mark Warschauer, and Peter Gerjets. 2023. ChatGPT in Education: Global Reactions to AI Innovations. (2023).

[11]

Dragan Gašević, George Siemens, and Shazia Sadiq. 2023. Empowering learners for the age of artificial intelligence. Computers and Education: Artificial Intelligence (2023), 100130.

[12]

Chuqin Geng, Zhang Yihan, Brigitte Pientka, and Xujie Si. 2023. Can ChatGPT Pass An Introductory Level Functional Language Programming Course?arXiv preprint arXiv:2305.02230 (2023).

[13]

Reza Hadi Mogavi, Chao Deng, Justin Juho Kim, Pengyuan Zhou, Young D Kwon, Ahmed Hosny Saleh Metwally, Ahmed Tlili, Simone Bassanelli, Antonio Bucchiarone, Sujit Gujar, 2023. Exploring User Perspectives on ChatGPT: Applications, Perceptions, and Implications for AI-Integrated Education. arXiv e-prints (2023), arXiv–2305.

[14]

Lingchen Huang, Huazi Zhang, Rong Li, Yiqun Ge, and Jun Wang. 2019. AI coding: Learning to construct error correction codes. IEEE Transactions on Communications 68, 1 (2019), 26–39.

[15]

Nayab Iqbal, Hassaan Ahmed, and Kaukab Abid Azhar. 2022. Exploring teachers’ attitudes towards using chatgpt. Glob. J. Manag. Adm. Sci 3 (2022), 97–111.

[16]

Sung-Hee Jin, Kowoon Im, Mina Yoo, Ido Roll, and Kyoungwon Seo. 2023. Supporting students’ self-regulated learning in online learning using artificial intelligence applications. International Journal of Educational Technology in Higher Education 20, 1 (2023), 1–21.

[17]

Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–23.

Digital Library

[18]

Natalie Kiesler and Daniel Schiffner. 2023. Large Language Models in Introductory Programming Education: ChatGPT’s Performance and Implications for Assessments. arXiv preprint arXiv:2308.08572 (2023).

[19]

Weng Marc Lim, Asanka Gunasekara, Jessica Leigh Pallant, Jason Ian Pallant, and Ekaterina Pechenkina. 2023. Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education 21, 2 (2023), 100790.

[20]

Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021).

[21]

Iain Milne and Glenn Rowe. 2002. Difficulties in learning and teaching programming—views of students and tutors. Education and Information technologies 7 (2002), 55–66.

[22]

Marta Montenegro-Rueda, José Fernández-Cerero, José María Fernández-Batanero, and Eloy López-Meneses. 2023. Impact of the Implementation of ChatGPT in Education: A Systematic Review. Computers 12, 8 (2023), 153.

[23]

Davy Tsz Kit Ng, Jac Ka Lok Leung, Jiahong Su, Ross Chi Wui Ng, and Samuel Kai Wah Chu. 2023. Teachers’ AI digital competencies and twenty-first century skills in the post-pandemic world. Educational technology research and development 71, 1 (2023), 137–161.

[24]

Radek Pelánek. 2016. Applications of the Elo rating system in adaptive educational systems. Computers & Education 98 (2016), 169–179.

Digital Library

[25]

Jaromir Savelka, Arav Agarwal, Christopher Bogart, Yifan Song, and Majd Sakr. 2023. Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?arXiv preprint arXiv:2303.09325 (2023).

[26]

Priynka Sharma and Mayuri Harkishan. 2022. Designing an intelligent tutoring system for computer programing in the Pacific. Education and Information Technologies 27, 5 (2022), 6197–6209.

Digital Library

[27]

Jennifer L Steele. 2023. To GPT or not GPT? Empowering our students to learn with AI. Computers and Education: Artificial Intelligence 5 (2023), 100160.

[28]

Miriam Sullivan, Andrew Kelly, and Paul McLaughlan. 2023. ChatGPT in higher education: Considerations for academic integrity and student learning. (2023).

[29]

Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé F Bissyandé. 2023. Is ChatGPT the Ultimate Programming Assistant–How far is it?arXiv preprint arXiv:2304.11938 (2023).

[30]

Dwayne Towell and Brent Reeves. 2010. From Walls to Steps: Using online automatic homework checking tools to improve learning in introductory programming courses. (2010).

[31]

Zhihong Xu, Yingying Zhao, Jeffrey Liew, Xuan Zhou, and Ashlynn Kogut. 2023. Synthesizing research evidence on self-regulated learning and academic achievement in online and blended learning environments: A scoping review. Educational Research Review (2023), 100510.

[32]

Ramazan Yilmaz and Fatma Gizem Karaoglan Yilmaz. 2023. The effect of generative artificial intelligence (AI)-based tool use on students’ computational thinking skills, programming self-efficacy and motivation. Computers and Education: Artificial Intelligence (2023), 100147.

[33]

Cynthia Zastudil, Magdalena Rogalska, Christine Kapp, Jennifer Vaughn, and Stephen MacNeil. 2023. Generative AI in Computing Education: Perspectives of Students and Instructors. arXiv preprint arXiv:2308.04309 (2023).

Cited By

Arıcı F(2024)Examination of Research Conducted on the Use of Artificial Intelligence in Science EducationSakarya University Journal of Education10.19126/suje.148511414:3(543-568)Online publication date: 10-Nov-2024
https://doi.org/10.19126/suje.1485114
Humble N(2024)Risk management strategy for generative AI in computing education: how to handle the strengths, weaknesses, opportunities, and threats?International Journal of Educational Technology in Higher Education10.1186/s41239-024-00494-x21:1Online publication date: 11-Dec-2024
https://doi.org/10.1186/s41239-024-00494-x
Viberg OMutimukwe CHrastinski SCerratto‐Pargman TLilliesköld J(2024)Exploring teachers' (future) digital assessment practices in higher education: Instrument and model developmentBritish Journal of Educational Technology10.1111/bjet.1346255:6(2597-2616)Online publication date: 30-Mar-2024
https://doi.org/10.1111/bjet.13462
Show More Cited By

Index Terms

Kattis vs ChatGPT: Assessment and Evaluation of Programming Tasks in the Age of Artificial Intelligence

Recommendations

How should we change teaching and assessment in response to increasingly powerful generative Artificial Intelligence? Outcomes of the ChatGPT teacher survey
Abstract
There has been widespread media commentary about the potential impact of generative Artificial Intelligence (AI) such as ChatGPT on the Education field, but little examination at scale of how educators believe teaching and assessment should change ...
Can you learn to teach programming in two days?
ACSW '16: Proceedings of the Australasian Computer Science Week Multiconference

Between 2011 and 2013, an updated set of national standards for secondary school computer science education was introduced in New Zealand. This change caused great difficulties for many existing "computing" teachers. After many years of teaching ...
Reported Development of Computational Thinking, Through Computer Science and Programming, and its Benefits for Primary School Students: (Abstract Only)
SIGCSE '18: Proceedings of the 49th ACM Technical Symposium on Computer Science Education

Across the world there has been a paradigm shift in school education, as many countries are incorporating Computer Science (CS), programming, and Computational Thinking (CT), into their K-12 curriculums for the first time. In 2018 these subjects will be ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

LAK '24: Proceedings of the 14th Learning Analytics and Knowledge Conference

March 2024

962 pages

ISBN:9798400716188

DOI:10.1145/3636555

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution-NoDerivatives International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 March 2024

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Funding Sources

Digital Futures

Conference

LAK '24

LAK '24: The 14th Learning Analytics and Knowledge Conference

March 18 - 22, 2024

Kyoto, Japan

Acceptance Rates

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
773
Total Downloads

Downloads (Last 12 months)773
Downloads (Last 6 weeks)114

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Arıcı F(2024)Examination of Research Conducted on the Use of Artificial Intelligence in Science EducationSakarya University Journal of Education10.19126/suje.148511414:3(543-568)Online publication date: 10-Nov-2024
https://doi.org/10.19126/suje.1485114
Humble N(2024)Risk management strategy for generative AI in computing education: how to handle the strengths, weaknesses, opportunities, and threats?International Journal of Educational Technology in Higher Education10.1186/s41239-024-00494-x21:1Online publication date: 11-Dec-2024
https://doi.org/10.1186/s41239-024-00494-x
Viberg OMutimukwe CHrastinski SCerratto‐Pargman TLilliesköld J(2024)Exploring teachers' (future) digital assessment practices in higher education: Instrument and model developmentBritish Journal of Educational Technology10.1111/bjet.1346255:6(2597-2616)Online publication date: 30-Mar-2024
https://doi.org/10.1111/bjet.13462
Becerra ÁMohseni ZSanz JCobos R(2024)A Generative AI-Based Personalized Guidance Tool for Enhancing the Feedback to MOOC Learners2024 IEEE Global Engineering Education Conference (EDUCON)10.1109/EDUCON60312.2024.10578809(1-8)Online publication date: 8-May-2024
https://doi.org/10.1109/EDUCON60312.2024.10578809
Yousef MMohamed KMedhat WMohamed EKhoriba GArafa T(2024)BeGrading: large language models for enhanced feedback in programming educationNeural Computing and Applications10.1007/s00521-024-10449-yOnline publication date: 16-Oct-2024
https://doi.org/10.1007/s00521-024-10449-y

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents