Cited By
View all- Sychev O(2024)Searching Questions and Learning Problems in Large Problem Banks: Constructing Tests and Assignments on the FlyComputers10.3390/computers1306014413:6(144)Online publication date: 5-Jun-2024
Generation of Code Tracing Problems from Open-Source Code
Pages 875 - 881
Abstract
When developing automatic quizzing systems and intelligent tutoring systems, significant effort has to be spent on developing the problem bank. Question and problem generation is a field of study concerning automating this routine work. The two most common methods of problem generation are constrained random generation and template-based generation, but each of them has disadvantages. In this work, we study a possibility of generating code-tracing learning problems from existing open-source code. We generated a set of learning problems and evaluated their distinctness from human-authored problems and readiness for usage in the learning process. Both teachers and students showed the rate of determining machine-generated problems only slightly above random guessing. Teachers strongly agreed with the problems' relevance and agreed with their suitability for the learning process. Automatic labeling to filter desired problems for the assignment includes used concepts, possible errors during solving, and difficulty estimates. The studied type of learning problem required little additional data to add to the code; our further work will concern problem types with more dynamic data to overcome this limitation.
Supplementary Material
This is a video presentation of the full paper "Generation of Code Tracing Problems From Open-Source Code." We describe a method of generating learning problems from program code, compare it to other methods of learning problem generation and discuss the evaluation results. Using the proposed method, we generated 22 369 learning problems for teaching control flow statements from 20 GitHub repositories which proved practically indistinguishable from human-authored problems. The problems are marked with the used concepts, the required knowledge to solve, and difficulty measures.
- Download
- 56.82 MB
References
[1]
Abejide Ade-Ibijola. 2019. Syntactic generation of practice novice programs in python. Communications in Computer and Information Science, Vol. 963, May (2019), 158--172. https://doi.org/10.1007/978-3-030-05813-5_11
[2]
Erik Andersen, Sumit Gulwani, and Zoran Popovic. 2013. A Trace-Based Framework for Analyzing and Synthesizing Educational Progressions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI '13). Association for Computing Machinery, New York, NY, USA, 773--782. https://doi.org/10.1145/2470654.2470764
[3]
David Beckett, Tim Berners-Lee, Eric Prud'hommeaux, and Gavin Carothers. 2014. RDF 1.1 Turtle. https://www.w3.org/TR/turtle/ Accessed: 2022--11-05.
[4]
Jens Bennedsen and Michael E. Caspersen. 2019. Failure Rates in Introductory Programming: 12 Years Later. ACM Inroads, Vol. 10, 2 (apr 2019), 30--36. https://doi.org/10.1145/3324888
[5]
Valerio Cosentino, Javier Luis Cánovas Izquierdo, and Jordi Cabot. 2016. Findings from GitHub: Methods, Datasets and Limitations. In 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR). IEEE, 137--141. https://ieeexplore.ieee.org/document/7832894
[6]
Yannic Fischler, Jan-Patrick Lehr, Matthäus Magnus Kiehn, and Christian Bischof. 2019. Automatic Identification of Standard Template Algorithms in Raw Loops. In Proceedings of the 6th ACM SIGPLAN International Workshop on AI-Inspired and Empirical Methods for Software Engineering on Parallel Computing Systems (Athens, Greece) (AI-SEPS 2019). Association for Computing Machinery, New York, NY, USA, 11--14. https://doi.org/10.1145/3358500.3361568
[7]
I-Han Hsiao, Peter Brusilovsky, and Sergey Sosnovsky. 2008. Web-based Parameterized Questions for Object-Oriented Programming. In Proceedings of E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2008, Curtis J. Bonk, Mimi Miyoung Lee, and Tom Reynolds (Eds.). Association for the Advancement of Computing in Education (AACE), Las Vegas, Nevada, USA, 3728--3735. https://www.learntechlib.org/p/30206
[8]
Amruth N. Kumar. 2005. Generation of Problems, Answers, Grade, and Feedback--Case Study of a Fully Automated Tutor. J. Educ. Resour. Comput., Vol. 5, 3 (sep 2005), 3--es. https://doi.org/10.1145/1163405.1163408
[9]
Amruth N. Kumar. 2015. Solving Code-Tracing Problems and Its Effect on Code-Writing Skills Pertaining to Program Semantics. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education (Vilnius, Lithuania) (ITiCSE '15). Association for Computing Machinery, New York, NY, USA, 314--319. https://doi.org/10.1145/2729094.2742587
[10]
Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari. 2020. A systematic review of automatic question generation for educational purposes. Int. J. Artif. Intell. Educ., Vol. 30, 1 (March 2020), 121--204. https://doi.org/10.1007/s40593-019-00186-y
[11]
Brent Martin and Antonija Mitrovic. 2002. Automatic Problem Generation in Constraint-Based Tutors. In Proceedings of the 6th International Conference on Intelligent Tutoring Systems (ITS '02). Springer-Verlag, Berlin, Heidelberg, 388--398. https://dl.acm.org/doi/abs/10.5555/648031.744052
[12]
T.J. McCabe. 1976. A Complexity Measure. IEEE Transactions on Software Engineering, Vol. SE-2, 4 (1976), 308--320. https://doi.org/10.1109/TSE.1976.233837
[13]
Eleanor O'Rourke, Eric Butler, Armando Díaz Tolentino, and Zoran Popović. 2019. Automatic Generation of Problems and Explanations for an Intelligent Algebra Tutor. In Lecture Notes in Computer Science. Springer International Publishing, Cham, 383--395. https://doi.org/10.1007/978-3-030-23204-7_32
[14]
Darrel A. Regier, William E. Narrow, Diana E. Clarke, Helena C. Kraemer, S. Janet Kuramoto, Emily A. Kuhl, and David J. Kupfer. 2013. DSM-5 field trials in the United States and Canada, part II: Test-retest reliability of selected categorical diagnoses. American Journal of Psychiatry, Vol. 170, 1 (2013), 59--70. https://doi.org/10.1176/appi.ajp.2012.12070999
[15]
Kanda Runapongsa Saikaew, Chanuwas Aswamenakul, and Marut Buranarach. 2014. Design and evaluation of a NoSQL database for storing and querying RDF data. KKU Engineering Journal, Vol. 41 (12 2014), 537--545. https://doi.org/10.14456/kkuenj.2014.38
[16]
Bonnie R. Rush, David C. Rankin, and Brad J. White. 2016. The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Medical Education, Vol. 16, 1 (sep 2016). https://doi.org/10.1186/s12909-016-0773-3
[17]
Harsh Shah and Amruth N. Kumar. 2002. A Tutoring System for Parameter Passing in Programming Languages. SIGCSE Bull., Vol. 34, 3 (jun 2002), 170--174. https://doi.org/10.1145/637610.544464
[18]
Adam M. Smith and Michael Mateas. 2011. Answer Set Programming for Procedural Content Generation: A Design Space Approach. IEEE Transactions on Computational Intelligence and AI in Games, Vol. 3, 3 (2011), 187--200. https://doi.org/10.1109/TCIAIG.2011.2158545
[19]
Oleg Sychev, Anton Anikin, Nikita Penskoy, Mikhail Denisov, and Artem Prokudin. 2021a. CompPrehension - Model-Based Intelligent Tutoring System on Comprehension Level. In Intelligent Tutoring Systems: 17th International Conference, ITS 2021, Virtual Event, June 7-11, 2021, Proceedings. Springer-Verlag, Berlin, Heidelberg, 52--59. https://doi.org/10.1007/978-3-030-80421-3_6
[20]
Oleg Sychev, Mikhail Denisov, and Grigory Terekhov. 2021b. How It Works: Algorithms - A Tool for Developing an Understanding of Control Structures. In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 2 (Virtual Event, Germany) (ITiCSE '21). Association for Computing Machinery, New York, NY, USA, 621--622. https://doi.org/10.1145/3456565.3460032
[21]
Oleg Sychev, Nikita Penskoy, Anton Anikin, Mikhail Denisov, and Artem Prokudin. 2021c. Improving Comprehension: Intelligent Tutoring System Explaining the Domain Rules When Students Break Them. Education Sciences, Vol. 11, 11 (2021), 26 pages. https://doi.org/10.3390/educsci11110719
[22]
Anderson Thomas, Troy Stopera, Pablo Frank-Bolton, and Rahul Simha. 2019. Stochastic Tree-Based Generation of Program-Tracing Practice Questions. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE '19). Association for Computing Machinery, New York, NY, USA, 91--97. https://doi.org/10.1145/3287324.3287492
[23]
Jim Whitehead. 2020. Spatial Layout of Procedural Dungeons Using Linear Constraints and SMT Solvers. In International Conference on the Foundations of Digital Games (Bugibba, Malta) (FDG '20). Association for Computing Machinery, New York, NY, USA, Article 101, 9 pages. https://doi.org/10.1145/3402942.3409603
Index Terms
- Generation of Code Tracing Problems from Open-Source Code
Recommendations
Routine run-time code generation
Run-time code generation (RTCG) would be used routinely if application programmers had a facility with which they could easily create their own run-time code generators, because it would offer benefits both in terms of the efficiency of the code that ...
How Novices Use LLM-based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment
Koli Calling '23: Proceedings of the 23rd Koli Calling International Conference on Computing Education ResearchAs Large Language Models (LLMs) gain in popularity, it is important to understand how novice programmers use them and the effect they have on learning to code. We present the results of a thematic analysis on a data set from 33 learners, aged 10-17, as ...
Transformers in source code generation: A comprehensive survey
AbstractTransformers have revolutionized natural language processing (NLP) and have had a huge impact on automating tasks. Recently, transformers have led to the development of powerful large language models (LLMs), which have advanced automatic code ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2023
1481 pages
ISBN:9781450394314
DOI:10.1145/3545945
- General Chairs:
- Maureen Doyle,
- Ben Stephenson,
- Program Chairs:
- Brian Dorn,
- Leen-Kiat Soh,
- Lina Battestilli
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 March 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
SIGCSE 2023
Sponsor:
SIGCSE 2023: The 54th ACM Technical Symposium on Computer Science Education
March 15 - 18, 2023
Toronto ON, Canada
Acceptance Rates
Overall Acceptance Rate 1,595 of 4,542 submissions, 35%
Upcoming Conference
SIGCSE TS 2025
- Sponsor:
- sigcse
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 140Total Downloads
- Downloads (Last 12 months)53
- Downloads (Last 6 weeks)4
Reflects downloads up to 28 Dec 2024
Other Metrics
Citations
Cited By
View all- Sychev O(2024)Searching Questions and Learning Problems in Large Problem Banks: Constructing Tests and Assignments on the FlyComputers10.3390/computers1306014413:6(144)Online publication date: 5-Jun-2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in