More Web Proxy on the site http://driver.im/

research-article

Exploring ‘reverse-tracing’ Questions as a Means of Assessing the Tracing Skill on Computer-based CS 1 Exams

Authors:

Mohammed Hassan,

Craig ZillesAuthors Info & Claims

ICER 2021: Proceedings of the 17th ACM Conference on International Computing Education Research

Pages 115 - 126

https://doi.org/10.1145/3446871.3469765

Published: 17 August 2021 Publication History

Abstract

In this paper, we perform a comparative analysis using a within-subjects ‘think-aloud’ protocol of introductory programming students solving tracing problems in both paper-based and computer-based formats. We demonstrate that, on computer-based exams with compiler/interpreter access, students can achieve significantly higher scores on tracing problems than they do on similar paper-based questions, through brute-force execution of the provided code. Furthermore, we characterize the students’ usage of machine execution as they solve computer-based tracing problems.

We, then, suggest “reverse-tracing” questions, where a block of code is provided and students must identify an input that will produce a specified output, as a potential alternative means of assessing the same skill as tracing questions on such computer-based exams. Our initial investigation suggests correctly-designed reverse-tracing problems on computer-based exams more closely track a student’s performance on similar questions in a paper-based format. In addition, we find that the thought process while solving tracing and reverse-tracing problems is similar, but not identical.

References

[1]

Joao Paulo Barros. 2018. Students’ perceptions of paper-based vs. computer-based testing in an introductory programming course. In CSEDU 2018-Proceedings of the 10th International Conference on Computer Supported Education, Vol. 2. SciTePress, 303–308.

[2]

Joao Paulo Barros, Luís Estevens, Rui Dias, Rui Pais, and Elisabete Soeiro. 2003. Using lab exams to ensure programming practice in an introductory programming course. In Proceedings of the 8th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, (ITiCSE). 16–20.

Digital Library

[3]

Roman Bednarik. 2012. Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations. International Journal of Human-Computer Studies 70, 2 (2012), 143–155.

Digital Library

[4]

John B. Biggs and K. F. Collis. 1982. Evaluating the quality of learning : the SOLO taxonomy (structure of the observed learning outcome) / John B. Biggs, Kevin F. Collis. Academic Press New York. xiii, 245 p. : pages.

[5]

Yorah Bosse and Marco Aurélio Gerosa. 2017. Why is programming so difficult to learn? Patterns of Difficulties Related to Programming Learning Mid-Stage. ACM SIGSOFT Software Engineering Notes 41, 6 (2017), 1–6.

Digital Library

[6]

Yorah Bosse, David F Redmiles, and Marco Gerosa. 2019. Connections and Influences Among Topics of Learning How to Program. In 2019 IEEE Frontiers in Education Conference (FIE). IEEE, 1–8.

Digital Library

[7]

Ruven Brooks. 1983. Towards a theory of the comprehension of computer programs. International journal of man-machine studies 18, 6 (1983), 543–554.

[8]

Alan C Bugbee Jr. 1996. The equivalence of paper-and-pencil and computer-based testing. Journal of Research on Computing in Education 28, 3 (1996), 282–299.

[9]

Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye movements in code reading: Relaxing the linear order. In 2015 IEEE 23rd International Conference on Program Comprehension. IEEE, 255–265.

Digital Library

[10]

Jacabo Carrasquel, Dennis R. Goldenson, and Philip L. Miller. 1985. Competency testing in introductory computer science: the mastery examination at Carnegie-Mellon University. In SIGCSE ’85.

[11]

Jonathan Corley, Ana Stanescu, Lewis Baumstark, and Michael C Orsega. 2020. Paper Or IDE? The Impact of Exam Format on Student Performance in a CS1 Course. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. 706–712.

Digital Library

[12]

Martha E Crosby, Jean Scholtz, and Susan Wiedenbeck. 2002. The Roles Beacons Play in Comprehension for Novice and Expert Programmers. In PPIG. Citeseer, 5.

[13]

Kathryn Cunningham, Sarah Blanchard, Barbara Ericson, and Mark Guzdial. 2017. Using tracing and sketching to solve programming problems: replicating and extending an analysis of what students draw. In Proceedings of the 2017 ACM Conference on International Computing Education Research. 164–172.

Digital Library

[14]

Kathryn Cunningham, Shannon Ke, Mark Guzdial, and Barbara Ericson. 2019. Novice rationales for sketching and tracing, and how they try to avoid it. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education. 37–43.

Digital Library

[15]

Adrienne Decker, Lauren Margulieux, and Briana Morrison. 2019. Using the SOLO Taxonomy to Understand Subgoal Labels Effect in CS1. ICER’19 - Proceedings of the 2019 ACM Conference on International Computing Education Research, 209–217. https://doi.org/10.1145/3291279.3339405

Digital Library

[16]

Ronald F. DeMara, Navid Khoshavi, Steven D. Pyle, John Edison, Richard Hartshorne, Baiyun Chen, and Michael Georgiopoulos. 2016. Redesigning Computer Engineering Gateway Courses Using a Novel Remediation Hierarchy. In 2016 ASEE Annual Conference & Exposition. ASEE Conferences, New Orleans, Louisiana. https://peer.asee.org/26063.

[17]

Françoise Détienne and Elliot Soloway. 1990. An empirically-derived control structure for the process of program understanding. International Journal of Man-Machine Studies 33, 3 (1990), 323–342.

Digital Library

[18]

K. Anders Ericsson and Herbert A. Simon. 1980. Verbal reports as data.Psychological Review 87(1980), 215 – 251. http://www.library.illinois.edu.proxy2.library.illinois.edu/proxy/go.php?url=http://search.ebscohost.com.proxy2.library.illinois.edu/login.aspx?direct=true&db=hsr&AN=521000032&site=eds-live&scope=site

[19]

Sue Fitzgerald, Beth Simon, and Lynda Thomas. 2005. Strategies that students use to trace code: an analysis based in grounded theory. In Proceedings of the first international workshop on Computing education research. 69–80.

Digital Library

[20]

Scott Grissom, Laurie Murphy, Renée McCauley, and Sue Fitzgerald. 2016. Paper vs. computer-based exams: A study of errors in recursive binary tree algorithms. In Proceedings of the 47th acm technical symposium on computing science education. 6–11.

Digital Library

[21]

Leo Gugerty and Gary Olson. 1986. Debugging by skilled and novice programmers. In Proceedings of the SIGCHI conference on human factors in computing systems. 171–174.

Digital Library

[22]

Philip J Guo. 2013. Online python tutor: embeddable web-based program visualization for cs education. In Proceeding of the 44th ACM technical symposium on Computer science education. 579–584.

[23]

Nanna Suryana Herman, Sazilah Binti Salam, Edi Noersasongko, 2011. A study of tracing and writing performance of novice students in introductory programming. In International Conference on Software Engineering and Computer Systems. Springer, 557–570.

[24]

Cruz Izu and Claudio Mirolo. 2020. Comparing small programs for equivalence: A code comprehension task for novice programmers. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education. 466–472.

Digital Library

[25]

Norman Jacobson. 2000. Using On-computer Exams to Ensure Beginning Students’ Programming Competency. SIGCSE Bull. 32, 4 (Dec. 2000), 53–56. https://doi.org/10.1145/369295.369324

Digital Library

[26]

Lisa C. Kaczmarczyk, Elizabeth R. Petrick, J. Philip East, and Geoffrey L. Herman. 2010. Identifying student misconceptions of programming.SIGCSE: Technical Symposium on Computer Science Education (2010), 107 – 111. http://www.library.illinois.edu.proxy2.library.illinois.edu/proxy/go.php?url=http://search.ebscohost.com.proxy2.library.illinois.edu/login.aspx?direct=true&db=edb&AN=73688244&site=eds-live&scope=site

[27]

Ivar Krumpal. 2013. Determinants of social desirability bias in sensitive surveys: a literature review. Quality & Quantity 47, 4 (2013), 2025–2047.

[28]

Amruth N Kumar. 2013. A study of the influence of code-tracing problems on code-writing skills. In Proceedings of the 18th ACM conference on Innovation and technology in computer science education. 183–188.

Digital Library

[29]

Amruth N Kumar. 2015. Solving code-tracing problems and its effect on code-writing skills pertaining to program semantics. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education. 314–319.

Digital Library

[30]

Vesa Lappalainen, Antti-Jussi Lakanen, and Harri Högmander. 2017. Towards computer-based exams in CS1. In CSEDU 2017: Proceedings of the 9th International Conference on Computer Supported Education. Vol. 2, ISBN 978-989-758-240-0. SCITEPRESS Science And Technology Publications.

[31]

Raymond Lister, Elizabeth S Adams, Sue Fitzgerald, William Fone, John Hamer, Morten Lindholm, Robert McCartney, Jan Erik Moström, Kate Sanders, Otto Seppälä, 2004. A multi-national study of reading and tracing skills in novice programmers. ACM SIGCSE Bulletin 36, 4 (2004), 119–150.

Digital Library

[32]

Raymond Lister, Colin Fidge, and Donna Teague. 2009. Further evidence of a relationship between explaining, tracing and writing skills in introductory programming. Acm sigcse bulletin 41, 3 (2009), 161–165.

[33]

Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. 2008. Relationships between reading, tracing and writing skills in introductory programming. In Proceedings of the fourth international workshop on computing education research. 101–112.

Digital Library

[34]

Claudio Mirolo, Cruz Izu, and Emanuele Scapin. 2020. High-school students’ mastery of basic flow-control constructs through the lens of reversibility. In Proceedings of the 15th Workshop on Primary and Secondary Computing Education. 1–10.

Digital Library

[35]

Motherboard (Tech by Vice). 2021. Schools Are Abandoning Invasive Proctoring Software After Student Backlash. https://www.vice.com/en/article/7k9ag4/schools-are-abandoning-invasive-proctoring-software-after-student-backlash.

[36]

Terence Nip, Elsa L. Gunter, Geoffrey L. Herman, Jason W. Morphew, and Matthew West. 2018. Using a Computer-Based Testing Facility to Improve Student Learning in a Programming Languages and Compilers Course. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (Baltimore, Maryland, USA) (SIGCSE ’18). Association for Computing Machinery, New York, NY, USA, 568–573. https://doi.org/10.1145/3159450.3159500

Digital Library

[37]

Nancy Pennington. 1987. Stimulus structures and mental representations in expert comprehension of computer programs. Cognitive psychology 19, 3 (1987), 295–341.

[38]

Anna Agripina Prisacari and Jared Danielson. 2017. Rethinking testing mode: Should I offer my next chemistry test on paper or computer?Computers & Education 106 (2017), 1 – 12. https://doi.org/10.1016/j.compedu.2016.11.008

[39]

replit. [n.d.]. Instant IDE: Code from your browser. https://replit.com/.

[40]

Anni Rytkönen and Venla Virtakoivu. 2019. Comparative student experiences on electronic examining in a programming course-case c. In Proceedings of the 19th Koli Calling International Conference on Computing Education Research. 1–10.

Digital Library

[41]

Carsten Schulte. 2008. Block Model: an educational model of program comprehension as a tool for a scholarly approach to teaching. In Proceedings of the Fourth international Workshop on Computing Education Research. 149–160.

Digital Library

[42]

Takayuki Sekiya and Kazunori Yamaguchi. 2013. Tracing quiz set to identify novices’ programming misconceptions. In Proceedings of the 13th Koli Calling International Conference on Computing Education Research. 87–95.

Digital Library

[43]

Ben Stephenson. 2018. An Experience Using On-Computer Programming Questions During Exams. In Proceedings of the 23rd Western Canadian Conference on Computing Education (Victoria, BC, Canada) (WCCCE ’18). Association for Computing Machinery, New York, NY, USA, Article 11, 6 pages. https://doi.org/10.1145/3209635.3209639

Digital Library

[44]

William T Tarimo, Fatima Abu Deeb, and Timothy J Hickey. 2015. Computers in the CS1 Classroom. In CSEDU (2). 67–74.

[45]

Donna Teague, Malcolm Corney, Alireza Ahadi, and Raymond Lister. 2013. A qualitative think aloud study of the early neo-piagetian stages of reasoning in novice programmers. In Proceedings of the 15th Australasian Computing Education Conference [Conferences in Research and Practice in Information Technology, Volume 136]. Australian Computer Society, 87–95.

[46]

Vesa Vainio and Jorma Sajaniemi. 2007. Factors in novice programmers’ poor tracing skills. ACM SIGCSE Bulletin 39, 3 (2007), 236–240.

Digital Library

[47]

Anne Venables, Grace Tan, and Raymond Lister. 2009. A closer look at tracing, explaining and code writing skills in the novice programmer. In Proceedings of the fifth international workshop on Computing education research workshop. 117–128.

Digital Library

[48]

Matthew West, Geoffrey L. Herman, and Craig Zilles. 2015. PrairieLearn: Mastery-based Online Problem Solving with Adaptive Scoring and Recommendations Driven by Machine Learning. In 2015 ASEE Annual Conference & Exposition. ASEE Conferences, Seattle, Washington.

[49]

Matthew West, Nathan Walters, Mariana Silva, Timothy Bretl, and Craig Zilles. 2021. Integrating Diverse Learning Tools using the PrairieLearn Platform. In Seventh SPLICE Workshop at SIGCSE 2021 (Virtual event).

[50]

Jacqueline Whalley, Raymond Lister, Errol Thompson, Tony Clear, Phil Robbins, P K Ajith Kumar, and Christine Prasad. 2006. An Australasian study of Reading and Comprehension Skills in Novice Programmers, using the Bloom and SOLO Taxonomies. Eighth Australasian Computing Education Conference (ACE2006) (2006).

Digital Library

[51]

Benjamin Xie, Greg L Nelson, and Andrew J Ko. 2018. An explicit strategy to scaffold novice program tracing. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education. 344–349.

Digital Library

[52]

C. Zilles, R. T. Deloatch, J. Bailey, B. B. Khattar, W. Fagen, C. Heeren, D Mussulman, and M. West. 2015. Computerized Testing: A Vision and Initial Experiences. In American Society for Engineering Education (ASEE) Annual Conference.

[53]

C. Zilles, M. West, D. Mussulman, and C. Sacris. 2018. Student and Instructor Experiences with a Computer-Based Testing Facility. In 10th annual International Conference on Education and New Learning Technologies (EDULEARN).

[54]

Craig B Zilles, Matthew West, Geoffrey L Herman, and Timothy Bretl. 2019. Every University Should Have a Computer-Based Testing Facility. In CSEDU (1). 414–420.

Cited By

Denny PSmith DFowler MPrather JBecker BLeinonen JMonga MLonati VBarendsen ESheard JPaterson J(2024)Explaining Code with a Purpose: An Integrated Approach for Developing Code Comprehension and Prompting SkillsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653587(283-289)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653587
Graf OThorgeirsson SSu ZMonga MLonati VBarendsen ESheard JPaterson J(2024)Assessing Live Programming for Program ComprehensionProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653547(520-526)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653547
Hassan MZeng GZilles C(2024)Evaluating How Novices Utilize Debuggers and Code Execution to Understand CodeProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671126(65-83)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671126
Show More Cited By

Exploring ‘reverse-tracing’ Questions as a Means of Assessing the Tracing Skill on Computer-based CS 1 Exams
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs

Recommendations

Improving Student Self-efficacy in CS 1 Using Examples of Erroneous Code
SIGCSE '19: Proceedings of the 50th ACM Technical Symposium on Computer Science Education

Research has shown that self-efficacy, a student's confidence in his or her ability, contributes to student success. Here, we examine if teaching students to recognize common programming errors increases student self-efficacy in an introductory ...
Oracle8i Certified Professional DBA Practice Exams
Oracle8i Certified Professional DBA Practice Exams

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICER 2021: Proceedings of the 17th ACM Conference on International Computing Education Research

August 2021

451 pages

ISBN:9781450383264

DOI:10.1145/3446871

Editors:
Amy J. Ko
University of Washington, Seattle, USA
,
Jan Vahrenhold
Westfälische Wilhelms-Universität Münster, Germany
,
Renée McCauley
College of Charleston, USA
,
Matthias Hauswirth
Università della Svizzera italiana, Switzerland

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICER 2021

Sponsor:

SIGCSE

ICER 2021: ACM Conference on International Computing Education Research

August 16 - 19, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 189 of 803 submissions, 24%

Upcoming Conference

ICER 2025

Sponsor:
sigcse

ACM Conference on International Computing Education Research

August 3 - 6, 2025

Charlottesville , VA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
245
Total Downloads

Downloads (Last 12 months)50
Downloads (Last 6 weeks)4

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Denny PSmith DFowler MPrather JBecker BLeinonen JMonga MLonati VBarendsen ESheard JPaterson J(2024)Explaining Code with a Purpose: An Integrated Approach for Developing Code Comprehension and Prompting SkillsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653587(283-289)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653587
Graf OThorgeirsson SSu ZMonga MLonati VBarendsen ESheard JPaterson J(2024)Assessing Live Programming for Program ComprehensionProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653547(520-526)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653547
Hassan MZeng GZilles C(2024)Evaluating How Novices Utilize Debuggers and Code Execution to Understand CodeProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671126(65-83)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671126
Imbulpitiya AWhalley JSenapathi M(2023)Reflections on Conducting Online Think-AloudsACM Inroads10.1145/359691714:2(26-34)Online publication date: 19-May-2023
https://dl.acm.org/doi/10.1145/3596917
Sanders KVahrenhold JMcCartney R(2023)How Do Computing Education Researchers Talk About Threats and Limitations?Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600114(381-396)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600114
Sarsa SDenny PHellas ALeinonen J(2022)Automatic Generation of Programming Exercises and Code Explanations Using Large Language ModelsProceedings of the 2022 ACM Conference on International Computing Education Research - Volume 110.1145/3501385.3543957(27-43)Online publication date: 3-Aug-2022
https://dl.acm.org/doi/10.1145/3501385.3543957
Hassan MZilles CMerkle LDoyle MSheard JSoh LDorn B(2022)On Students' Ability to Resolve their own Tracing Errors through Code ExecutionProceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 110.1145/3478431.3499400(251-257)Online publication date: 22-Feb-2022
https://dl.acm.org/doi/10.1145/3478431.3499400
Fowler MSmith IV DHassan MPoulsen SWest MZilles C(2022)Reevaluating the relationship between explaining, tracing, and writing skills in CS1 in a replication studyComputer Science Education10.1080/08993408.2022.207986632:3(355-383)Online publication date: 10-Jun-2022
https://doi.org/10.1080/08993408.2022.2079866

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents