More Web Proxy on the site http://driver.im/

research-article

Pensieve: Feedback on Coding Process for Novices

Authors:

Chris PiechAuthors Info & Claims

SIGCSE '19: Proceedings of the 50th ACM Technical Symposium on Computer Science Education

Pages 253 - 259

https://doi.org/10.1145/3287324.3287483

Published: 22 February 2019 Publication History

Abstract

In large undergraduate computer science classrooms, student learning on assignments is often gauged only by the work on their final solution, not by their programming process. As a consequence, teachers are unable to give detailed feedback on how students implement programming methodology, and novice students often lack a metacognitive understanding of how they learn. We introduce Pensieve as a drag-and-drop, open-source tool that organizes snapshots of student code as they progress through an assignment. The tool is designed to encourage sit-down conversations between student and teacher about the programming process. The easy visualization of code evolution over time facilitates the discussion of intermediate work and progress towards learning goals, both of which would otherwise be unapparent from a single final submission. This paper discusses the pedagogical foundations and technical details of Pensieve and describes results from a particular 207-student classroom deployment, suggesting that the tool has meaningful impacts on education for both the student and the teacher.

References

[1]

Kirsti M. Ala-Mutka. 2005. A Survey of Automated Assessment Approaches for Programming Assignments. Computer Science Education, Vol. 15, 2 (2005), 83--102.

[2]

Lorin W. Anderson and David R. Krathwohl (Eds.). 2001. A Taxonomy for Learning, Teaching, and Assessing. A Revision of Bloom's Taxonomy of Educational Objectives 2 ed.). Allyn & Bacon, New York.

[3]

Randy Elliot Bennett. 2011. Formative assessment: A critical review. Assessment in Education: Principles, Policy & Practice, Vol. 18, 1 (2011), 5--25.

[4]

Susan Bergin, Ronan Reilly, and Desmond Traynor. 2005. Examining the Role of Self-Regulated Learning on Introductory Programming Performance. In Proceedings of the first international workshop on Computing education research (ICER '05). ACM, New York, NY, USA, 81--86.

Digital Library

[5]

Paul Black and Dylan Wiliam. 2009. Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability(formerly: Journal of Personnel Evaluation in Education), Vol. 21, 1 (23 Jan 2009), 5.

[6]

David Boud and Elizabeth Molloy. 2013. Rethinking models of feedback for learning: the challenge of design. Assessment & Evaluation in Higher Education, Vol. 38, 6 (2013), 698--712.

[7]

John D. Bransford, Ann L. Brown, and Rodney R. Cocking. 2000. How People Learn .National Academy Press.

[8]

Tracey Bretag. 2013. Challenges in Addressing Plagiarism in Education. PLoS Medicine, Vol. 10, 12 (2013).

[9]

David Carless, Diane Salter, Min Yang, and Joy Lam. 2011. Developing sustainable feedback practices. Studies in Higher Education, Vol. 36, 4 (2011), 395--407.

[10]

D. Joseph Clark and Mark V. Redmond. 1982. Small group instructional diagnosis. ERIC.

[11]

Curtis Clifton, Lisa C. Kaczmarczyk, and Michael Mrozek. 2007. Subverting the Fundamentals Sequence: Using Version Control to Enhance Course Management. In Proceedings of the 38th SIGCSE technical symposium on Computer science education. ACM, 86--90.

Digital Library

[12]

Quintin Cutts, Emily Cutts, Stephen Draper, Patrick O'Donnell, and Peter Saffrey. 2010. Manipulating Mindset to Positively Influence Introductory Programming Performance. In Proceedings of the 41st ACM Technical Symposium on Computer Science Education (SIGCSE '10). ACM, New York, NY, USA, 431--435.

Digital Library

[13]

Nell B. Dale. 2006. Most difficult topics in CS1: results of an online survey of educators. ACM SIGCSE Bulletin, Vol. 38, 2 (2006), 49--53.

Digital Library

[14]

Thanasis Daradoumis, Roxana Bassi, Fatos Xhafa, and Santi Caballé. 2013. A Review on Massive E-Learning (MOOC) Design, Delivery and Assessment. 2013 Eighth International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (2013), 208--213.

Digital Library

[15]

Carol Dweck. 2008. Mindsets and Math/Science Achievement .New York: Carnegie Corporation of New York, Institute for Advanced Study, Commission on Mathematics and Science Education, New York, NY, USA.

[16]

Stephen H. Edwards and Manuel A. Perez-Quinones. 2008. Web-CAT: Automatically Grading Programming Assignments. SIGCSE Bull., Vol. 40, 3 (June 2008), 328--328.

Digital Library

[17]

Susan E. Embretson and Steven P. Reise. 2013. Item response theory .Psychology Press.

[18]

Anneli Etel"apelto. 1993. Metacognition and the Expertise of Computer Program Comprehension. Scandinavian Journal of Educational Research, Vol. 37, 3 (1993), 243--254.

[19]

Peter Ferguson. 2011. Student perceptions of quality feedback in teacher education. Assessment & Evaluation in Higher Education, Vol. 36, 1 (2011), 51--62.

[20]

Ursula Fuller, Colin G. Johnson, Tuukka Ahoniemi, Diana Cukierman, Isidoro Hernán-Losada, Jana Jackova, Essi Lahtinen, Tracy L. Lewis, Donna McGee Thompson, Charles Riedesel, and Errol Thompson. 2007. Developing a Computer Science-specific Learning Taxonomy. In Working Group Reports on ITiCSE on Innovation and Technology in Computer Science Education (ITiCSE-WGR '07). ACM, New York, NY, USA, 152--170.

Digital Library

[21]

Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Sepp"al"a. 2010. Review of Recent Systems for Automatic Assessment of Programming Assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli Calling '10). ACM, New York, NY, USA, 86--93.

Digital Library

[22]

Charles Juwah, Debra Macfarlane-Dick, Bob Matthew, David Nicol, David Ross, and Brenda Smith. 2004. Enhancing student learning through effective formative feedback. The Higher Education Academy, Vol. 140 (2004).

[23]

Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2016. Towards a Systematic Review of Automated Feedback Generation for Programming Exercises. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '16). ACM, New York, NY, USA, 41--46.

Digital Library

[24]

Oren Laadan, Jason Nieh, and Nicolas Viennot. 2010. Teaching Operating Systems Using Virtual Appliances and Distributed Version Control. In Proceedings of the 41st SIGCSE technical symposium on Computer science education. ACM, 480--484.

Digital Library

[25]

Joseph Lawrance, Seikyung Jung, and Charles Wiseman. 2013. Git on the cloud in the classroom. In Proceedings of the 44th ACM technical symposium on Computer science education. ACM, 639--644.

Digital Library

[26]

Michael J. Lee and Andrew J. Ko. 2011. Personifying Programming Tool Feedback Improves Novice Programmers' Learning. In Proceedings of the Seventh International Workshop on Computing Education Research (ICER '11). ACM, New York, NY, USA, 109--116.

Digital Library

[27]

Ying Liu, Eleni Stroulia, Kenny Wong, and Daniel German. 2004. CVS historical information to understand how students develop software. In 26th International Conference on Software Engineering. International Workshop on Mining Software Repositories (MSR), 32--36.

[28]

Dastyni Loksa, Andrew J. Ko, Will Jernigan, Alannah Oleson, Christopher J. Mendez, and Margaret M. Burnett. 2016. Programming, Problem Solving, and Self-Awareness: Effects of Explicit Guidance. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1449--1461.

Digital Library

[29]

Guillaume Marceau, Kathi Fisler, and Shriram Krishnamurthi. 2011. Mind Your Language: On Novices' Interactions with Error Messages. In Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! 2011). ACM, New York, NY, USA, 3--18.

Digital Library

[30]

David J. Nicol and Debra Macfarlan-Dick. 2006. Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, Vol. 31, 2 (2006), 199--218.

[31]

Chris Piech, Mehran Sahami, Daphne Koller, Steve Cooper, and Paulo Blikstein. 2012. Modeling how students learn to program. In Proceedings of the 43rd ACM technical symposium on Computer Science Education. ACM, 153--160.

Digital Library

[32]

James Prather, Raymond Pettit, Kayla McMurry, Alani Peters, John Homer, and Maxine Cohen. 2018. Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools. In Proceedings of the 2018 ACM Conference on International Computing Education Research (ICER '18). ACM, New York, NY, USA, 41--50.

Digital Library

[33]

Thomas W. Price, Neil C.C. Brown, Chris Piech, and Kelly Rivers. 2017. Sharing and Using Programming Log Data (Abstract Only). In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE '17). ACM, New York, NY, USA, 729--729.

Digital Library

[34]

Sarah Quinton and Teresa Smallbone. 2010. Feeding forward: using feedback to promote student reflection and learning--a teaching model. Innovations in Education and Teaching International, Vol. 47, 1 (2010), 125--135.

[35]

Karen L. Reid and Gregory V. Wilson. 2005. Learning by Doing: Introducing Version Control as a Way to Manage Student Assignments. In Proceedings of the 36th SIGCSE technical symposium on Computer science education. ACM, 272--276.

Digital Library

[36]

Kelly Rivers and Kenneth R. Koedinger. 2017. Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor. International Journal of Artificial Intelligence in Education, Vol. 27, 1 (01 Mar 2017), 37--64.

[37]

Eric Roberts, John Lilly, and Bryan Rollins. 1995. Using Undergraduates as Teaching Assistants in Introductory Programming Courses: An Update on the Stanford Experience. In Proceedings of the twenty-sixth SIGCSE technical symposium on Computer science education. ACM, 48--52.

Digital Library

[38]

Kimberly Tanner. 2012. Promoting Student Metacognition. CBE Life Sciences Education, Vol. 11, 2 (2012), 113--120.

[39]

Errol Thompson, Andrew Luxton-Reilly, Jacqueline L. Whalley, Minjie Hu, and Phil Robbins. 2008. Bloom's Taxonomy for CS Assessment. In Proceedings of the Tenth Conference on Australasian Computing Education - Volume 78 (ACE '08). Australian Computer Society, Inc., Darlinghurst, Australia, Australia, 155--161. http://dl.acm.org/citation.cfm?id=1379249.1379265

Digital Library

[40]

Peter Vamplew and Julian Dermoudy. 2005. An Anti-Plagiarism Editor for Software Development Courses. In Proceedings of the 7th Australasian Conference on Computing Education. 83--90.

Digital Library

[41]

Fabienne M. Van der Kleij, Remco C. W. Feskens, and Theo J. H. M. Eggen. 2015. Effects of Feedback in a Computer-Based Learning Environment on Students' Learning Outcomes: A Meta-Analysis. Review of Educational Research, Vol. 85, 4 (2015), 475--511.

[42]

Lisa Yan, Nick McKeown, and Chris Piech. 2019. The PyramidSnapshot Challenge: Understanding student process from visual output of programs. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE '19). ACM, New York, NY, USA.

Digital Library

[43]

Lisa Yan, Nick McKeown, Mehran Sahami, and Chris Piech. 2018. TMOSS: Using Intermediate Assignment Work to Understand Excessive Collaboration in Large Classes. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (SIGCSE '18). ACM, New York, NY, USA, 110--115.

Digital Library

Cited By

Li YChen MHunt AZhang HO'Rourke E(2024)Exploring the Interplay of Metacognition, Affect, and Behaviors in an Introductory Computer Science Course for Non-MajorsProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671119(27-41)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671119
Izu CWeeransinghe A(2024)Testing and Debugging Habits of Intermediate Student Programmers2024 IEEE Global Engineering Education Conference (EDUCON)10.1109/EDUCON60312.2024.10578650(1-10)Online publication date: 8-May-2024
https://doi.org/10.1109/EDUCON60312.2024.10578650
Xie BLim JPham PLi MKo A(2023)Developing Novice Programmers’ Self-Regulation Skills with Code ReplaysProceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600127(298-313)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600127
Show More Cited By

Index Terms

Pensieve: Feedback on Coding Process for Novices
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education
        CS1
      2. Student assessment

Recommendations

Using MOOC technology and formative assessment in a conceptual modelling course: an experience report
MODELS '18: Proceedings of the 21st ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings

Online and blended learning are employed by educators in a wide variety of fields, from literature to computer science, from medicine to astronomy. However, online and, specifically, MOOC and SPOC technologies are still not widespread in the field of ...
Learning How to Learn through Experiential Learning Promoting Metacognitive Skills to Improve Knowledge Co-creation Ability

The development of metacognitive skills is important to train people to work in the knowledge society, which requires the creation of value. However, within the limited period of university education, it is not reasonable to teach metacognitive skills ...
Flipped University Classrooms: Using Technology to Enable Sound Pedagogy

The three case studies in this paper show how flipped classroom approaches can facilitate the renewal of university teaching. The case studies form part of a scholarship of teaching and learning that provides opportunities for educators to learn from ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCSE '19: Proceedings of the 50th ACM Technical Symposium on Computer Science Education

February 2019

1364 pages

ISBN:9781450358903

DOI:10.1145/3287324

General Chairs:
Elizabeth K. Hawthorne
Union County College, USA
,
Manuel A. Pérez-Quiñones
University of North Carolina at Charlotte, USA
,
Program Chairs:
Sarah Heckman
North Carolina State University, USA
,
Jian Zhang
Texas Woman's University, USA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 February 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGCSE '19

Sponsor:

SIGCSE

SIGCSE '19: The 50th ACM Technical Symposium on Computer Science Education

February 27 - March 2, 2019

MN, Minneapolis, USA

Acceptance Rates

SIGCSE '19 Paper Acceptance Rate 169 of 526 submissions, 32%;

Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

Upcoming Conference

SIGCSE TS 2025

Sponsor:
sigcse

The 56th ACM Technical Symposium on Computer Science Education

February 26 - March 1, 2025

Pittsburgh , PA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
679
Total Downloads

Downloads (Last 12 months)63
Downloads (Last 6 weeks)5

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li YChen MHunt AZhang HO'Rourke E(2024)Exploring the Interplay of Metacognition, Affect, and Behaviors in an Introductory Computer Science Course for Non-MajorsProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671119(27-41)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671119
Izu CWeeransinghe A(2024)Testing and Debugging Habits of Intermediate Student Programmers2024 IEEE Global Engineering Education Conference (EDUCON)10.1109/EDUCON60312.2024.10578650(1-10)Online publication date: 8-May-2024
https://doi.org/10.1109/EDUCON60312.2024.10578650
Xie BLim JPham PLi MKo A(2023)Developing Novice Programmers’ Self-Regulation Skills with Code ReplaysProceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600127(298-313)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600127
Charitsis CPiech CMitchell JDoyle MStephenson BDorn BSoh LBattestilli L(2023)Detecting the Reasons for Program Decomposition in CS1 and Evaluating Their ImpactProceedings of the 54th ACM Technical Symposium on Computer Science Education V. 110.1145/3545945.3569763(1014-1020)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3545945.3569763
Sterman SNicholas MVivrekar JMindel JPaulos E(2023)Kaleidoscope: A Reflective Documentation Tool for a User Interface Design CourseProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581255(1-19)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581255
Meier HLepp M(2022)Effectiveness of Feedback Based on Log File Analysis in Introductory Programming CoursesJournal of Educational Computing Research10.1177/0735633122113265161:3(696-719)Online publication date: 25-Oct-2022
https://doi.org/10.1177/07356331221132651
Koutcheme C(2022)Towards Open Natural Language Feedback Generation for Novice Programmers using Large Language ModelsProceedings of the 22nd Koli Calling International Conference on Computing Education Research10.1145/3564721.3565955(1-2)Online publication date: 17-Nov-2022
https://dl.acm.org/doi/10.1145/3564721.3565955
Sterman SNicholas MPaulos E(2022)Towards Creative Version ControlProceedings of the ACM on Human-Computer Interaction10.1145/35557566:CSCW2(1-25)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555756
Nicholas MSterman SPaulos EMaiden NSas C(2022)Creative and Motivational Strategies Used by Expert Creative PractitionersProceedings of the 14th Conference on Creativity and Cognition10.1145/3527927.3532870(323-335)Online publication date: 20-Jun-2022
https://dl.acm.org/doi/10.1145/3527927.3532870
Sarsa SDenny PHellas ALeinonen J(2022)Automatic Generation of Programming Exercises and Code Explanations Using Large Language ModelsProceedings of the 2022 ACM Conference on International Computing Education Research - Volume 110.1145/3501385.3543957(27-43)Online publication date: 3-Aug-2022
https://dl.acm.org/doi/10.1145/3501385.3543957
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents