More Web Proxy on the site http://driver.im/

research-article

Automated Assessment: Experiences From the Trenches

Authors:

Andrew Luxton-Reilly,

Nalin Arachchilage,

Nasser Giacaman,

Igor Kontorovich,

Danielle Lottridge,

Sathiamoorthy Manoharan,

Shyamli Sindhwani,

Paramvir Singh,

Ulrich Speidel,

Sudeep Stephen,

Valerio Terragni,

Jacqueline Whalley,

Burkhard Wuensche,

Xinfeng YeAuthors Info & Claims

ACE '23: Proceedings of the 25th Australasian Computing Education Conference

Pages 1 - 10

https://doi.org/10.1145/3576123.3576124

Published: 30 January 2023 Publication History

Abstract

Automated assessment is commonly used across the spectrum of computing courses offered by Tertiary institutions. Such assessment is frequently intended to address the scalability of feedback that is essential for learning, and assessment for accreditation purposes. Although many reviews of automated assessment have been reported, the voices of teachers are not present. In this paper we present a variety of cases that illustrate some of the varied motivations and experiences of teaching using automated assessment.

References

[1]

Kirsti M Ala-Mutka. 2005. A Survey of Automated Assessment Approaches for Programming Assignments. Computer Science Education 15, 2 (2005), 83–102.

[2]

Elisa Baniassad, Lucas Zamprogno, Braxton Hall, and Reid Holmes. 2021. STOP THE (AUTOGRADER) INSANITY: Regression Penalties to Deter Autograder Overreliance. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (Virtual Event, USA) (SIGCSE ’21). ACM, New York, NY, USA, 1062–1068. https://doi.org/10.1145/3408877.3432430

Digital Library

[3]

Leopold Bayerlein. 2014. Students’ feedback preferences: how do students react to timely and automatically generated assessment feedback?Assessment & Evaluation in Higher Education 39, 8 (2014), 916–931.

[4]

Janet Carter, Kirsti Ala-Mutka, Ursula Fuller, Martin Dick, John English, William Fone, and Judy Sheard. 2003. How Shall We Assess This?SIGCSE Bull. 35, 4 (June 2003), 107–123. https://doi.org/10.1145/960492.960539

Digital Library

[5]

Sophie H. Cormack, Laurence A. Eagle, and Mark S. Davies. 2020. A large-scale test of the relationship between procrastination and performance using learning analytics. Assessment & Evaluation in Higher Education 45, 7 (2020), 1046–1059.

[6]

Galina Deeva, Daria Bogdanova, Estefanía Serral, Monique Snoeck, and Jochen De Weerdt. 2021. A review of automated feedback systems for learners: Classification framework, challenges and opportunities. Computers & Education 162 (2021), 104094. https://doi.org/10.1016/j.compedu.2020.104094

[7]

Paul Denny, Jacqueline Whalley, and Juho Leinonen. 2021. Promoting Early Engagement with Programming Assignments Using Scheduled Automated Feedback. In Australasian Computing Education Conference(Virtual, SA, Australia) (ACE ’21). ACM, New York, NY, USA, 88–95. https://doi.org/10.1145/3441636.3442309

Digital Library

[8]

Christopher Douce, David Livingstone, and James Orwell. 2005. Automatic test-based assessment of programming: A review. Journal on Educational Resources in Computing (JERIC) 5, 3 (2005), 1–14.

[9]

Stephen H. Edwards, Joshua Martin, and Clfford A. Shaffer. 2015. Examining Classroom Interventions to Reduce Procrastination. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education(Vilnius, Lithuania) (ITiCSE ’15). ACM, New York, NY, USA, 254–259.

Digital Library

[10]

Emily Faulconer, John Griffith, and Amy Gruss. 2022. The impact of positive feedback on student outcomes and perceptions. Assessment & Evaluation in Higher Education 47, 2 (2022), 259–268. https://doi.org/10.1080/02602938.2021.1910140

[11]

Graham Gibbs and Claire Simpson. 2005. Conditions under which assessment supports students’ learning. Learning and teaching in higher education1 (2005), 3–31.

[12]

Yoshiko Goda, Masanori Yamada, Hiroshi Kato, Takeshi Matsuda, Yutaka Saito, and Hiroyuki Miyagawa. 2015. Procrastination and other learning behavioral types in e-learning and their relationship with learning outcomes. Learning and Individual Differences 37 (2015), 72–80. https://doi.org/10.1016/j.lindif.2014.11.001

[13]

John Hattie and Helen Timperley. 2007. The Power of Feedback. Review of Educational Research 77, 1 (2007), 81–112. https://doi.org/10.3102/003465430298487

[14]

Fiona Hyland and Ken Hyland. 2001. Sugaring the pill: Praise and criticism in written feedback. Journal of Second Language Writing 10, 3 (2001), 185–212. https://doi.org/10.1016/S1060-3743(01)00038-8

[15]

Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010. Review of Recent Systems for Automatic Assessment of Programming Assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research(Koli, Finland) (Koli Calling ’10). ACM, New York, NY, USA, 86–93. https://doi.org/10.1145/1930464.1930480

Digital Library

[16]

Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018. A systematic literature review of automated feedback generation for programming exercises. ACM Transactions on Computing Education (TOCE) 19, 1 (2018), 1–43.

Digital Library

[17]

Eun Jung Kim and Kyeong Ryong Lee. 2019. Effects of an examiner’s positive and negative feedback on self-assessment of skill performance, emotional response, and self-efficacy in Korea: a quasi-experimental study. BMC medical education 19, 1 (2019), 1–7.

[18]

George Kinnear, Ian Jones, Chris Sangwin, Maryam Alarfaj, Ben Davies, Sam Fearn, Colin Foster, André Heck, Karen Henderson, Tim Hunt, Paola Iannone, Igor’ Kontorovich, Niclas Larson, Tim Lowe, John Christopher Meyer, Ann O’Shea, Peter Rowlett, Indunil Sikurajapathi, and Thomas Wong. 2022. A collaboratively-derived research agenda for e-assessment in undergraduate mathematics. Int. J. Res. Undergrad. Math. Ed. (8 2022), 31 pages. https://doi.org/10.1007/s40753-022-00189-6

[19]

Richard Lobb and Jenny Harlow. 2016. Coderunner: A Tool for Assessing Computer Programming Skills. ACM Inroads 7, 1 (feb 2016), 47–51. https://doi.org/10.1145/2810041

Digital Library

[20]

Andrew Luxton-Reilly, Simon, Ibrahim Albluwi, Brett A. Becker, Michail Giannakos, Amruth N. Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, and Claudia Szabo. 2018. Introductory Programming: A Systematic Literature Review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (Larnaca, Cyprus) (ITiCSE 2018 Companion). ACM, New York, NY, USA, 55–106. https://doi.org/10.1145/3293881.3295779

Digital Library

[21]

Sathiamoorthy Manoharan and Ulrich Speidel. 2020. Individualized Assessments using Dividni-Enhancing Learning via Assessments Unique to Every Student. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. ACM, New York, NY, USA, 1419–1419. https://doi.org/10.1145/3328778.3372545

Digital Library

[22]

Emma Mulliner and Matthew Tucker. 2017. Feedback on feedback practice: perceptions of students and academics. Assessment & Evaluation in Higher Education 42, 2 (2017), 266–288. https://doi.org/10.1080/02602938.2015.1103365

[23]

David J. Nicol and Debra Macfarlane-Dick. 2006. Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education 31, 2 (2006), 199–218. https://doi.org/10.1080/03075070600572090

[24]

Claudia Ott, Anthony Robins, and Kerry Shephard. 2016. Translating Principles of Effective Feedback for Students into the CS1 Context. ACM Trans. Comput. Educ. 16, 1, Article 1 (jan 2016), 27 pages. https://doi.org/10.1145/2737596

Digital Library

[25]

José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. 2022. Automated Assessment in Computer Science Education: A State-of-the-Art Review. ACM Trans. Comput. Educ. 22, 3, Article 34 (jun 2022), 40 pages. https://doi.org/10.1145/3513140

Digital Library

[26]

Chris Sangwin. 2013. Computer aided assessment of mathematics. Oxford University Press.

[27]

Draylson M. Souza, Katia R. Felizardo, and Ellen F. Barbosa. 2016. A Systematic Literature Review of Assessment Tools for Programming Assignments. In 2016 IEEE 29th International Conference on Software Engineering Education and Training (CSEET). IEEE, 147–156. https://doi.org/10.1109/CSEET.2016.48

[28]

Jacqueline L. Whalley and Anne Philpott. 2011. A Unit Testing Approach to Building Novice Programmers’ Skills and Confidence. In Proceedings of the 13th Australasian Computing Education Conference (Perth, Australia) (ACE ’11). Australian Computer Society, Inc., Australia, 113–118.

[29]

Benedikt Wisniewski, Klaus Zierer, and John Hattie. 2020. The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research. Frontiers in Psychology 10 (2020). https://doi.org/10.3389/fpsyg.2019.03087

[30]

Shulin Yu, Feng Geng, Chunhong Liu, and Yao Zheng. 2021. What works may hurt: The negative side of feedback in second language writing. Journal of Second Language Writing 54 (2021), 100850. https://doi.org/10.1016/j.jslw.2021.100850

Cited By

Hooper SWünsche BDenny PLuxton-Reilly AKonings NCampbell AStone JYuen TShoop LRebelsky SPrather J(2025)Educator Experiences with Automated Marking of Programming Assessments in a Computer Graphics-based Design CourseProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 110.1145/3641554.3701969(520-526)Online publication date: 12-Feb-2025
https://dl.acm.org/doi/10.1145/3641554.3701969
Niu JWong JLake CRahardjo JZarkoob HOla OBelleville PMochetti KAllen MMoosvi FWolfman SStone JYuen TShoop LRebelsky SPrather J(2025)Expanding the Horizons of Autograding: Innovative Questions at UBCProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 110.1145/3641554.3701892(868-874)Online publication date: 12-Feb-2025
https://dl.acm.org/doi/10.1145/3641554.3701892
Pereira Cipriano BSilva MCorreia RAlves P(2024)Towards the Integration of Large Language Models and Automatic Assessment Tools: Enhancing Student Support in Programming AssignmentsProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699588(1-2)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699588
Show More Cited By

Index Terms

Automated Assessment: Experiences From the Trenches
1. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

Automated Assessment in Computer Science Education: A State-of-the-Art Review

Practical programming competencies are critical to the success in computer science (CS) education and go-to-market of fresh graduates. Acquiring the required level of skills is a long journey of discovery, trial and error, and optimization seeking through ...
Feedback on Student Programming Assignments: Teaching Assistants vs Automated Assessment Tool
Koli Calling '23: Proceedings of the 23rd Koli Calling International Conference on Computing Education Research

Existing research does not quantify and compare the differences between automated and manual assessment in the context of feedback on programming assignments. This makes it hard to reason about the effects of adopting automated assessment at the expense ...
Improving Student Learning with Automated Assessment
ITiCSE 2024: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1

Students taking computing courses develop skills by applying programming to various problems. In the past decade, courses have started to move from manually graded assessments to those which can be automated. A typical motivation for the use of automated ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ACE '23: Proceedings of the 25th Australasian Computing Education Conference

January 2023

139 pages

ISBN:9781450399418

DOI:10.1145/3576123

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 January 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ACE '23

ACE '23: Australasian Computing Education Conference

January 30 - February 3, 2023

VIC, Melbourne, Australia

Acceptance Rates

Overall Acceptance Rate 161 of 359 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
219
Total Downloads

Downloads (Last 12 months)82
Downloads (Last 6 weeks)6

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hooper SWünsche BDenny PLuxton-Reilly AKonings NCampbell AStone JYuen TShoop LRebelsky SPrather J(2025)Educator Experiences with Automated Marking of Programming Assessments in a Computer Graphics-based Design CourseProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 110.1145/3641554.3701969(520-526)Online publication date: 12-Feb-2025
https://dl.acm.org/doi/10.1145/3641554.3701969
Niu JWong JLake CRahardjo JZarkoob HOla OBelleville PMochetti KAllen MMoosvi FWolfman SStone JYuen TShoop LRebelsky SPrather J(2025)Expanding the Horizons of Autograding: Innovative Questions at UBCProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 110.1145/3641554.3701892(868-874)Online publication date: 12-Feb-2025
https://dl.acm.org/doi/10.1145/3641554.3701892
Pereira Cipriano BSilva MCorreia RAlves P(2024)Towards the Integration of Large Language Models and Automatic Assessment Tools: Enhancing Student Support in Programming AssignmentsProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699588(1-2)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699588
Hooper SWünsche BMarks SLuxton-Reilly ADenny P(2024)Projects and Portfolios — An Educator's Reflections on the Summative Assessments in a Game Programming CourseSIGGRAPH Asia 2024 Educator's Forum10.1145/3680533.3697071(1-8)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680533.3697071
Soares JBrito MGonçalves C(2023)Codeflex 2.0Internet of Behaviors Implementation in Organizational Contexts10.4018/978-1-6684-9039-6.ch003(40-67)Online publication date: 1-Nov-2023
https://doi.org/10.4018/978-1-6684-9039-6.ch003

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten