Testing Programming Aptitude through Commonsense Computing
Author: 
Svana EscheAuthors Info & Claims
Pages 104 - 113
Abstract
Background. Programming aptitude tests are of interest since the beginning of computing education research. Many novices have no experience with programming languages before their first course. Yet they have different levels of commonsense computing.
Research Question.How successful is a commonsense computing test based on natural language as a programming aptitude test?
Method. We developed the Natural Language Computing Test (NLCT) as such a test. Our quantitative data consisted of CS1 students (N=681) who completed the NLCT during the winter 2022/23 semester. We analyzed our test with three methods. These were inter-rater agreement, item response theory, and appropriateness as predictive factor for student success.
Findings. The NLCT performed well in terms of inter-rater agreement and accuracy, according to item response theory analysis. However, the test was a weak predictor of student success as measured by correlation.
Implications. A test based solely on natural language can succeed as a programming aptitude test. Thus, a programming aptitude test need not be based on prior knowledge of programming languages or related sciences such as mathematics. However, iterative improvement of the developed test is warranted so that it can be used with less personnel effort.
References
[1]
Sattar Ameri, Mahtab J. Fard, Ratna B. Chinnam, and Chandan K. Reddy. 2016. Survival Analysis Based Framework for Early Prediction of Student Dropouts. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management(CIKM ’16). ACM, New York, NY, USA, 903–912. https://doi.org/10.1145/2983323.2983351
[2]
Glenda Barlow-Jones and Duan van der Westhuizen. 2017. Pre-entry Attributes Thought to Influence the Performance of Students in Computer Programming. In ICT Education, Janet Liebenberg and Stefan Gruner (Eds.). Springer International Publishing, Cham, 217–226.
[3]
Brett A. Becker. 2019. Parlez-vous Java? Bonjour La Monde != Hello World: Barriers to Programming Language Acquisition for Non-Native English Speakers. In 30th Workshop of the Psychology of Programming Interest Group. PPIG, Newcastle, UK, 40–52.
[4]
Susan Bergin. 2006. Statistical and machine learning models to predict programming performance. Ph. D. Dissertation. National University of Ireland Maynooth.
[5]
William J Boone, John R Staver, and Melissa S Yale. 2013. Rasch analysis in the human sciences. Springer, Dordrecht.
[6]
Pat Byrne and Gerry Lyons. 2001. The Effect of Student Attributes on Success in Programming. In Proceedings of the 6th Annual Conference on Innovation and Technology in Computer Science Education(ITiCSE ’01). ACM, New York, NY, USA, 49–52. https://doi.org/10.1145/377435.377467
[7]
Tzu-Yi Chen, Gary Lewandowski, Robert McCartney, Kate Sanders, and Beth Simon. 2007. Commonsense Computing: Using Student Sorting Abilities to Improve Instruction. In Proceedings of the 38th SIGCSE Technical Symposium on Computer Science Education(SIGCSE ’07). ACM, New York, NY, USA, 276–280. https://doi.org/10.1145/1227310.1227408
[8]
Rodrigo Silva Duran, Jan-Mikael Rybicki, Arto Hellas, and Sanna Suoranta. 2019. Towards a Common Instrument for Measuring Prior Programming Knowledge. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education(ITiCSE ’19). ACM, New York, NY, USA, 443–449. https://doi.org/10.1145/3304221.3319755
[9]
Gerald E. Evans and Mark G. Simkin. 1989. What Best Predicts Computer Proficiency?Commun. ACM 32, 11 (nov 1989), 1322–1327. https://doi.org/10.1145/68814.68817
[10]
Joseph F. Jr. Hair, William C. Black, and Rolph E. Babin, Barry J. Anderson. 2019. Multivariate data analysis (8 ed.). Cengage, Boston.
[11]
James Harris. 2014. Testing Programming Aptitude in Introductory Programming Courses. J. Comput. Sci. Coll. 30, 2 (dec 2014), 149–156.
[12]
Arto Hellas, Petri Ihantola, Andrew Petersen, Vangel V. Ajanovski, Mirela Gutica, Timo Hynninen, Antti Knutas, Juho Leinonen, Chris Messom, and Soohyun Nam Liao. 2018. Predicting Academic Performance: A Systematic Literature Review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education(ITiCSE 2018 Companion). ACM, New York, NY, USA, 175–199. https://doi.org/10.1145/3293881.3295783
[13]
Vivien Heller and Miriam Morek. 2015. Academic discourse as situated practice: An introduction. Linguistics and Education 31 (09 2015), 174–186.
[14]
Geoffrey L. Herman, Lisa Kaczmarczyk, Michael C. Loui, and Craig Zilles. 2008. Proof by Incomplete Enumeration and Other Logical Misconceptions. In Proceedings of the Fourth International Workshop on Computing Education Research(ICER ’08). ACM, New York, NY, USA, 59–70. https://doi.org/10.1145/1404520.1404527
[15]
Diane Horton and Michelle Craig. 2015. Drop, Fail, Pass, Continue: Persistence in CS1 and Beyond in Traditional and Inverted Delivery. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education(SIGCSE ’15). ACM, New York, NY, USA, 235–240. https://doi.org/10.1145/2676723.2677273
[16]
Juha Huoman. 1986. Ohjelmointitaidon mittaaminen. Master’s thesis. Department of Computer Science, University of Joensuu.
[17]
Päivi Kinnunen, Maija Marttila-Kontio, and Erkki Pesonen. 2013. Getting to Know Computer Science Freshmen. In Proceedings of the 13th Koli Calling International Conference on Computing Education Research(Koli Calling ’13). ACM, New York, NY, USA, 59–66. https://doi.org/10.1145/2526968.2526975
[18]
Klaus Krippendorff. 2022. The Reliability of Generating Data. Chapman and Hall/CRC, Boca Raton, FL, USA. https://doi.org/10.1201/9781003112020
[19]
José Paulo Leal. 2013. Testing the perception of time, state and causality to predict programming aptitude. In 2013 Federated Conference on Computer Science and Information Systems. IEEE, Krakow, Poland, 721–726.
[20]
R. R. Leeper and J. L. Silver. 1982. Predicting Success in a First Programming Course. In Proceedings of the Thirteenth SIGCSE Technical Symposium on Computer Science Education(SIGCSE ’82). ACM, New York, NY, USA, 147–150. https://doi.org/10.1145/800066.801357
[21]
Raymond Lister, Elizabeth S. Adams, Sue Fitzgerald, William Fone, John Hamer, Morten Lindholm, Robert McCartney, Jan Erik Moström, Kate Sanders, Otto Seppälä, Beth Simon, and Lynda Thomas. 2004. A Multi-National Study of Reading and Tracing Skills in Novice Programmers. In Working Group Reports from ITiCSE on Innovation and Technology in Computer Science Education(ITiCSE-WGR ’04). ACM, New York, NY, USA, 119–150. https://doi.org/10.1145/1044550.1041673
[22]
Raymond Lister, Beth Simon, Errol Thompson, Jacqueline L. Whalley, and Christine Prasad. 2006. Not Seeing the Forest for the Trees: Novice Programmers and the SOLO Taxonomy. In Proceedings of the 11th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education(ITICSE ’06). ACM, New York, NY, USA, 118–122. https://doi.org/10.1145/1140124.1140157
[23]
Torben Lorenzen and Hang-Ling Chang. 2006. MasterMind©: A Predictor of Computer Programming Aptitude. SIGCSE Bull. 38, 2 (jun 2006), 69–71. https://doi.org/10.1145/1138403.1138436
[24]
Lauren Margulieux, Tuba Ayer Ketenci, and Adrienne Decker. 2019. Review of measurements used in computing education research and suggestions for increasing standardization. Computer Science Education 29, 1 (2019), 49–78. https://doi.org/10.1080/08993408.2018.1562145
[25]
Richard E. Mayer, Jennifer L. Dyck, and William Vilberg. 1986. Learning to Program and Learning to Think: What’s the Connection?Commun. ACM 29, 7 (jul 1986), 605–610. https://doi.org/10.1145/6138.6142
[26]
Merriam-Webster.com Dictionary. 2023. “aptitude test”. https://www.merriam-webster.com/dictionary/aptitude_test
[27]
Insu Paek and Ki Cole. 2020. Using R for item response theory model applications. Taylor & Francis, Abingdon, Oxon.
[28]
Branimir Pejcinovic, Melinda Holtzman, Phillip K Wong, and Gerald Recktenwald. 2017. Assessing student preparedness for introductory engineering and programming courses. In 2017 IEEE Frontiers in Education Conference (FIE). IEEE, Indianapolis, IN, USA, 1–5. https://doi.org/10.1109/FIE.2017.8190539
[29]
Yizhou Qian and James D Lehman. 2016. Correlates of success in introductory programming: A study with middle school students.Journal of Education and Learning 5, 2 (2016), 73–83.
[30]
Keith Quille, Soohyun Nam Liao, Eileen Costelloe, Keith Nolan, Aidan Mooney, and Kartik Shah. 2022. PreSS: Predicting Student Success Early in CS1. A Pilot International Replication and Generalization Study. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1(ITiCSE ’22). ACM, New York, NY, USA, 54–60. https://doi.org/10.1145/3502718.3524755
[31]
Shanon Marie Reckinger. 2016. Implementation and evaluation of different types of peer learning instruction in a MATLAB programming course. In 2016 ASEE Annual Conference & Exposition. American Society for Engineering Education, New Orleans, LA, USA. https://doi.org/10.18260/p.25561
[32]
Jeff Ringenberg, Marcial Lapp, Apoorva Bansal, and Parth Shah. 2011. The Programming Performance Prophecies: Predicting Student Achievement in a First-Year Introductory Programming Course. In 2011 ASEE Annual Conference & Exposition. ASEE Conferences. https://doi.org/10.18260/1-2–18930
[33]
T. C. Rowan. 1957. Psychological Tests and Selection of Computer Programmers. J. ACM 4, 3 (jul 1957), 348–353. https://doi.org/10.1145/320881.320891
[34]
Robert M Siegfried, Katherine G Herbert-Berger, Kees Leune, and Jason P Siegfried. 2021. Trends Of Commonly Used Programming Languages in CS1 And CS2 Learning. In 2021 16th International Conference on Computer Science & Education (ICCSE). IEEE, Lancaster, UK, 407–412. https://doi.org/10.1109/ICCSE51940.2021.9569444
[35]
Guttorm Sindre. 2020. Code Writing vs Code Completion Puzzles: Analyzing Questions in an E-exam. In 2020 IEEE Frontiers in Education Conference (FIE). IEEE, Uppsala, Sweden, 1–9. https://doi.org/10.1109/FIE44824.2020.9273919
[36]
David H. Smith, Qiang Hao, Filip Jagodzinski, Yan Liu, and Vishal Gupta. 2019. Quantifying the Effects of Prior Knowledge in Entry-Level Programming Courses. In Proceedings of the ACM Conference on Global Computing Education(CompEd ’19). ACM, New York, NY, USA, 30–36. https://doi.org/10.1145/3300115.3309503
[37]
Xiaodan Tang, Yue Yin, Qiao Lin, Roxana Hadad, and Xiaoming Zhai. 2020. Assessing computational thinking: A systematic review of empirical studies. Computers & Education 148 (2020), 103798.
[38]
Markku Tukiainen and Eero Mönkkönen. 2002. Programming Aptitude Testing as a Prediction of Learning to Program. In 14th Workshop of the Psychology of Programming Interest Group. PPIG, London, UK, 47–57.
[39]
Tammy VanDeGrift, Dennis Bouvier, Tzu-Yi Chen, Gary Lewandowski, Robert McCartney, and Beth Simon. 2010. Commonsense Computing (Episode 6): Logic is Harder than Pie. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research(Koli Calling ’10). ACM, New York, NY, USA, 76–85. https://doi.org/10.1145/1930464.1930479
[40]
James V. Wertsch. 1990. Dialogue and dialogism in a socio-cultural approach to mind. In The dynamics of dialogue, Klaus Marková, Ivana; Foppa (Ed.). Harvester Wheatsheaf, New York, London, 62–82.
[41]
Stephen G. West, Wei Wu, Daniel McNeish, and Andrea Savord. 2023. Model Fit in structural Equation Modeling. In Handbook of structural equation modeling (second ed.), Rick H. Hoyle (Ed.). Guilford Publications, New York, 184–205.
[42]
Chris Wilcox and Albert Lionelle. 2018. Quantifying the Benefits of Prior Programming Experience in an Introductory Computer Science Course. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education(SIGCSE ’18). ACM, New York, NY, USA, 80–85. https://doi.org/10.1145/3159450.3159480
[43]
Wendy M Yen. 1984. Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement 8, 2 (1984), 125–145.
Index Terms
- Testing Programming Aptitude through Commonsense Computing
Recommendations
Perspectives on testing for programming aptitude
ACM '71: Proceedings of the 1971 26th annual conferenceThis paper discusses limitations of programming aptitude tests. The use of multiple-choice type questions, the testwiseness of the college graduate group, and the inclusion of questions of mathematical information tend to diminish the effectiveness of ...
Mental models and programming aptitude
Proceedings of the 12th annual SIGCSE conference on Innovation and technology in computer science education (ITiCSE'07)Predicting the success of students participating in introductory programming courses has been an active research area for more than 25 years. Until recently, no variables or tests have had any significant predictive power. However, Dehnadi and Bornat ...
Mental models and programming aptitude
ITiCSE '07: Proceedings of the 12th annual SIGCSE conference on Innovation and technology in computer science educationPredicting the success of students participating in introductory programming courses has been an active research area for more than 25 years. Until recently, no variables or tests have had any significant predictive power. However, Dehnadi and Bornat ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 29 January 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ACE 2024
ACE 2024: Australian Computing Education Conference
January 29 - February 2, 2024
NSW, Sydney, Australia
Acceptance Rates
Overall Acceptance Rate 161 of 359 submissions, 45%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 28Total Downloads
- Downloads (Last 12 months)28
- Downloads (Last 6 weeks)0
Reflects downloads up to 16 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format