More Web Proxy on the site http://driver.im/

research-article

Automatic Inference of Programming Performance and Experience from Typing Patterns

Authors:

Arto VihavainenAuthors Info & Claims

SIGCSE '16: Proceedings of the 47th ACM Technical Symposium on Computing Science Education

Pages 132 - 137

https://doi.org/10.1145/2839509.2844612

Published: 17 February 2016 Publication History

Abstract

Studies on retention and success in introductory programming course have suggested that previous programming experience contributes to students' course outcomes. If such background information could be automatically distilled from students' working process, additional guidance and support mechanisms could be provided even to those, who do not wish to disclose such information. In this study, we explore methods for automatically distinguishing novice programmers from more experienced programmers using fine-grained source code snapshot data. We approach the issue by partially replicating a previous study that used students' keystroke latencies as a proxy to introductory programming course outcomes, and follow this by an exploration of machine learning methods to separate those students with little to no previous programming experience from those with more experience. Our results confirm that students' keystroke latencies can be used as a metric for measuring course outcomes. At the same time, our results show that students programming experience can be identified to some extent from keystroke latency data, which means that such data has potential as a source of information for customizing the students' learning experience.

References

[1]

A. Ahadi and R. Lister. Geek genes, prior knowledge, stumbling points and learning edge momentum: Parts of the one elephant? In Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research, ICER '13, pages 123--128, New York, NY, USA, 2013. ACM.

Digital Library

[2]

A. Ahadi, R. Lister, H. Haapala, and A. Vihavainen. Exploring machine learning methods to automatically identify students in need of assistance. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research, ICER '15, pages 121--130, New York, NY, USA, 2015. ACM.

Digital Library

[3]

S. Bergin and R. Reilly. Programming: factors that influence success. ACM SIGCSE Bull., 37(1):411--415, 2005.

Digital Library

[4]

R. Bixler and S. D'Mello. Detecting boredom and engagement during writing with keystroke analysis, task appraisals, and stable traits. In Proceedings of the 2013 International Conference on Intelligent User Interfaces, IUI '13, pages 225--234, New York, NY, USA, 2013. ACM.

Digital Library

[5]

G. D. Borich and M. L. Tombari. Educational Psychology: A Contemporary Approach. Longman Publishing/Addison Wesley, 2nd edition, 1997.

[6]

B. Cantwell Wilson and S. Shrock. Contributing to success in an introductory computer science course: a study of twelve factors. In ACM SIGCSE Bull., volume 33, pages 184--188. ACM, 2001.

Digital Library

[7]

A. S. Carter, C. D. Hundhausen, and O. Adesope. The normalized programming state model: Predicting student performance in computing courses based on programming behavior. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research, ICER '15, pages 141--150, New York, NY, USA, 2015. ACM.

Digital Library

[8]

P. Dowland and S. Furnell. A long-term trial of keystroke profiling using digraph, trigraph and keyword latencies. In Y. Deswarte, F. Cuppens, S. Jajodia, and L. Wang, editors, Security and Protection in Information Processing Systems, volume 147 of IFIP - The International Federation for Information Processing, pages 275--289. Springer, 2004.

[9]

C. Epp, M. Lippold, and R. L. Mandryk. Identifying emotional states using keystroke dynamics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '11, pages 715--724, New York, NY, USA, 2011. ACM.

Digital Library

[10]

G. E. Evans and M. G. Simkin. What best predicts computer proficiency? Comm. of the ACM, 32(11):1322--1327, 1989.

Digital Library

[11]

R. S. Gaines, W. Lisowski, S. J. Press, and N. Shapiro. Authentication by keystroke timing: Some preliminary results. Technical report, 1980.

[12]

D. Hagan and S. Markham. Does it help to have some programming experience before beginning a computing degree program? ACM SIGCSE Bull., 32(3):25--28, 2000.

Digital Library

[13]

M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten. The weka data mining software: an update. ACM SIGKDD explorations newsletter, 11(1):10--18, 2009.

Digital Library

[14]

M. C. Jadud. Methods and tools for exploring novice compilation behaviour. In Proceedings of the Second International Workshop on Computing Education Research, ICER '06, pages 73--84, New York, NY, USA, 2006. ACM.

Digital Library

[15]

M. Karnan, M. Akila, and N. Krishnaraj. Biometric personal authentication using keystroke dynamics: A review. Applied Soft Computing, 11(2):1565--1573, 2011. The Impact of Soft Computing for the Progress of Artificial Intelligence.

Digital Library

[16]

K. S. Killourhy and R. A. Maxion. Free vs. transcribed text for keystroke-dynamics evaluations. In Proceedings of the 2012 Workshop on Learning from Authoritative Security Experiment Results, LASER '12, pages 1--8, New York, NY, USA, 2012. ACM.

Digital Library

[17]

J. Kurhila and A. Vihavainen. Management, structures and tools to scale up personal advising in large programming courses. In Proceedings of the 2011 Conference on Information Technology Education, SIGITE '11, pages 3--8, New York, NY, USA, 2011. ACM.

Digital Library

[18]

K. Longi, J. Leinonen, H. Nygren, J. Salmi, A. Klami, and A. Vihavainen. Identification of programmers from typing patterns. In Proceedings of the 15th Koli Calling Conference on Computing Education Research, Koli Calling '15, pages 60--67, New York, NY, USA, 2015. ACM.

Digital Library

[19]

A. Peacock, X. Ke, and M. Wilkerson. Typing patterns: A key to user identification. IEEE Security & Privacy, 2(5):40--47, 2004.

Digital Library

[20]

D. M. Powers. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. 2011.

[21]

C. Romero and S. Ventura. Educational data mining: A review of the state of the art. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 40(6):601--618, Nov 2010.

Digital Library

[22]

T. C. Rowan. Psychological tests and selection of computer programmers. J. ACM, 4(3):348--353, 1957.

Digital Library

[23]

M. Rybnik, M. Tabedzki, and K. Saeed. A keystroke dynamics based system for user identification. In Computer Information Systems and Industrial Management Applications, 2008., pages 225--230, June 2008.

Digital Library

[24]

R. C. Thomas, A. Karahasanovic, and G. E. Kennedy. An investigation into keystroke latency metrics as an indicator of programming performance. In Proceedings of the 7th Australasian Conference on Computing Education - Volume 42, ACE '05, pages 127--134, Darlinghurst, Australia, 2005. Australian Computer Society, Inc.

Digital Library

[25]

A. Vihavainen, T. Vikberg, M. Luukkainen, and M. Pärtel. Scaffolding students' learning using test my code. In Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE '13, pages 117--122, New York, NY, USA, 2013. ACM.

Digital Library

[26]

M. Villani, C. Tappert, G. Ngo, J. Simone, H. Fort, and S.-H. Cha. Keystroke biometric recognition studies on long-text input under ideal and application-oriented conditions. In Computer Vision and Pattern Recognition Workshop, 2006. CVPRW '06. Conference on, pages 39--39, June 2006.

Digital Library

[27]

L. M. Vizer, L. Zhou, and A. Sears. Automated stress detection using keystroke and linguistic features: An exploratory study. Int. J. Hum.-Comput. Stud., 67(10):870--886, Oct. 2009.

Digital Library

[28]

C. Watson, F. W. Li, and J. L. Godwin. No tests required: comparing traditional and dynamic predictors of programming success. In Proceedings of the 45th ACM technical symposium on Computer science education, pages 469--474. ACM, 2014.

Digital Library

[29]

C. Watson, F. W. B. Li, and J. L. Godwin. Predicting performance in an introductory programming course by logging and analyzing student programming behavior. In Proceedings of the 2013 IEEE 13th International Conference on Advanced Learning Technologies, ICALT '13, pages 319--323, Washington, DC, USA, 2013. IEEE Computer Society.

Digital Library

[30]

S. Wiedenbeck, D. Labelle, and V. N. Kain. Factors affecting course outcomes in introductory programming. In 16th Annual Workshop of the Psychology of Programming Interest Group, pages 97--109, 2004.

Cited By

Sheard JDenny PHellas ALeinonen JMalmi LSimon Stephenson BStone JBattestilli LRebelsky SShoop L(2024)Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview StudyProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630880(1223-1229)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630880
Urry JEdwards JStephenson BStone JBattestilli LRebelsky SShoop L(2024)A Framework that Explores the Cognitive Load of CS1 Assignments Using Pausing BehaviorProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630760(1347-1353)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630760
Mosquera JIturbide JVelasco MGuerrero V(2024)Assessment of a Predictive Model for Academic Performance in a Small-Sized Programming Course2024 International Symposium on Computers in Education (SIIE)10.1109/SIIE63180.2024.10604641(1-6)Online publication date: 19-Jun-2024
https://doi.org/10.1109/SIIE63180.2024.10604641
Show More Cited By

Index Terms

Automatic Inference of Programming Performance and Experience from Typing Patterns

Recommendations

Identification of programmers from typing patterns
Koli Calling '15: Proceedings of the 15th Koli Calling Conference on Computing Education Research

Being able to identify the user of a computer solely based on their typing patterns can lead to improvements in plagiarism detection, provide new opportunities for authentication, and enable novel guidance methods in tutoring systems. However, at the ...
Identification based on typing patterns between programming and free text
Koli Calling '17: Proceedings of the 17th Koli Calling International Conference on Computing Education Research

Identifying people based on their typing has been studied successfully in multiple different contexts. Previous research has shown that identification is possible based on writing predetermined texts such as typing passwords, free text such as essays, ...
Typing Patterns and Authentication in Practical Programming Exams
ITiCSE '16: Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education

In traditional programming courses, students have usually been at least partly graded using pen and paper exams. One of the problems related to such exams is that they only partially connect to the practice conducted within such courses. Testing ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCSE '16: Proceedings of the 47th ACM Technical Symposium on Computing Science Education

February 2016

768 pages

ISBN:9781450336857

DOI:10.1145/2839509

General Chairs:
Carl Alphonce
University at Buffalo
,
Jodi Tims
Baldwin Wallace University
,
Program Chairs:
Michael Caspersen
Aarhus University
,
Stephen Edwards
Virginia Tech University

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 February 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Academy of Finland
Finnish Funding Agency for Innovation

Conference

SIGCSE '16

Sponsor:

SIGCSE

SIGCSE '16: The 47th ACM Technical Symposium on Computing Science Education

March 2 - 5, 2016

Tennessee, Memphis, USA

Acceptance Rates

SIGCSE '16 Paper Acceptance Rate 105 of 297 submissions, 35%;

Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

Upcoming Conference

SIGCSE TS 2025

Sponsor:
sigcse

The 56th ACM Technical Symposium on Computer Science Education

February 26 - March 1, 2025

Pittsburgh , PA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

54
Total Citations
View Citations
551
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)4

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sheard JDenny PHellas ALeinonen JMalmi LSimon Stephenson BStone JBattestilli LRebelsky SShoop L(2024)Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview StudyProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630880(1223-1229)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630880
Urry JEdwards JStephenson BStone JBattestilli LRebelsky SShoop L(2024)A Framework that Explores the Cognitive Load of CS1 Assignments Using Pausing BehaviorProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630760(1347-1353)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630760
Mosquera JIturbide JVelasco MGuerrero V(2024)Assessment of a Predictive Model for Academic Performance in a Small-Sized Programming Course2024 International Symposium on Computers in Education (SIIE)10.1109/SIIE63180.2024.10604641(1-6)Online publication date: 19-Jun-2024
https://doi.org/10.1109/SIIE63180.2024.10604641
Llanos JBucheli VRestrepo-Calle F(2023)Early prediction of student performance in CS1 programming coursesPeerJ Computer Science10.7717/peerj-cs.16559(e1655)Online publication date: 31-Oct-2023
https://doi.org/10.7717/peerj-cs.1655
Snider JLaakso MMonga MSimon Sheard J(2023)Edit, Run, Error, Repeat: Learning Analytics to Find Struggling Students in Upper Secondary Programming ClassesProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 210.1145/3587103.3594144(629-630)Online publication date: 29-Jun-2023
https://dl.acm.org/doi/10.1145/3587103.3594144
Lehtinen THaaranen LLeinonen J(2023)Automated Questionnaires About Students’ JavaScript Programs: Towards Gauging Novice Programming ProcessesProceedings of the 25th Australasian Computing Education Conference10.1145/3576123.3576129(49-58)Online publication date: 30-Jan-2023
https://dl.acm.org/doi/10.1145/3576123.3576129
Pullar-Strecker ZPereira FDenny PLuxton-Reilly ALeinonen JDoyle MStephenson BDorn BSoh LBattestilli L(2023)G is for GeneralisationProceedings of the 54th ACM Technical Symposium on Computer Science Education V. 110.1145/3545945.3569824(1028-1034)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3545945.3569824
Hart KMano CEdwards JDoyle MStephenson BDorn BSoh LBattestilli L(2023)Plagiarism Deterrence in CS1 Through Keystroke DataProceedings of the 54th ACM Technical Symposium on Computer Science Education V. 110.1145/3545945.3569805(493-499)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3545945.3569805
Heinonen AHellas ADoyle MStephenson BDorn BSoh LBattestilli L(2023)Time-constrained Code Recall Tasks for Monitoring the Development of Programming PlansProceedings of the 54th ACM Technical Symposium on Computer Science Education V. 110.1145/3545945.3569757(806-812)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3545945.3569757
Matsumoto KNishiura KSasakura MMonden A(2023)Analysis of Programming Performance Based on 2-grams of Keystrokes and Mouse Operations2023 IEEE/ACIS 21st International Conference on Software Engineering Research, Management and Applications (SERA)10.1109/SERA57763.2023.10197645(301-306)Online publication date: 23-May-2023
https://doi.org/10.1109/SERA57763.2023.10197645
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents