More Web Proxy on the site http://driver.im/

short-paper

MOOC Dropout Prediction: How to Measure Accuracy?

Authors:

Jacob Whitehill,

Dustin TingleyAuthors Info & Claims

L@S '17: Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

Pages 161 - 164

https://doi.org/10.1145/3051457.3053974

Published: 12 April 2017 Publication History

Abstract

In order to obtain reliable accuracy estimates for automatic MOOC dropout predictors, it is important to train and test them in a manner consistent with how they will be used in practice. Yet most prior research on MOOC dropout prediction has measured test accuracy on the same course used for training, which can lead to overly optimistic accuracy estimates. In order to understand better how accuracy is affected by the training+testing regime, we compared the accuracy of a standard dropout prediction architecture (clickstream features + logistic regression) across 4 different training paradigms. Results suggest that (1) training and testing on the same course ("post-hoc") can significantly overestimate accuracy. Moreover, (2) training dropout classifiers using proxy labels based on students' persistence -- which are available before a MOOC finishes -- is surprisingly competitive with post-hoc training (87.33% v.~90.20% AUC averaged over 8 weeks of 40 HarvardX MOOCs) and can support real-time MOOC interventions.

References

[1]

G. Balakrishnan and D. Coetzee. Predicting student retention in massive open online courses using hidden markov models. Technical report, UC Berkeley, 2013.

[2]

S. Boyer and K. Veeramachaneni. Transfer learning for predictive models in massive open online courses. In International Conference on Artificial Intelligence in Education, 2015.

[3]

C. Coleman, D. Seaton, and I. Chuang. Probabilistic use cases: Discovering behavioral patterns for predicting certification. In Learning at Scale, 2015.

[4]

S. Crossley, L. Paquette, M. Dascalu, D. S. McNamara, and R. S. Baker. Combining click-stream data with nlp tools to better understand MOOC completion. In Learning Analytics & Knowledge, pages 6--14. ACM, 2016.

Digital Library

[5]

M. Fei and D.-Y. Yeung. Temporal models for predicting student dropout in massive open online courses. In International Conference on Data Mining Workshop (ICDMW), 2015.

Digital Library

[6]

S. Halawa, D. Greene, and J. Mitchell. Dropout prediction in MOOCs using learner activity features. In European MOOC Summit, 2014.

[7]

J. He, J. Bailey, Benjamin, I. Rubinstein, and R. Zhang. Identifying at-risk students in massive open online courses. In AAAI, 2015.

[8]

S. Jiang, A. Williams, K. Schenke, M. Warschauer, and D. O'Dowd. Predicting MOOC performance with week 1 behavior. In Educational Data Mining, 2014.

[9]

R. Kizilcec and S. Halawa. Attrition and achievement gaps in online learning. In Learning at Scale, 2015.

Digital Library

[10]

M. Kloft, F. Stiehler, Z. Zheng, and N. Pinkwart. Predicting MOOC dropout over weeks using machine learning methods. In Proceedings of the EMNLP 2014 Workshop on Analysis of Large Scale Social Interaction in MOOCs, pages 60--65, 2014.

[11]

K. R. Koedinger, J. Kim, J. Z. Jia, E. A. McLaughlin, and N. L. Bier. Learning is not a spectator sport: Doing is better than watching for learning from a MOOC. In Learning at Scale, 2015.

[12]

C. Robinson, M. Yeomans, J. Reich, C. Hulleman, and H. Gehlbach. Forecasting student achievement in MOOCs with natural language processing. In Learning Analytics & Knowledge, 2016.

Digital Library

[13]

C. P. Rosé, R. Carlson, D. Yang, M. Wen, L. Resnick, P. Goldman, and J. Sherer. Social factors that contribute to attrition in MOOCs. In Learning at Scale, pages 197--198. ACM, 2014.

Digital Library

[14]

R. Stein and G. Allione. Mass attrition: An analysis of drop out from a principles of microeconomics MOOC. PIER Working Paper, 14(031), 2014.

[15]

C. Taylor, K. Veeramachaneni, and U.-M. O'Reilly. Likely to stop? Predicting stopout in massive open online courses. arXiv, 2014. http://arxiv.org/abs/1408.3382.

[16]

J. Whitehill, K. Mohan, D. Seaton, Y. Rosen, and D. Tingley. Delving deeper into MOOC student dropout prediction. arXiv, 2017. http://arxiv.org/abs/1702.06404.

[17]

J. Whitehill, J. Williams, G. Lopez, C. Coleman, and J. Reich. Beyond prediction: Toward automatic intervention to reduce MOOC student stopout. In Educational Data Mining, 2015.

[18]

W. Xing, X. Chen, J. Stein, and M. Marcinkowski. Temporal predication of dropouts in MOOCs: Reaching the low hanging fruit through stacking generalization. Computers in Human Behavior, 58:119--129, 2016.

Digital Library

[19]

D. Yang, T. Sinha, D. Adamson, and C. P. Rose. "Turn on, tune in, drop out": Anticipating student dropouts in massive open online courses. In NIPS Workshop on Data-Driven Education, 2014.

[20]

C. Ye and G. Biswas. Early prediction of student dropout and performance in MOOCs using higher granularity temporal information. Journal of Learning Analytics, 1(3), 2014.

Cited By

Hooshyar DAzevedo RYang Y(2024)Augmenting Deep Neural Networks with Symbolic Educational Knowledge: Towards Trustworthy and Interpretable AI for EducationMachine Learning and Knowledge Extraction10.3390/make60100286:1(593-618)Online publication date: 10-Mar-2024
https://doi.org/10.3390/make6010028
Qiu WKhong ASupraja STang W(2024)A Dual-Mode Grade Prediction Architecture for Identifying At-Risk StudentsIEEE Transactions on Learning Technologies10.1109/TLT.2023.333302917(803-814)Online publication date: 2024
https://doi.org/10.1109/TLT.2023.3333029
Swacha JMuszyńska K(2023)Predicting Dropout in Programming MOOCs through Demographic InsightsElectronics10.3390/electronics1222467412:22(4674)Online publication date: 16-Nov-2023
https://doi.org/10.3390/electronics12224674
Show More Cited By

Index Terms

MOOC Dropout Prediction: How to Measure Accuracy?
1. Applied computing
  1. Education
    1. E-learning
2. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
        Supervised learning by classification

Recommendations

Predicting Student Dropout in a MOOC: An Evaluation of a Deep Neural Network Model
ICCAI '19: Proceedings of the 2019 5th International Conference on Computing and Artificial Intelligence

Massive Open Online Courses (MOOCs) have transformed the way educational institutions deliver high-quality educational material to the onsite and distance learners across the globe. As a result, a new paradigm shifts as to how learners acquire and ...
MOOC Dropout Prediction: Lessons Learned from Making Pipelines Interpretable
WWW '17 Companion: Proceedings of the 26th International Conference on World Wide Web Companion

Dropout prediction in MOOCs is a well-researched problem where we classify which students are likely to persist or drop out of a course. Most research into creating models which can predict outcomes is based on student engagement data. Why these ...
MOOC Dropout Prediction Based on Dynamic Embedding Representation Learning
CSAE '21: Proceedings of the 5th International Conference on Computer Science and Application Engineering

Massive Open Online Courses (MOOCs) received great attentions in recent years. Most MOOCs have huge number of participants, which usually introduce another challenge—the extremely high dropout rate. Thus, people use a large amount of user-item ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

L@S '17: Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

April 2017

352 pages

ISBN:9781450344500

DOI:10.1145/3051457

General Chair:
Claudia Urrea
Massachusetts Institute of Technology, USA
,
Program Chairs:
Justin Reich
Massachusetts Institute of Technology, USA
,
Candace Thille
Stanford University, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

L@S 2017

Sponsor:

ACM

L@S 2017: Fourth (2017) ACM Conference on Learning @ Scale

April 20 - 21, 2017

Massachusetts, Cambridge, USA

Acceptance Rates

L@S '17 Paper Acceptance Rate 14 of 105 submissions, 13%;

Overall Acceptance Rate 117 of 440 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

84
Total Citations
View Citations
759
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)4

Reflects downloads up to 17 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hooshyar DAzevedo RYang Y(2024)Augmenting Deep Neural Networks with Symbolic Educational Knowledge: Towards Trustworthy and Interpretable AI for EducationMachine Learning and Knowledge Extraction10.3390/make60100286:1(593-618)Online publication date: 10-Mar-2024
https://doi.org/10.3390/make6010028
Qiu WKhong ASupraja STang W(2024)A Dual-Mode Grade Prediction Architecture for Identifying At-Risk StudentsIEEE Transactions on Learning Technologies10.1109/TLT.2023.333302917(803-814)Online publication date: 2024
https://doi.org/10.1109/TLT.2023.3333029
Swacha JMuszyńska K(2023)Predicting Dropout in Programming MOOCs through Demographic InsightsElectronics10.3390/electronics1222467412:22(4674)Online publication date: 16-Nov-2023
https://doi.org/10.3390/electronics12224674
Porras JLara JRomero CVentura S(2023)A Case-Study Comparison of Machine Learning Approaches for Predicting Student’s Dropout from Multiple Online Educational EntitiesAlgorithms10.3390/a1612055416:12(554)Online publication date: 3-Dec-2023
https://doi.org/10.3390/a16120554
Albreiki BHabuza TZaki N(2023)Extracting topological features to identify at-risk students using machine learning and graph convolutional network modelsInternational Journal of Educational Technology in Higher Education10.1186/s41239-023-00389-320:1Online publication date: 10-Apr-2023
https://doi.org/10.1186/s41239-023-00389-3
Bañeres DRodríguez-González MGuerrero-Roldán ACortadas P(2023)An early warning system to identify and intervene online dropout learnersInternational Journal of Educational Technology in Higher Education10.1186/s41239-022-00371-520:1Online publication date: 10-Jan-2023
https://doi.org/10.1186/s41239-022-00371-5
Baneres DRodríguez-González MGuerrero-Roldán A(2023)A Real-Time Predictive Model for Identifying Course Dropout in Online Higher EducationIEEE Transactions on Learning Technologies10.1109/TLT.2023.326727516:4(484-499)Online publication date: 14-Apr-2023
https://dl.acm.org/doi/10.1109/TLT.2023.3267275
Perdue M(2023)Incorporating Learner Perspectives into Course Design2023 IEEE Learning with MOOCS (LWMOOCS)10.1109/LWMOOCS58322.2023.10306167(1-7)Online publication date: 11-Oct-2023
https://doi.org/10.1109/LWMOOCS58322.2023.10306167
Hooshyar DYang Y(2023)ImageLM: Interpretable image-based learner modelling for classifying learners’ computational thinkingExpert Systems with Applications10.1016/j.eswa.2023.122283(122283)Online publication date: Oct-2023
https://doi.org/10.1016/j.eswa.2023.122283
Yürüm OTaşkaya-Temizel TYıldırım S(2023)Predictive Video Analytics in Online Courses: A Systematic Literature ReviewTechnology, Knowledge and Learning10.1007/s10758-023-09697-z29:4(1907-1937)Online publication date: 4-Nov-2023
https://doi.org/10.1007/s10758-023-09697-z
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents