More Web Proxy on the site http://driver.im/

research-article

Open access

Evidence-centered Assessment for Writing with Generative AI

Authors:

Guanliang Chen,

Dragan Gašević,

Zachari SwieckiAuthors Info & Claims

LAK '24: Proceedings of the 14th Learning Analytics and Knowledge Conference

Pages 178 - 188

https://doi.org/10.1145/3636555.3636866

Published: 18 March 2024 Publication History

All formats PDF

Abstract

We propose a learning analytics-based methodology for assessing the collaborative writing of humans and generative artificial intelligence. Framed by the evidence-centered design, we used elements of knowledge-telling, knowledge transformation, and cognitive presence to identify assessment claims; we used data collected from the CoAuthor writing tool as potential evidence for these claims; and we used epistemic network analysis to make inferences from the data about the claims. Our findings revealed significant differences in the writing processes of different groups of CoAuthor users, suggesting that our method is a plausible approach to assessing human-AI collaborative writing.

References

[1]

Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software 67, 1 (2015), 1–48.

[2]

Carl Bereiter and Marlene Scardamalia. 1987. The psychology of written composition. Erlbaum.

[3]

Dale Bowman, Zachari Swiecki, Zhiqiang Cai, Yeyu Wang, Brendan Eagan, Jeff Linderoth, and David Williamson Shaffer. 2021. The mathematical foundations of epistemic network analysis. In Advances in Quantitative Ethnography(Communications in Computer and Information Science). Springer, 91–105.

[4]

Daniel Chandler. 1997. An introduction to genre theory. (1997).

[5]

Andy Clark and David Chalmers. 1998. The Extended Mind. Analysis 58, 1 (1998), 7–19.

[6]

J. Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates.

[7]

Joshua Cramp, John F. Medlin, Phoebe Lake, and Colin Sharp. 2019. Lessons learned from implementing remotely invigilated online exams. Journal of University Teaching and Learning Practice 16, 1 (2019), 137–155.

[8]

Yizhou Fan, Mladen Rakovic, Joep van der Graaf, Lyn Lim, Shaveen Singh, Johanna Moore, Inge Molenaar, Maria Bannert, and Dragan Gašević. 2023. Towards a fuller picture: Triangulation and integration of the measurement of self-regulated learning based on trace and think aloud data. Journal of Computer Assisted Learning 39, 4 (2023), 1303–1324.

[9]

D.Randy Garrison, Terry Anderson, and Walter Archer. 1999. Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education. The Internet and Higher Education 2, 2 (1999), 87–105.

[10]

Dragan Gasevic, Shane Dawson, and George Siemens. 2015. Let’s not forget: Learning analytics are about learning. TechTrends 59 (01 2015).

[11]

Andrew Gibson and Antonette Shibani. 2022. Natural Language Processing - Writing Analytics. In The Handbook of Learning Analytics (2 ed.). SoLAR, 96–104.

[12]

H. Guo, M. Zhang, P. Deane, and R. E. Bennett. 2019. Writing Process Differences in Subgroups Reflected in Keystroke Logs. Journal of Educational and Behavioral Statistics 44(5) (2019), 571–596.

[13]

Yuanyuan Hu, Rafael Ferreira Mello, and Dragan Gašević. 2021. Automatic analysis of cognitive presence in online discussions: An approach using deep learning and explainable artificial intelligence. Computers and Education: Artificial Intelligence 2 (2021), 100037.

[14]

Rob Hyndman and Yanan Fan. 1996. Sample Quantiles in Statistical Packages. The American Statistician 50 (11 1996), 361–365.

[15]

Sehrish Iqbal, Mladen Rakovic, Guanliang Chen, Tongguang Li, Rafael Ferreira Mello, Yizhou Fan, Giuseppe Fiorentino, Naif Radi Aljohani, and Dragan Gasevic. 2023. Towards Automated Analysis of Rhetorical Categories in Students Essay Writings Using Bloom’s Taxonomy. In LAK23: 13th International Learning Analytics and Knowledge Conference(LAK2023). Association for Computing Machinery, 418–429.

[16]

Sehrish Iqbal, Zachari Swiecki, Srecko Joksimovic, Rafael Ferreira Mello, Naif Aljohani, Saeed Ul Hassan, and Dragan Gasevic. 2022. Uncovering Associations Between Cognitive Presence and Speech Acts: A Network-Based Approach. In LAK22: 12th International Learning Analytics and Knowledge Conference(LAK22). Association for Computing Machinery, 315–325.

[17]

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. Comput. Surveys 55, 12 (mar 2023), 1–38.

Digital Library

[18]

Zixuan Ke and Vincent Ng. 2019. Automated Essay Scoring: A Survey of the State of the Art. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization, 6300–6308.

[19]

Alexandra Kuznetsova, Per B. Brockhoff, and Rune H. B. Christensen. 2017. lmerTest Package: Tests in Linear Mixed Effects Models. Journal of Statistical Software 82, 13 (2017), 1–26.

[20]

Mina Lee, Percy Liang, and Qian Yang. 2022. CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. CoRR abs/2201.06796 (2022). arXiv:2201.06796

[21]

Mariëlle Leijten and Luuk Van Waes. 2013. Keystroke logging in writing research. Written Communication 30, 3 (2013), 358–392.

[22]

Tongguang Li, Yizhou Fan, Yuanru Tan, Yeyu Wang, Shaveen Singh, Xinyu Li, Mladen Raković, Joep van der Graaf, Lyn Lim, Binrui Yang, Inge Molenaar, Maria Bannert, Johanna Moore, Zachari Swiecki, Yi-Shan Tsai, David Williamson Shaffer, and Dragan Gašević. 2023. Analytics of self-regulated learning scaffolding: effects on learning processes. Frontiers in Psychology 14 (2023).

[23]

Nikolaos Limnios and Ghorghe Oprisan. 2001. Semi-Markov Processes and Reliability.

[24]

Marquart, C. L., Swiecki, Z., Collier, W., Eagan, B., Woodward, R., Shaffer, and D. W.2022. rENA: Epistemic Network Analysis.

[25]

Robert Mislevy, Russell Almond, and Janice Lukas. 2003. A Brief Introduction to Evidence-Centered Design.US Department of Education (06 2003).

[26]

Daniel Naber. 2003. A Rule-Based Style and Grammar Checker. (01 2003).

[27]

OpenAI. 2023. GPT-4 Technical Report.

[28]

Lodge J. M.and Howard S.and Bearman M.and Dawson P and Associates. 2023. Assessment reform for the age of Artificial Intelligence. Tertiary Education Quality and Standards Agency (2023).

[29]

Sunder Pichai. 2023. An important next step on our AI journey.

[30]

Mladen Raković. 2019. Automatic Identification of Knowledge Transforming Content in Argument Essays Developed from Multiple Sources. PhD thesis. Simon Fraser University, British Columbia, CA.

[31]

Mladen Raković, Sehrish Iqbal, Tongguang Li, Yizhou Fan, Shaveen Singh, Surya Surendrannair, Jonathan Kilgour, Joep van der Graaf, Lyn Lim, Inge Molenaar, Maria Bannert, Johanna Moore, and Dragan Gašević. 2023. Harnessing the potential of trace data and linguistic analysis to predict learner performance in a multi-text writing task. Journal of Computer Assisted Learning 39, 3 (2023), 703–718.

[32]

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. CoRR abs/1908.10084 (2019).

[33]

F. E. Satterthwaite. 1946. An Approximate Distribution of Estimates of Variance Components. Biometrics Bulletin 2, 6 (Dec. 1946), 110–114.

[34]

David Shaffer and Andrew Ruis. 2017. Epistemic Network Analysis: A Worked Example of Theory-Based Learning Analytics. 175–187.

[35]

David Williamson Shaffer and A. R. Ruis. 2021. How We Code. In Advances in Quantitative Ethnography. Springer International Publishing, 62–77.

[36]

Antonette Shibani, Simon Knight, and Simon Buckingham Shum. 2018. Understanding Revisions in Student Writing Through Revision Graphs. In Artificial Intelligence in Education. Springer International Publishing, Cham, 332–336.

[37]

Antonette Shibani, Ming Liu, Christian Rapp, and Simon Knight. 2019. Advances in Writing Analytics: Mapping the state of the field. In Companion Proceedings of the 9th International Conference on Learning Analytics & Knowledge(LAK ’19). Association for Computing Machinery.

[38]

Antonette Shibani, Ratnavel Rajalakshmi, Faerie Mattins, Srivarshan Selvaraj, and Simon Knight. 2023. Visual representation of co-authorship with GPT-3: Studying human-machine interaction for effective writing. In Proceedings of the 16th International Conference on Educational Data Mining. International Educational Data Mining Society, 183–193.

[39]

Vilaythong Southavilay, Kalina Yacef, Peter Reimann, and Rafael A. Calvo. 2013. Analysis of Collaborative Writing Processes Using Revision Maps and Probabilistic Topic Models. In Proceedings of 3rd International Conference on Learning Analytics and Knowledge(LAK ’13). Association for Computing Machinery, 38–47.

Digital Library

[40]

Zachari Swiecki, Hassan Khosravi, Guanliang Chen, Roberto Martinez-Maldonado, Jason M. Lodge, Sandra Milligan, Neil Selwyn, and Dragan Gašević. 2022. Assessment in the age of artificial intelligence. Computers and Education: Artificial Intelligence 3 (2022), 100075.

[41]

Yongjie Wang, Chuang Wang, Ruobing Li, and Hui Lin. 2022. On the Use of Bert for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Seattle, United States, 3416–3425.

[42]

James V. Wertsch. 1998. Mind As Action. Oxford University Press, Incorporated, New York, UNITED STATES.

[43]

Mo Zhang, Hongwen Guo, and Xiang Liu. 2021. Using Keystroke Analytics to Understand Cognitive Processes during Writing.International Educational Data Mining Society (2021).

[44]

Peiying Zhang, Xingzhe Huang, and Lei Zhang. 2020. Information mining and similarity computation for semi- / un-structured sentences from the social data. Digital Communications and Networks 7 (08 2020).

[45]

Alain Zuur, EN Ieno, Neil Walker, Anatoly Saveliev, and GM Smith. 2009. Mixed Effects Models and Extensions in Ecology With R. Vol. 1-574.

Cited By

Wang YLai YHuang X(2024)Innovations in Online Learning Analytics: A Review of Recent Research and Emerging TrendsIEEE Access10.1109/ACCESS.2024.349362112(166761-166775)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3493621
Almegren AMahdi HHazaea AAli JAlmegren R(2024)Evaluating the quality of AI feedback: A comparative study of AI and human essay gradingInnovations in Education and Teaching International10.1080/14703297.2024.2437122(1-16)Online publication date: 3-Dec-2024
https://doi.org/10.1080/14703297.2024.2437122
Olivier CWeilbach L(2024)Enhancing Online Learning Experiences: A Systematic Review on Integrating GenAI Chatbots into the Community of Inquiry FrameworkDisruptive Innovation in a Digitally Connected Healthy World10.1007/978-3-031-72234-9_7(77-89)Online publication date: 10-Sep-2024
https://doi.org/10.1007/978-3-031-72234-9_7

Index Terms

Evidence-centered Assessment for Writing with Generative AI

Recommendations

Integrating Real-Time Drawing and Writing Diagnostic Models: An Evidence-Centered Design Framework for Multimodal Science Assessment
ITS 2016: Proceedings of the 13th International Conference on Intelligent Tutoring Systems - Volume 9684

Interactively modeling science phenomena enables students to develop rich conceptual understanding of science. While this understanding is often assessed through summative, multiple-choice instruments, science notebooks have been used extensively in ...
Assessments for Computational Thinking in K-12 (Abstract Only)
SIGCSE '15: Proceedings of the 46th ACM Technical Symposium on Computer Science Education

As computer science (CS) and computational thinking (CT) make their way into all levels of K-12 education, valid assessments aligned with new curricula can assist in measuring student learning, easing the way for adoption of new computing courses, and ...
A Principled Approach to Designing a Computational Thinking Practices Assessment for Early Grades
SIGCSE '20: Proceedings of the 51st ACM Technical Symposium on Computer Science Education

In today's increasingly digital world, it is critical that all students learn to think computationally from an early age. Assessments of Computational Thinking (CT) are essential for capturing information about student learning and challenges. Several ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

LAK '24: Proceedings of the 14th Learning Analytics and Knowledge Conference

March 2024

962 pages

ISBN:9798400716188

DOI:10.1145/3636555

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 March 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Discovery Projects
Discovery Projects
CELLA 2 CERES

Conference

LAK '24

LAK '24: The 14th Learning Analytics and Knowledge Conference

March 18 - 22, 2024

Kyoto, Japan

Acceptance Rates

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
827
Total Downloads

Downloads (Last 12 months)827
Downloads (Last 6 weeks)146

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang YLai YHuang X(2024)Innovations in Online Learning Analytics: A Review of Recent Research and Emerging TrendsIEEE Access10.1109/ACCESS.2024.349362112(166761-166775)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3493621
Almegren AMahdi HHazaea AAli JAlmegren R(2024)Evaluating the quality of AI feedback: A comparative study of AI and human essay gradingInnovations in Education and Teaching International10.1080/14703297.2024.2437122(1-16)Online publication date: 3-Dec-2024
https://doi.org/10.1080/14703297.2024.2437122
Olivier CWeilbach L(2024)Enhancing Online Learning Experiences: A Systematic Review on Integrating GenAI Chatbots into the Community of Inquiry FrameworkDisruptive Innovation in a Digitally Connected Healthy World10.1007/978-3-031-72234-9_7(77-89)Online publication date: 10-Sep-2024
https://doi.org/10.1007/978-3-031-72234-9_7

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents