Enhancing Tutoring Effectiveness Through Automated Feedback: Preliminary Findings from a Pilot Randomized Controlled Trial on SAT Tutoring
Pages 422 - 426
Abstract
To address educational inequities, high-quality SAT tutoring is crucial for students who need support. Many tutors, however, are novices and require coaching to be effective. To examine this issue, we conducted a pilot randomized controlled trial (RCT) to evaluate the effectiveness of personalized, automated feedback for novice tutors. This feedback aimed to enhance their tutoring skills and, consequently, improve student outcomes. In our RCT, we not only assessed the effectiveness of the feedback provided to novice tutors but also examined the impact of extending this feedback to both tutors and their students, compared to just the tutors alone. Furthermore, we explored how the use of social versus personal goal-oriented language in the feedback influences educational outcomes. Our preliminary findings from this pilot indicate that providing feedback to both tutors and learners led to a statistically significant improvement in average SAT practice test scores. Additionally, this approach significantly increased the tutor talk time ratio, an outcome that was somewhat unexpected and requires further investigation. These initial results form the foundation for a more comprehensive RCT currently underway. This ongoing study aims to delve deeper into these initial findings and refine our understanding of the effectiveness of feedback in tutoring settings.
References
[1]
Sinem Aslan, Nese Alyuz, Cagri Tanriover, Sinem E Mete, Eda Okur, Sidney K D'Mello, and Asli Arslan Esme. 2019. Investigating the impact of a real-time, multimodal student engagement analytics technology in authentic classrooms. In Proceedings of the 2019 chi conference on human factors in computing systems. 1--12.
[2]
Roghayeh Barmaki and Charles E Hughes. 2015. Providing real-time feedback for student teachers in a virtual rehearsal environment. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. 531--537.
[3]
Kelsey Chamberlin, Maï Yasué, and I-Chant A Chiang. 2023. The impact of grades on student motivation. Active Learning in Higher Education 24, 2 (2023), 109--124. https://doi.org/10.1177/1469787418819728 arXiv:https://doi.org/10.1177/1469787418819728
[4]
Dorottya Demszky and Heather Hill. 2023. The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023). 528--538.
[5]
Dorottya Demszky and Jing Liu. 2023. M-Powering Teachers: Natural Language Processing Powered Feedback Improves 1:1 Instruction and Student Outcomes. In Proceedings of the Tenth ACM Conference on Learning @ Scale (Copenhagen, Denmark) (L@S '23). Association for Computing Machinery, New York, NY, USA, 59--69. https://doi.org/10.1145/3573051.3593379
[6]
Dorottya Demszky, Jing Liu, Heather C Hill, Dan Jurafsky, and Chris Piech. 2023. Can automated feedback improve teachers' uptake of student ideas? evidence from a randomized controlled trial in a large-scale online course. Educational Evaluation and Policy Analysis (2023), 01623737231169270.
[7]
Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori Hashimoto. 2021. Measuring conversational uptake: A case study on student-teacher interactions. arXiv preprint arXiv:2106.03873 (2021).
[8]
Dorottya Demszky, Rose Wang, Sean Geraghty, and Carol Yu. 2024. Does Feedback on Talk Time Increase Student Engagement? Evidence from a Randomized Controlled Trial on a Math Tutoring Platform. In Proceedings of the 14th Learning Analytics and Knowledge Conference (LAK '24). Association for Computing Machinery, New York, NY, USA, 632--644. https://doi.org/10.1145/3636555.3636924
[9]
John F Dovidio, Jane Allyn Piliavin, David A Schroeder, and Louis A Penner. 2017. The social psychology of prosocial behavior. Psychology Press.
[10]
Google. 2024. LearnLM. https://storage.googleapis.com/deepmind-media/ LearnLM/LearnLM_paper.pdf. Accessed: 2024-05--15.
[11]
Kunal Handa, Margaret Clapper, Jessica Boyle, Rose E Wang, Diyi Yang, David S Yeager, and Dorottya Demszky. 2023. " Mistakes Help Us Grow": Facilitating and Evaluating Growth Mindset Supportive Language in Classrooms. arXiv preprint arXiv:2310.10637 (2023).
[12]
John Hattie and Helen Timperley. 2007. The Power of Feedback. Review of Educational Research 77, 1 (2007), 81--112. https://doi.org/10.3102/003465430298487 arXiv:https://doi.org/10.3102/003465430298487
[13]
Nicholas Hunkins, Sean Kelly, and Sidney D'Mello. 2022. ?Beautiful work, you're rock stars!": Teacher analytics to uncover discourse that supports or undermines student motivation, identity, and belonging in classrooms. In Lak22: 12th international learning analytics and knowledge conference. 230--238.
[14]
Jennifer Jacobs, Karla Scornavacco, Charis Harty, Abhijit Suresh, Vivian Lai, and Tamara Sumner. 2022. Promoting rich discussions in mathematics classrooms: Using personalized, automated feedback to support reflection and instructional change. Teaching and Teacher Education 112 (2022), 103631.
[15]
Matthew A. Kraft, David Blazar, and Dylan Hogan. 2018. The Effect of Teacher Coaching on Instruction and Achievement: A Meta-Analysis of the Causal Evidence. Review of Educational Research 88, 4 (2018), 547--588. https://doi.org/10. 3102/0034654318759268 arXiv:https://doi.org/10.3102/0034654318759268
[16]
Ashlee Kupor, Candice Morgan, and Dorottya Demszky. 2023. Measuring Five Accountable Talk Moves to Improve Instruction at Scale. arXiv preprint arXiv:2311.10749 (2023).
[17]
Diane Litman. 2016. Natural language processing for enhancing teaching and learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 30.
[18]
Jing Liu and Julie Cohen. 2021. Measuring teaching practices at scale: A novel application of text-as-data methods. Educational Evaluation and Policy Analysis 43, 4 (2021), 587--614.
[19]
Catherine O'Connor, Sarah Michaels, and Suzanne Chapin. 2015. "Scaling Down" to Explore the Role of Talk in Learning: From District Intervention to Controlled Classroom Study. 111--126. https://doi.org/10.3102/978-0--935302--43--1_9
[20]
Schoolhouse. 2024. FAQ Schoolhouse. https://faq.schoolhouse.world/en/article/ what-tutoring-rubric-do-you-use-to-peer-review-sessions-1w0mrw/. Accessed: 2024-05--15.
[21]
Valerie J. Shute. 2008. Focus on Formative Feedback. Review of Educational Research 78, 1 (2008), 153--189. https://doi.org/10.3102/0034654307313795 arXiv:https://doi.org/10.3102/0034654307313795
[22]
Rodrigo Smiderle, Sandro José Rigo, Leonardo B Marques, Jorge Arthur Peçanha de Miranda Coelho, and Patricia A Jaques. 2020. The impact of gamification on students' learning, engagement and behavior based on their personality traits. Smart Learning Environments 7, 1 (2020), 3.
[23]
Abhijit Suresh, Jennifer Jacobs, Charis Harty, Margaret Perkoff, James H Martin, and Tamara Sumner. 2022. The TalkMoves dataset: K-12 mathematics lesson transcripts annotated for teacher and student discursive moves. arXiv preprint arXiv:2204.09652 (2022).
[24]
Abhijit Suresh, Jennifer Jacobs, Margaret Perkoff, James H Martin, and Tamara Sumner. 2022. Fine-tuning transformers with additional context to classify discursive moves in mathematics classrooms. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications.
[25]
Rose E Wang and Dorottya Demszky. 2023. Is chatgpt a good teacher coach? measuring zero-shot performance for scoring and providing actionable insights on classroom instruction. arXiv preprint arXiv:2306.03090 (2023).
[26]
Rose E Wang and Dorottya Demszky. 2024. Edu-ConvoKit: An Open-Source Library for Education Conversation Data. arXiv preprint arXiv:2402.05111 (2024).
Index Terms
- Enhancing Tutoring Effectiveness Through Automated Feedback: Preliminary Findings from a Pilot Randomized Controlled Trial on SAT Tutoring
Recommendations
Does Feedback on Talk Time Increase Student Engagement? Evidence from a Randomized Controlled Trial on a Math Tutoring Platform
LAK '24: Proceedings of the 14th Learning Analytics and Knowledge ConferenceProviding ample opportunities for students to express their thinking is pivotal to their learning of mathematical concepts. We introduce the Talk Meter, which provides in-the-moment automated feedback on student-teacher talk ratios. We conduct a ...
Automated Personalized Feedback Improves Learning Gains in An Intelligent Tutoring System
Artificial Intelligence in EducationAbstractWe investigate how automated, data-driven, personalized feedback in a large-scale intelligent tutoring system (ITS) improves student learning outcomes. We propose a machine learning approach to generate personalized feedback, which takes ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2024
582 pages
ISBN:9798400706332
DOI:10.1145/3657604
- General Chair:
- David Joyner,
- Program Chairs:
- Min Kyu Kim,
- Xu Wang,
- Meng Xia
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 15 July 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
Conference
L@S '24
Acceptance Rates
Overall Acceptance Rate 117 of 440 submissions, 27%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 110Total Downloads
- Downloads (Last 12 months)110
- Downloads (Last 6 weeks)22
Reflects downloads up to 11 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in