More Web Proxy on the site http://driver.im/

research-article

Continuous Evaluation of Video Lectures from Real-Time Difficulty Self-Report

Authors:

Namrata Srivastava,

Eduardo Velloso,

Jason M. Lodge,

James BaileyAuthors Info & Claims

CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Paper No.: 586, Pages 1 - 12

https://doi.org/10.1145/3290605.3300816

Published: 02 May 2019 Publication History

Abstract

With the increased reach and impact of video lectures, it is crucial to understand how they are experienced. Whereas previous studies typically present questionnaires at the end of the lecture, they fail to capture students' experience in enough granularity. In this paper we propose recording the lecture difficulty in real-time with a physical slider, enabling continuous and fine-grained analysis of the learning experience. We evaluated our approach in a study with 100 participants viewing two variants of two short lectures. We demonstrate that our approach helps us paint a more complete picture of the learning experience. Our analysis has design implications for instructors, providing them with a method that helps them compare their expectations with students' beliefs about the lectures and to better understand the specific effects of different instructional design decisions.

Supplementary Material

ZIP File (paper586.zip)

The supplementary files include the following study-design documents 1. Demographics and Prior Knowledge Questionnaire 2. Pre-test and Post-test questionnaires 3. Lecture Feedback forms All the files are saved together in a single PDF file, reflecting the original study design used in the paper.

Download
315.93 KB

MP4 File (paper586.mp4)

Supplemental video

Download
45.16 MB

MP4 File (paper586p.mp4)

Preview video

Download
3.37 MB

References

[1]

Ashton Anderson, Daniel Huttenlocher, Jon Kleinberg, and Jure Leskovec. 2014. Engaging with massive online courses. In Proceedings of the 23rd international conference on World wide web. ACM, 687--698.

Digital Library

[2]

Amaël Arguel, Jason Lodge, Mariya Pachman, and Paula de Barba. 2016. Confidence drives exploration strategies in interactive simulations. ASCILITE.

[3]

S Adams Becker, Michele Cummins, Adam Davis, Alex Freeman, C Glesinger Hall, and Vanish Ananthanarayanan. 2017. NMC horizon report: 2017 higher education edition. Technical Report. The New Media Consortium.

[4]

Alberto Betella and Paul FMJ Verschure. 2016. The affective slider: A digital self-assessment scale for the measurement of human emotions. PloS one 11, 2 (2016), e0148037.

[5]

Lori Breslow, David E Pritchard, Jennifer DeBoer, Glenda S Stump, Andrew D Ho, and Daniel T Seaton. 2013. Studying learning in the worldwide classroom research into edX's first MOOC. Research & Practice in Assessment 8 (2013), 13--25.

[6]

Roland Brünken, Jan L Plass, and Detlev Leutner. 2004. Assessment of cognitive load in multimedia learning with dual-task methodology: Auditory load and modality effects. Instructional Science 32, 1--2 (2004), 115--132.

[7]

Kelly Caine. 2016. Local Standards for Sample Size at CHI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 981--992.

Digital Library

[8]

Hee Jun Choi and Scott D Johnson. 2005. The effect of context-based video instruction on learning and motivation in online courses. The American Journal of Distance Education 19, 4 (2005), 215--227.

[9]

Konstantinos Chorianopoulos. 2013. Collective intelligence within web video. Human-centric Computing and Information Sciences 3, 1 (15 Jun 2013), 10.

[10]

Andrew Cross, Mydhili Bayyapunedi, Edward Cutrell, Anant Agarwal, and William Thies. 2013. TypeRighting: combining the benefits of handwriting and typeface in online educational videos. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 793--796.

Digital Library

[11]

Michael N. Giannakos, Konstantinos Chorianopoulos, and Nikos Chrisochoides. 2014. Collecting and making sense of video learning analytics. In 2014 IEEE Frontiers in Education Conference (FIE) Proceedings. 1--7.

[12]

Michail N Giannakos, Konstantinos Chorianopoulos, and Nikos Chrisochoides. 2014. Collecting and making sense of video learning analytics. In Frontiers in Education Conference (FIE), 2014 IEEE. IEEE, 1--7.

[13]

Philip J Guo, Juho Kim, and Rob Rubin. 2014. How video production affects student engagement: an empirical study of MOOC videos. In Proceedings of the first ACM conference on Learning@ scale conference. ACM, 41--50.

Digital Library

[14]

Päivi Karppinen. 2005. Meaningful learning with digital and online videos: Theoretical perspectives. AACE Journal 13, 3 (2005), 233--250.

[15]

Juho Kim, Philip J Guo, Daniel T Seaton, Piotr Mitros, Krzysztof Z Gajos, and Robert C Miller. 2014. Understanding in-video dropouts and interaction peaks inonline lecture videos. In Proceedings of the first ACM conference on Learning@ scale conference. ACM, 31--40.

Digital Library

[16]

René F Kizilcec, Jeremy N Bailenson, and Charles J Gomez. 2015. The instructor's face in video instruction: Evidence from two large-scale field studies. Journal of Educational Psychology 107, 3 (2015), 724.

[17]

René F Kizilcec, Kathryn Papadopoulos, and Lalida Sritanyaratana. 2014. Showing face in video instruction: effects on information retention, visual attention, and affect. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 2095--2102.

Digital Library

[18]

KH Vincent Lau, Pue Farooque, Gary Leydon, Michael L Schwartz, R Mark Sadler, and Jeremy J Moeller. 2018. Using learning analytics to evaluate a video-based lecture series. Medical teacher 40, 1 (2018), 91--98.

[19]

Gaël Laurans, Pieter MA Desmet, and Paul Hekkert. 2009. The emotion slider: A self-report device for the continuous measurement of emotion. In 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops (ACII 2009). Ieee, 1--6.

[20]

Nan Li, Lukasz Kidzinski, Patrick Jermann, and Pierre Dillenbourg. 2015. How Do In-video Interactions Reflect Perceived Video Difficulty? Proceedings Papers (2015), 112.

[21]

Shih-Yin Lin, John M Aiken, Daniel T Seaton, Scott S Douglas, Edwin F Greco, Brian D Thoms, and Michael F Schatz. 2016. Exploring University Students' Engagement with Online Video Lectures in a Blended Introductory Mechanics Course. arXiv preprint arXiv:1603.03348 (2016).

[22]

Tharindu Rekha Liyanagunawardena, Andrew Alexandar Adams, and Shirley Ann Williams. 2013. MOOCs: A systematic study of the published literature 2008--2012. The International Review of Research in Open and Distributed Learning 14, 3 (2013), 202--227.

[23]

Jason Lodge, Jared Cooney Horvath, Alex Horton, Gregor Kennedy, Sven Venema, and Shane Dawson. 2017. Designing videos for learning: Separating the good from the bad and the ugly. (01 2017).

[24]

Jason Lodge and Gregor Kennedy. 2015. Prior knowledge, confidence and understanding in interactive tutorials and simulations. ASCILITE.

[25]

Jason M Lodge, Gregor Kennedy, Lori Lockyer, Amael Arguel, and Mariya Pachman. 2018. Understanding difficulties and resulting confusion in learning: An integrative review. In Frontiers in Education, Vol. 3. Frontiers, 49.

[26]

Richard E Mayer and Roxana Moreno. 2003. Nine ways to reduce cognitive load in multimedia learning. Educational psychologist 38, 1 (2003), 43--52.

[27]

Barbara Mitra, Jenny Lewin-Jones, Heather Barrett, and Stella Williamson. 2010. The use of video to enable deep learning. Research in Post-Compulsory Education 15, 4 (2010), 405--414.

[28]

Dan R Olsen and Brandon Moon. 2011. Video summarization based on user interaction. In Proceedings of the 9th European Conference on Interactive TV and Video. ACM, 115--122.

Digital Library

[29]

Fred Paas, Juhani E Tuovinen, Huib Tabbers, and Pascal WM Van Gerven. 2003. Cognitive load measurement as a means to advance cognitive load theory. Educational psychologist 38, 1 (2003), 63--71.

[30]

Fred GWC Paas and Jeroen JG Van Merriënboer. 1993. The efficiency of instructional conditions: An approach to combine mental effort and performance measures. Human factors 35, 4 (1993), 737--743.

[31]

Evan F Risko, Tom Foulsham, Shane Dawson, and Alan Kingstone. 2013. The collaborative lecture annotation system (CLAS): A new TOOL for distributed learning. IEEE Transactions on Learning Technologies 6, 1 (2013), 4--13.

Digital Library

[32]

Jennifer Sabourin, Bradford Mott, and James C Lester. 2011. Modeling learner affect with theoretically grounded dynamic Bayesian networks. In International Conference on Affective Computing and Intelligent Interaction. Springer, 286--295.

Digital Library

[33]

Ryan Shaw and Marc Davis. 2005. Toward emergent representations for video. In Proceedings of the 13th annual ACM international conference on Multimedia. ACM, 431--434.

Digital Library

[34]

Hyungyu Shin, Eun-Young Ko, Joseph Jay Williams, and Juho Kim. 2018. Understanding the Effect of In-Video Prompting on Learners and Instructors. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 319.

Digital Library

[35]

Nicholas C Soderstrom, Carole L Yue, and Elizabeth Ligon Bjork. 2016. 11 Metamemory and Education. The Oxford Handbook of Metamemory (2016), 197.

[36]

Jason M Tangen, Merryn D Constable, Eric Durrant, Chris Teeter, Brett R Beston, and Joseph A Kim. 2011. The role of interest and images in slideware presentations. Computers & Education 56, 3 (2011), 865--872.

Digital Library

[37]

Peter Tiernan. 2015. An inquiry into the current and future uses of digital video in University teaching. Education and Information Technologies 20, 1 (2015), 75--90.

Digital Library

[38]

Frans Van der Sluis, Jasper Ginn, and Tim Van der Zee. 2016. Explaining Student Behavior at Scale: The influence of video complexity on student dwelling time. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. ACM, 51--60.

Digital Library

Cited By

Stamm JRichardson DWard J(2024)Squeeze and Slide: Real-time continuous self-reports with physiological arousal to evaluate emotional engagement in short films of contemporary danceExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651886(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651886
Watanabe KDengel AIshimaru S(2023)Accelerating Knowledge Transfer by Sensing and Actuating Social-Cognitive StatesAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610769(258-262)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610769
Kosch TKarolus JZagermann JReiterer HSchmidt AWoźniak P(2023)A Survey on Measuring Cognitive Workload in Human-Computer InteractionACM Computing Surveys10.1145/358227255:13s(1-39)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3582272
Show More Cited By

Index Terms

Continuous Evaluation of Video Lectures from Real-Time Difficulty Self-Report
1. Applied computing
  1. Education
    1. Interactive learning environments

Recommendations

The Effects of Flipping an English for Academic Purposes Course

This article addresses the lack of research into the effects of flipping tertiary level English for Academic Purposes courses. An experimental method was used to compare the outcomes and satisfaction of students n=29 enrolled in a flipped and a ...
Pedagogy Experiments with Recorded Video Lectures
T4E '14: Proceedings of the 2014 IEEE Sixth International Conference on Technology for Education

This paper explores the effectiveness of various video based learning methods. In recent years the flipped classroom has emerged as an innovative teaching-learning process. In a flipped classroom, study materials are made available to students for self ...
Usability design for video lectures
EuroITV '13: Proceedings of the 11th European Conference on Interactive TV and Video

There is a growing number and variety of educational video lectures online, but there is limited understanding of their effectiveness in terms of learning and usability. Although there is significant research literature within the individual domains of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

May 2019

9077 pages

ISBN:9781450359702

DOI:10.1145/3290605

General Chairs:
Stephen Brewster
University of Glasgow, Scotland, UK
,
Geraldine Fitzpatrick
TU Wien, Austria
,
Program Chairs:
Anna Cox
University College London, UK
,
Vassilis Kostakos
University of Melbourne, Australia

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Australian Research Council

Conference

CHI '19

Sponsor:

SIGCHI

CHI '19: CHI Conference on Human Factors in Computing Systems

May 4 - 9, 2019

Glasgow, Scotland Uk

Acceptance Rates

CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
584
Total Downloads

Downloads (Last 12 months)57
Downloads (Last 6 weeks)2

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Stamm JRichardson DWard J(2024)Squeeze and Slide: Real-time continuous self-reports with physiological arousal to evaluate emotional engagement in short films of contemporary danceExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651886(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651886
Watanabe KDengel AIshimaru S(2023)Accelerating Knowledge Transfer by Sensing and Actuating Social-Cognitive StatesAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610769(258-262)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610769
Kosch TKarolus JZagermann JReiterer HSchmidt AWoźniak P(2023)A Survey on Measuring Cognitive Workload in Human-Computer InteractionACM Computing Surveys10.1145/358227255:13s(1-39)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3582272
Chavan PMitra RSarkar APanwar A(2023)A contextualized assessment of reliability and validity of student-initiated momentary self-reports during lecturesEducational technology research and development10.1007/s11423-023-10304-2Online publication date: 29-Nov-2023
https://doi.org/10.1007/s11423-023-10304-2
Das SChakraborty SMitra BBellogin ABoratto LSantos OArdissono LKnijnenburg B(2022)I Cannot See Students Focusing on My Presentation; Are They Following Me? Continuous Monitoring of Student Engagement through “Stungage”Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization10.1145/3503252.3531307(243-253)Online publication date: 4-Jul-2022
https://dl.acm.org/doi/10.1145/3503252.3531307
Choi YKim JHong J(2022)Immersion Measurement in Watching Videos Using Eye-tracking DataIEEE Transactions on Affective Computing10.1109/TAFFC.2022.320931113:4(1759-1770)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TAFFC.2022.3209311
Lee HMasser BTangen JMcKimmie B(2022)The effects of victim testimony order and judicial education on juror decision-making in trials for rapePsychology, Crime & Law10.1080/1068316X.2022.209954630:6(509-537)Online publication date: 28-Jul-2022
https://doi.org/10.1080/1068316X.2022.2099546
Mohammadhassan NMitrovic A(2022)Investigating the Effectiveness of Visual Learning Analytics in Active Video WatchingArtificial Intelligence in Education10.1007/978-3-031-11644-5_11(127-139)Online publication date: 27-Jul-2022
https://doi.org/10.1007/978-3-031-11644-5_11
Amemiya TAoyama KIto K(2022)Effect of Face Appearance of a Teacher Avatar on Active Participation During Online Live ClassHuman Interface and the Management of Information: Applications in Complex Technological Environments10.1007/978-3-031-06509-5_7(99-110)Online publication date: 16-Jun-2022
https://doi.org/10.1007/978-3-031-06509-5_7
Srivastava NNawaz SNewn JLodge JVelloso EM. Erfani SGasevic DBailey JScheffel MDowell NJoksimovic SSiemens G(2021)Are you with me? Measurement of Learners’ Video-Watching Attention with Eye TrackingLAK21: 11th International Learning Analytics and Knowledge Conference10.1145/3448139.3448148(88-98)Online publication date: 12-Apr-2021
https://dl.acm.org/doi/10.1145/3448139.3448148
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents