More Web Proxy on the site http://driver.im/

research-article

Lessons Learnt from a Multimodal Learning Analytics Deployment In-the-Wild

Authors:

Roberto Martinez-Maldonado,

Vanessa Echeverria,

Gloria Fernandez-Nieto,

Riordan Alfredo,

Hollie Jaggard,

Rosie Wotherspoon,

Simon Buckingham Shum,

Dragan GaševićAuthors Info & Claims

ACM Transactions on Computer-Human Interaction, Volume 31, Issue 1

Article No.: 8, Pages 1 - 41

https://doi.org/10.1145/3622784

Published: 29 November 2023 Publication History

Abstract

Multimodal Learning Analytics (MMLA) innovations make use of rapidly evolving sensing and artificial intelligence algorithms to collect rich data about learning activities that unfold in physical spaces. The analysis of these data is opening exciting new avenues for both studying and supporting learning. Yet, practical and logistical challenges commonly appear while deploying MMLA innovations “in-the-wild”. These can span from technical issues related to enhancing the learning space with sensing capabilities, to the increased complexity of teachers’ tasks. These practicalities have been rarely investigated. This article addresses this gap by presenting a set of lessons learnt from a 2-year human-centred MMLA in-the-wild study conducted with 399 students and 17 educators in the context of nursing education. The lessons learnt were synthesised into topics related to (i) technological/physical aspects of the deployment; (ii) multimodal data and interfaces; (iii) the design process; (iv) participation, ethics and privacy; and (v) sustainability of the deployment.

References

[1]

Gregory D. Abowd and Elizabeth D. Mynatt. 2000. Charting past, present, and future research in ubiquitous computing. ACM Transactions on Computer–Human Interaction (TOCHI) 7, 1 (Mar.2000), 29–58. DOI:

Digital Library

[2]

June Ahn, Fabio Campos, Maria Hays, and Daniela Digiacomo. 2019. Designing in context: Reaching beyond usability in learning analytics dashboard design. Journal of Learning Analytics 6, 2 (Jul.2019), 70–85. DOI:

[3]

Karan Ahuja, Dohyun Kim, Franceska Xhakaj, Virag Varga, Anne Xie, Stanley Zhang, Jay Eric Townsend, Chris Harrison, Amy Ogan, and Yuvraj Agarwal. 2019. EduSense: Practical classroom sensing at scale. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1–26. DOI:

Digital Library

[4]

Hamed S. Alavi, Elizabeth F. Churchill, Mikael Wiberg, Denis Lalanne, Peter Dalsgaard, Ava Fatah gen Schieck, and Yvonne Rogers. 2019. Introduction to human-building interaction (HBI): Interfacing HCI with architecture and urban design. ACM Transactions on Computer-Human Interaction (TOCHI). 26, 2, Article 6 (Mar 2019), 10 pages.

Digital Library

[5]

Riordan Dervin Alfredo, Lanbing Nie, Paul Kennedy, Tamara Power, Carolyn Hayes, Hui Chen, Carolyn McGregor, Zachari Swiecki, Dragan Gašević, and Roberto Martinez-Maldonado. 2023. “That student should be a Lion Tamer!” StressViz: Designing a stress analytics dashboard for teachers. In Proceedings of the 13th International Learning Analytics and Knowledge Conference (Arlington, TX, USA) (LAK’23). Association for Computing Machinery, New York, NY, 57–67. DOI:

Digital Library

[6]

Liaqat Ali, Mohsen Asadi, Dragan Gašević, Jelena Jovanović, and Marek Hatala. 2013. Factors influencing beliefs for adoption of a learning analytics tool: An empirical study. Computers & Education 62 (March 2013), 130–148. DOI:

[7]

Haifa Alwahaby, Mutlu Cukurova, Zacharoula Papamitsiou, and Michail Giannakos. 2022. The Evidence of Impact and Ethical Considerations of Multimodal Learning Analytics: A Systematic Literature Review. Springer International Publishing, Cham, 289–325. DOI:

[8]

Roger Azevedo and Dragan Gašević. 2019. Analyzing multimodal multichannel data about self-regulated learning with advanced learning technologies: Issues and challenges. Computers in Human Behavior 96 (July 2019), 207–210. DOI:

Digital Library

[9]

Mara Balestrini, Sarah Gallacher, and Yvonne Rogers. 2020. Moving HCI outdoors: Lessons learned from conducting research in the wild. In Proceedings of the HCI Outdoors: Theory, Design, Methods and Applications. Springer, 83–98. DOI:

[10]

Oluwaseun Bamgboye, Xiaodong Liu, and Peter Cruickshank. 2018. Towards modelling and reasoning about uncertain data of sensor measurements for decision support in smart spaces. In Proceedings of the 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC’18), Vol. 2. IEEE, 744–749. DOI:

[11]

Glenn Barton, Anne Bruce, and Rita Schreiber. 2018. Teaching nurses teamwork: Integrative review of competency-based team training in nursing education. Nurse Education in Practice 32 (September 2018), 129–137.

[12]

Marc Beardsley, Judit Martínez Moreno, Milica Vujovic, Patricia Santos, and Davinia Hernández-Leo. 2020. Enhancing consent forms to support participant decision making in multimodal learning data research. British Journal of Educational Technology 51, 5 (2020), 1631–1652. DOI:

[13]

Steve Benford, Chris Greenhalgh, Bob Anderson, Rachel Jacobs, Mike Golembewski, Marina Jirotka, Bernd Carsten Stahl, Job Timmermans, Gabriella Giannachi, Matt Adams, Ju Row Farr, Nick Tandavanitj, and Kirsty Jennings. 2015. The ethical implications of HCI’s turn to the cultural. ACM Transactions on Computer–Human Interaction (TOCHI) 22, 5, Article 24 (Aug.2015), 37 pages. DOI:

Digital Library

[14]

Paulo Blikstein. 2013. Multimodal learning analytics. In Proceedings of the LAK’13. 102–106. DOI:

Digital Library

[15]

Virginia Braun and Victoria Clarke. 2012. Thematic analysis.APA, Washington, DC, 57–71. DOI:

[16]

Jason A. Brotherton and Gregory D. Abowd. 2004. Lessons learned from EClass: Assessing automated capture and access in the classroom. ACM Transactions on Computer–Human Interaction (TOCHI) 11, 2 (Jun.2004), 121–155. DOI:

Digital Library

[17]

Simon Buckingham Shum, Rebecca Ferguson, and Roberto Martinez-Maldonado. 2019. Human-centred learning analytics. Journal of Learning Analytics 6, 2 (Jul.2019), 1–9. DOI:

[18]

Davide Ceneda, Theresia Gschwandtner, Thorsten May, Silvia Miksch, Hans-Jörg Schulz, Marc Streit, and Christian Tominski. 2016. Characterizing guidance in visual analytics. IEEE Transactions on Visualization and Computer Graphics 23, 1 (2016), 111–120. DOI:

Digital Library

[19]

Pankaj Chejara, Reet Kasepalu, Luis P. Prieto, María Jesús Rodríguez-Triana, Adolfo Ruiz-Calleja, and Shashi Kant Shankar. 2023. Multimodal learning analytics research in the wild: Challenges and their potential solutions. In Proceedings of the CrossMMLA’23 Workshop: Leveraging Multimodal Data for Generating Meaningful Feedback. 1–7.

[20]

Yi Han Victoria Chua, Justin Dauwels, and Seng Chee Tan. 2019. Technologies for automated analysis of co-located, real-life, physical learning spaces: Where are we now?. In Proceedings of the 9th International Learning Analytics and Knowledge Conference. 11–20. DOI:

Digital Library

[21]

Hector Cornide-Reyes, René Noël, Fabián Riquelme, Matías Gajardo, Cristian Cechinel, Roberto Mac Lean, Carlos Becerra, Rodolfo Villarroel, and Roberto Munoz. 2019. Introducing low-cost sensors into the classroom settings: Improving the assessment in agile practices with multimodal learning analytics. Sensors 19, 15 (2019), 3291. DOI:

[22]

Andy Crabtree, Alan Chamberlain, Rebecca E. Grinter, Matt Jones, Tom Rodden, and Yvonne Rogers. 2013. Introduction to the special issue of “The Turn to The Wild”. ACM Transactions on Computer–Human Interaction (TOCHI) 20, 3 (2013), 1–4. DOI:

Digital Library

[23]

Lucrezia Crescenzi-Lanna. 2020. Multimodal learning analytics research with young children: A systematic review. British Journal of Educational Technology 51, 5 (2020), 1485–1504. DOI:

[24]

Mutlu Cukurova, Michail Giannakos, and Roberto Martinez-Maldonado. 2020. The promise and challenges of multimodal learning analytics. British Journal of Educational Technology 51, 5 (Sep2020), 1441–1449. DOI:

[25]

Fred D. Davis. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13, 3 (1989), 319–340.

Digital Library

[26]

Daniele Di Mitri, Jan Schneider, and Hendrik Drachsler. 2021. Keep me in the loop: Real-time feedback with multimodal data. International Journal of Artificial Intelligence in Education 13 (2021), 1–26. DOI:

[27]

Muhterem Dindar, Sanna Järvelä, and Eetu Haataja. 2020. What does physiological synchrony reveal about metacognitive experiences and group performance? British Journal of Educational Technology 51, 5 (2020), 1577–1596.

[28]

Sidney K. D’Mello, Andrew M. Olney, Nathan Blanchard, Borhan Samei, Xiaoyi Sun, Brooke Ward, and Sean Kelly. 2015. Multimodal capture of teacher-student interactions for automated dialogic analysis in live classrooms. In Proceedings of the ACM on International Conference on Multimodal Interaction. ACM, 557–566. DOI:

Digital Library

[29]

Sidney D’Mello and Art Graesser. 2012. Dynamics of affective states during complex learning. Learning and Instruction 22, 2 (2012), 145–157.

[30]

Vanessa Echeverria, Roberto Martinez-Maldonado, and Simon Buckingham Shum. 2019. Towards collaboration translucence: Giving meaning to multimodal group data. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’19). ACM, 1–16. DOI:

Digital Library

[31]

Vanessa Echeverria, Roberto Martinez-Maldonado, Lixiang Yan, Linxuan Zhao, Gloria Fernandez-Nieto, Dragan Gašević, and Simon Buckingham Shum. 2023. HuCETA: A framework for human-centered embodied teamwork analytics. IEEE Pervasive Computing 22, 1 (2023), 39–49. DOI:

Digital Library

[32]

Jennifer Fereday and Eimear Muir-Cochrane. 2006. Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International Journal of Qualitative Methods 5, 1 (2006), 80–92. DOI:

[33]

Rebecca Ferguson, Tore Hoel, Maren Scheffel, and Hendrik Drachsler. 2016. Guest editorial: Ethics and privacy in learning analytics. Journal of Learning Analytics 3, 1 (2016), 5–15. DOI:

[34]

Gloria Fernandez-Nieto, Roberto Martinez-Maldonado, Vanessa Echeverria, Kirsty Kitto, Pengcheng An, and Simon Buckingham Shum. 2021. What can analytics for teamwork proxemics reveal about positioning dynamics in clinical simulations? Proceedings of the ACM on Human–Computer Interaction 5, CSCW1 (2021), 1–24. DOI:

Digital Library

[35]

Gloria Milena Fernandez Nieto, Kirsty Kitto, Simon Buckingham Shum, and Roberto Martinez-Maldonado. 2022. Beyond the learning analytics dashboard: Alternative ways to communicate student data insights combining visualisation, narrative and storytelling. In Proceedings of the 12th International Learning Analytics and Knowledge Conference. ACM, 219–229. DOI:

Digital Library

[36]

Michail Giannakos, Daniel Spikol, Daniele Di Mitri, Kshitij Sharma, Xavier Ochoa, and Rawad Hammad. 2022. Introduction to Multimodal Learning Analytics. Springer International Publishing, Cham, 3–28. DOI:

[37]

Kirk Goldsberry. 2012. Courtvision: New visual and spatial analytics for the nba. In Proceedings of the 2012 MIT Sloan Sports Analytics Conference, Vol. 9. 12–15.

[38]

Jamie Gorson, Kathryn Cunningham, Marcelo Worsley, and Eleanor O’Rourke. 2022. Using electrodermal activity measurements to understand student emotions while programming. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1. 105–119. DOI:

Digital Library

[39]

Saul Greenberg, Michael Boyle, and Jason LaBerge. 1999. PDAs and shared public displays: Making personal information public, and public information personal. Personal Technologies 3, 1 (1999), 54–64. DOI:

[40]

David Gualda, María Carmen Pérez-Rubio, Jesús Ureña, Sergio Pérez-Bachiller, José Manuel Villadangos, Álvaro Hernández, Juan Jesús García, and Ana Jiménez. 2021. LOCATE-US: Indoor positioning for mobile devices using encoded ultrasonic signals, inertial sensors and graph-matching. Sensors 21, 6 (2021), 1950. DOI:

[41]

Carolina Guzmán-Valenzuela, Carolina Gómez-González, Andrés Rojas-Murphy Tagle, and Alejandro Lorca-Vyhmeister. 2021. Learning analytics in higher education: A preponderance of analytics but very little learning? International Journal of Educational Technology in Higher Education 18, 1 (2021), 1–19. DOI:

[42]

Jeffrey Heer. 2019. Agency plus automation: Designing artificial intelligence into interactive systems. Proceedings of the National Academy of Sciences 116, 6 (2019), 1844–1850. DOI:

[43]

Bernie Hogan, Joshua R. Melville, Gregory Lee Phillips II, Patrick Janulis, Noshir Contractor, Brian S. Mustanski, and Michelle Birkett. 2016. Evaluating the paper-to-screen translation of participant-aided sociograms with high-risk participants. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 5360–5371.

[44]

Kasper Hornbæk and Morten Hertzum. 2017. Technology acceptance and user experience: A review of the experiential component in HCI. ACM Transactions on Computer–Human Interaction (TOCHI) 24, 5 (2017), 1–30.

Digital Library

[45]

Jinglu Jiang, Surinder Kahai, and Ming Yang. 2022. Who needs explanation and when? Juggling explainable AI and user epistemic uncertainty. International Journal of Human–Computer Studies 165 (September 2022), 102839. DOI:

Digital Library

[46]

Rogers Kaliisa, Anders Kluge, and Anders I Mørch. 2022. Overcoming challenges to the adoption of learning analytics at the practitioner level: A critical analysis of 18 learning analytics frameworks. Scandinavian Journal of Educational Research 66, 3 (2022), 367–381.

[47]

Reet Kasepalu, Pankaj Chejara, Luis P. Prieto, and Tobias Ley. 2021. Do teachers find dashboards trustworthy, actionable and useful? A vignette study using a logs and audio dashboard. Technology, Knowledge and Learning (2021), 1–19. DOI:

[48]

Hassan Khosravi, Simon Buckingham Shum, Guanliang Chen, Cristina Conati, Yi-Shan Tsai, Judy Kay, Simon Knight, Roberto Martinez-Maldonado, Shazia Sadiq, and Dragan Gašević. 2022. Explainable artificial intelligence in education. Computers and Education: Artificial Intelligence 3 (2022), 100074.

[49]

Kirsty Kitto, Simon Buckingham Shum, and Andrew Gibson. 2018. Embracing imperfection in learning analytics. In Proceedings of the 8th ACM International Conference on Learning Analytics and Knowledge. ACM, New York, NY, 451–460. DOI:

Digital Library

[50]

Kristian Krogh, Margaret Bearman, and Debra Nestel. 2015. Expert practice of video-assisted debriefing: An Australian qualitative study. Clinical Simulation in Nursing 11, 3 (2015), 180–187.

[51]

Chen-Hsuan Liao and Jiun-Yu Wu. 2022. Deploying multimodal learning analytics models to explore the impact of digital distraction and peer learning on student performance. Computers & Education 190 (December 2022), 104599. DOI:

Digital Library

[52]

Duri Long and Brian Magerko. 2020. What is AI literacy? Competencies and design considerations. In Proceedings of the 2020 SIGCHI Conference on Human Factors in Computing Systems. ACM, 1–16. DOI:

Digital Library

[53]

Gonzalo Luzardo, Bruno Guamán, Katherine Chiluiza, Jaime Castells, and Xavier Ochoa. 2014. Estimation of presentations skills based on slides and audio features. In Proceedings of the 2014 ACM Workshop on Multimodal Learning Analytics Workshop and Grand Challenge. ACM, 37–44. DOI:

Digital Library

[54]

Yingbo Ma, Mehmet Celepkolu, and Kristy Elizabeth Boyer. 2022. Detecting impasse during collaborative problem solving with multimodal learning analytics. In Proceedings of the 12th International Learning Analytics and Knowledge Conference. ACM, New York, NY, 45–55. DOI:

Digital Library

[55]

Aditi Mallavarapu, Leilah Lyons, and Stephen Uzzo. 2022. Exploring the utility of social-network-derived collaborative opportunity temperature readings for informing design and research of large-group immersive learning environments. Journal of Learning Analytics 9, 1 (2022), 53–76.

[56]

Katerina Mangaroska and Michail Giannakos. 2018. Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies 12, 4 (2018), 516–534. DOI:

Digital Library

[57]

Katerina Mangaroska, Roberto Martinez-Maldonado, Boban Vesin, and Dragan Gašević. 2021. Challenges and opportunities of multimodal data in human learning: The computer science students’ perspective. Journal of Computer Assisted Learning 37, 4 (2021), 1030–1047.

[58]

Katerina Mangaroska, Kshitij Sharma, Dragan Gašević, and Michail Giannakos. 2022. Exploring students’ cognitive and affective states during problem solving through multimodal data: Lessons learned from a programming activity. Journal of Computer Assisted Learning 38, 1 (2022), 40–59. DOI:

[59]

Nikki J. Maran and Ronnie J. Glavin. 2003. Low-to high-fidelity simulation–a continuum of medical education? Medical Education 37, S1 (2003), 22–28. DOI:

[60]

Roberto Martinez-Maldonado, Vanessa Echeverria, Gloria Fernandez Nieto, and Simon Buckingham Shum. 2020. From data to insights: A layered storytelling approach for multimodal learning analytics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). ACM, New York, NY, 1–15. DOI:

Digital Library

[61]

Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. Proceedings of the ACM on Human–Computer Interaction 3, CSCW (2019), 1–23. DOI:

Digital Library

[62]

Mary Erickson Megel, Cynthia Bailey, Annette Schnell, Dina Whiteaker, and Angela Vogel. 2013. High-fidelity simulation: How are we using the videos? Clinical Simulation in Nursing 9, 8 (2013), e305–e310. DOI:

[63]

Louis-Philippe Morency, Sharon Oviatt, Stefan Scherer, Nadir Weibel, and Marcelo Worsley. 2013. ICMI 2013 grand challenge workshop on multimodal learning analytics. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction. 373–378.

Digital Library

[64]

Su Mu, Meng Cui, and Xiaodi Huang. 2020. Multimodal data fusion in learning analytics: A systematic review. Sensors 20, 23 (2020), 6856. DOI:

[65]

Michael Muller and Angelika Strohmayer. 2022. Forgetting practices in the data sciences. In Proceedings of the 2022 SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, Article 323, 19 pages. DOI:

Digital Library

[66]

Natsuki Nakayama, Naoko Arakawa, Harumi Ejiri, Reiko Matsuda, and Tsuneko Makino. 2018. Heart rate variability can clarify students’ level of stress during nursing simulation. PLoS One 13, 4 (2018), e0195280.

[67]

Tanya Nazaretsky, Moriah Ariely, Mutlu Cukurova, and Giora Alexandron. 2022. Teachers’ trust in AI-powered educational technology and a professional development program to improve it. British Journal of Educational Technology 53, 4 (2022), 914–931. DOI:

[68]

Tanya Nazaretsky, Mutlu Cukurova, and Giora Alexandron. 2022. An instrument for measuring teachers’ trust in AI-based educational technology. In Proceedings of the 12th International Learning Analytics and Knowledge Conference (Online, USA) (LAK22). ACM, New York, 56–66. DOI:

Digital Library

[69]

Davy Tsz Kit Ng, Jac Ka Lok Leung, Maggie Jiahong Su, Iris Heung Yue Yim, Maggie Shen Qiao, and Samuel Kai Wah Chu. 2022. AI Literacy from Educators’ Perspectives. Springer International Publishing, Cham, 131–139. DOI:

[70]

Omid Noroozi, Iman Alikhani, Sanna Järvelä, Paul A. Kirschner, Ilkka Juuso, and Tapio Seppänen. 2019. Multimodal data to design visual learning analytics for understanding regulation of learning. Computers in Human Behavior 100 (2019), 298–304. DOI:

Digital Library

[71]

Omid Noroozi, Héctor J. Pijeira-Díaz, Marta Sobocinski, Muhterem Dindar, Sanna Järvelä, and Paul A. Kirschner. 2020. Multimodal data indicators for capturing cognitive, motivational, and emotional learning processes: A systematic literature review. Education and Information Technologies 25, 6 (2020), 5499–5547. DOI:

Digital Library

[72]

Xavier Ochoa. 2022. Multimodal Learning Analytics—Rationale, Process, Examples, and Direction (2nd ed.). SoLAR, Vancouver, Canada, 54–65. Retrieved fromhttps://www.solaresearch.org/publications/hla-22/hla22-chapter6/

[73]

Xavier Ochoa and Federico Dominguez. 2020. Controlled evaluation of a multimodal system to improve oral presentation skills in a real learning setting. British Journal of Educational Technology 51, 5 (2020), 1615–1630. DOI:

[74]

Xavier Ochoa, Federico Domínguez, Bruno Guamán, Ricardo Maya, Gabriel Falcones, and Jaime Castells. 2018. The RAP system: Automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors. In Proceedings of the 8th ACM International Conference on Learning Analytics and Knowledge. ACM, Sydney New South Wales Australia, 360–364. DOI:

Digital Library

[75]

Xavier Ochoa and Marcelo Worsley. 2016. Augmenting learning analytics with multimodal sensory data. Journal of Learning Analytics 3, 2 (2016), 213–219.

[76]

Amy Ogan. 2019. Reframing classroom sensing: Promise and peril. Interactions 26, 6 (Oct.2019), 26–32. DOI:

Digital Library

[77]

Leif Oppermann, Alexander Boden, Britta Hofmann, Wolfgang Prinz, and Stefan Decker. 2019. Beyond HCI and CSCW: Challenges and useful practices towards a human-centred vision of AI and IA. In Proceedings of the Halfway to the Future Symposium 2019. 1–5.

Digital Library

[78]

Sharon Oviatt. 2018. Ten opportunities and challenges for advancing student-centered multimodal learning analytics. In Proceedings of the 20th ACM International Conference on Multimodal Interaction. ACM, New York, NY, 87–94. DOI:

Digital Library

[79]

Zacharoula Papamitsiou, Ilias O. Pappas, Kshitij Sharma, and Michail N. Giannakos. 2020. Utilizing multimodal data through fsQCA to explain engagement in adaptive learning. IEEE Transactions on Learning Technologies 13, 4 (2020), 689–703. DOI:

[80]

Stanislav Pozdniakov, Roberto Martinez-Maldonado, Yi-Shan Tsai, Vanessa Echeverria, Namrata Srivastava, and Dragan Gašević. 2023. How do teachers use dashboards enhanced with data storytelling elements according to their data visualisation literacy skills?. In Proceedings of the 13th International Learning Analytics and Knowledge Conference. 1–14.

Digital Library

[81]

Sambit Praharaj, Maren Scheffel, Hendrik Drachsler, and Marcus Specht. 2021. Literature review on co-located collaboration modeling using multimodal learning analytics–can we go the whole nine yards? IEEE Transactions on Learning Technologies 14, 3 (2021), 367–385. DOI:

[82]

Luis Pablo Prieto, Kshitij Sharma, Łukasz Kidzinski, María Jesús Rodríguez-Triana, and Pierre Dillenbourg. 2018. Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data. Journal of Computer Assisted Learning 34, 2 (2018), 193–203. DOI:

[83]

Fabián Riquelme, Rene Noel, Hector Cornide-Reyes, Gustavo Geldes, Cristian Cechinel, Diego Miranda, Rodolfo Villarroel, and Roberto Munoz. 2020. Where are you? Exploring micro-location in indoor learning environments. IEEE Access 8 (2020), 125776–125785.

[84]

Yvonne Rogers and Paul Marshall. 2017. Research in the Wild. Synthesis Lectures on Human-Centered Informatics, Vol. 10. Morgan & Claypool Publishers. i–97 pages. DOI:

[85]

Miguel A. Ronda-Carracao, Olga C. Santos, Gloria Fernandez Nieto, and Roberto Martínez Maldonado. 2021. Towards exploring stress reactions in teamwork using multimodal physiological data. In Proceedings of the First International Workshop on Multimodal Artificial Intelligence in Education (MAIED’21). 49–60.

[86]

John Rooksby. 2013. Wild in the laboratory: A discussion of plans and situated actions. ACM Transactions on Computer–Human Interaction (TOCHI) 20, 3 (2013), 1–17. DOI:

Digital Library

[87]

Selma Šabanović, Sarah M. Reeder, and Bobak Kechavarzi. 2014. Designing robots in the wild: In situ prototype evaluation for a break management robot. Journal of Human–Robot Interaction 3, 1 (2014), 70–88.

Digital Library

[88]

Nazmus Saquib, Ayesha Bose, Dwyane George, and Sepandar Kamvar. 2018. Sensei: Sensing educational interaction. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (Jan2018), 1–27. DOI:

Digital Library

[89]

Aleksandra Sarcevic, Ivan Marsic, and Randal S. Burd. 2012. Teamwork errors in trauma resuscitation. ACM Transactions on Computer–Human Interaction (TOCHI) 19, 2 (2012), 1–30. DOI:

Digital Library

[90]

Stefan Scherer, Marcelo Worsley, and Louis-Philippe Morency. 2012. 1st international workshop on multimodal learning analytics. In Proceedings of the 14th ACM International Conference on Multimodal Interaction. 609–610. DOI:

Digital Library

[91]

Kimberly Schertzer and Muhammad Waseem. 2020. Use of video during debriefing in medical simulation. StatPearls. Treasure Island (FL): StatPearls Publishing; 2020 (2020).

[92]

Bertrand Schneider, Kshitij Sharma, Sébastien Cuendet, Guillaume Zufferey, Pierre Dillenbourg, and Roy Pea. 2016. Using mobile eye-trackers to unpack the perceptual benefits of a tangible user interface for collaborative learning. ACM Transactions on Computer–Human Interaction (TOCHI) 23, 6, Article 39 (Dec.2016), 23 pages. DOI:

Digital Library

[93]

Bertrand Schneider, Kshitij Sharma, Sebastien Cuendet, Guillaume Zufferey, Pierre Dillenbourg, and Roy Pea. 2018. Leveraging mobile eye-trackers to capture joint visual attention in co-located collaborative learning groups. International Journal of Computer-Supported Collaborative Learning 13, 3 (2018), 241–261. DOI:

[94]

Neil Selwyn. 2019. What’s the problem with learning analytics? Journal of Learning Analytics 6, 3 (2019), 11–19. DOI:

[95]

Neil Selwyn and Luci Pangrazio. 2018. Doing data differently? Developing personal data tactics and strategies amongst young mobile media users. Big Data & Society 5, 1 (2018), 2053951718765021. DOI:

[96]

Shashi Kant Shankar, Luis P. Prieto, María Jesús Rodríguez-Triana, and Adolfo Ruiz-Calleja. 2018. A review of multimodal learning analytics architectures. In Proceedings of the IEEE 18th International Conference on Advanced Learning Technologies. IEEE, 212–214. DOI:

[97]

Kshitij Sharma and Michail Giannakos. 2020. Multimodal data capabilities for learning: What can multimodal data tell us about learning? British Journal of Educational Technology 51, 5 (2020), 1450–1484. DOI:

[98]

Ben Shneiderman. 2022. Human-Centered AI. Oxford.

[99]

Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents. Interactions 4, 6 (1997), 42–61. DOI:

Digital Library

[100]

George Siemens. 2012. Learning analytics: Envisioning a research discipline and a domain of practice. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge. 4–8.

Digital Library

[101]

Jocelyn Spence, Benjamin Bedwell, Michelle Coleman, Steve Benford, Boriana N. Koleva, Matt Adams, Ju Row Farr, Nick Tandavanitj, and Anders Sundnes Løvlie. 2019. Seeing with new eyes: Designing for in-the-wild museum gifting. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1–13.

[102]

Daniel Spikol, Emanuele Ruffaldi, Lorenzo Landolfi, and Mutlu Cukurova. 2017. Estimation of success in collaborative learning based on multimodal learning analytics features. In Proceedings of the IEEE 17th International Conference on Advanced Learning Technologies. IEEE, 269–273. DOI:

[103]

Chairs Constantine Stephanidis, Gavriel Salvendy, Members of the Group Margherita Antona, Jessie Y. C. Chen, Jianming Dong, Vincent G. Duffy, Xiaowen Fang, Cali Fidopiastis, Gino Fragomeni, Limin Paul Fu, Yinni Guo, Don Harris, Andri Ioannou, Kyeong-ah (Kate) Jeong, Shin’ichi Konomi, Heidi Krömker, Masaaki Kurosu, James R. Lewis, Aaron Marcus, Gabriele Meiselwitz, Abbas Moallem, Hirohiko Mori, Fiona Fui-Hoon Nah, Stavroula Ntoa, Pei-Luen Patrick Rau, Dylan Schmorrow, Keng Siau, Norbert Streitz, Wentao Wang, Sakae Yamamoto, Panayiotis Zaphiris, and Jia Zhou. 2019. Seven HCI grand challenges. International Journal of Human–Computer Interaction 35, 14 (2019), 1229–1269. DOI:

[104]

Ömer Sümer, Patricia Goldberg, Sidney D’Mello, Peter Gerjets, Ulrich Trautwein, and Enkelejda Kasneci. 2023. Multimodal engagement analysis from facial videos in the classroom. IEEE Transactions on Affective Computing 14, 2 (2023), 1012–1027.

Digital Library

[105]

Nick Taylor, Keith Cheverst, Peter Wright, and Patrick Olivier. 2013. Leaving the wild: Lessons from community technology handovers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1549–1558.

Digital Library

[106]

James Thomas and Angela Harden. 2008. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology 8, 1 (2008), 1–10. DOI:

[107]

Olga Viberg, Ioana Jivet, and Maren Scheffel. 2023. Designing Culturally Aware Learning Analytics: A Value Sensitive Perspective. Springer International Publishing, Cham, 177–192. DOI:

[108]

Clara Vite, Anca-Simona Horvath, Gina Neff, and Naja L. Holten Møller. 2021. Bringing human-centredness to technologies for buildings: An agenda for linking new types of data to the challenge of sustainability. In Proceedings of the 14th Biannual Conference of the Italian SIGCHI Chapter. 1–8.

Digital Library

[109]

Hana Vrzakova, Mary Jean Amon, Angela Stewart, Nicholas D. Duran, and Sidney K. D’Mello. 2020. Focused or stuck together: Multimodal patterns reveal triads’ performance in collaborative problem solving. In Proceedings of the 10th International Learning Analytics and Knowledge Conference. 295–304. DOI:

Digital Library

[110]

Mark Weiser. 1991. The computer for the 21 st century. Scientific American 265, 3 (1991), 94–105.

[111]

Alyssa F. Wise, Simon Knight, and Xavier Ochoa. 2021. What makes learning analytics research matter. Journal of Learning Analytics 8, 3 (2021), 1–9. DOI:

[112]

Marcelo Worsley and Paulo Blikstein. 2015. Leveraging multimodal learning analytics to differentiate student learning strategies. In Proceedings of the 5th International Conference on Learning Analytics and Knowledge. 360–367.

Digital Library

[113]

Marcelo Worsley, Roberto Martinez-Maldonado, and Cynthia D’Angelo. 2021. A new era in multimodal learning analytics: Twelve core commitments to ground and grow MMLA. Journal of Learning Analytics 8, 3 (2021), 10–27. DOI:

[114]

Lixiang Yan, Roberto Martinez-Maldonado, Beatriz Gallo Cordoba, Joanne Deppeler, Deborah Corrigan, Gloria Fernandez Nieto, and Dragan Gašević. 2021. Footprints at school: Modelling in-class social dynamics from students’ physical positioning traces. In Proceedings of the 11th International Learning Analytics and Knowledge Conference. ACM, 43–54. DOI:

Digital Library

[115]

Lixiang Yan, Roberto Martinez-Maldonado, Linxuan Zhao, Samantha Dix, Hollie Jaggard, Rosie Wotherspoon, Xinyu Li, and Dragan Gašević. 2023. The role of indoor positioning analytics in assessment of simulation-based learning. British Journal of Educational Technology 54, 1 (2023), 267–292.

[116]

Lixiang Yan, Roberto Martinez-Maldonado, Linxuan Zhao, Xinyu Li, and Dragan Gašević. 2023. Physiological synchrony and arousal as indicators of stress and learning performance in embodied collaborative learning. In Proceedings of the 24th Conference on Artificial Intelligence in Education. in press.

Digital Library

[117]

Lixiang Yan, Linxuan Zhao, Dragan Gašević, and Roberto Martinez-Maldonado. 2022. Scalability, sustainability, and ethicality of multimodal learning analytics. In Proceedings of the 12th International Learning Analytics and Knowledge Conference. ACM, New York, NY, 13–23. DOI:

Digital Library

[118]

Zhan Zhang and Aleksandra Sarcevic. 2015. Constructing awareness through speech, gesture, gaze and movement during a time-critical medical task. In Proceedings of the 14th European Conference on Computer Supported Cooperative Work, 19–23 September 2015, Oslo, Norway. Springer, 163–182.

[119]

Linxuan Zhao, Zachari Swiecki, Dragan Gasevic, Lixiang Yan, Samantha Dix, Hollie Jaggard, Rosie Wotherspoon, Abra Osborne, Xinyu Li, Riordan Alfredo, and Roberto Martinez-Maldonado. 2023. METS: Multimodal learning analytics of embodied teamwork learning. In Proceedings of the 13th International Learning Analytics and Knowledge Conference. in press. DOI:

Digital Library

Cited By

Spikol DLi ZNolte AOhsaki ARapur K(2024)Investigating Hackathons with Collaboration AnalyticsProceedings of the 8th International Conference on Game Jams, Hackathons and Game Creation Events10.1145/3697789.3697797(1-8)Online publication date: 11-Oct-2024
https://dl.acm.org/doi/10.1145/3697789.3697797
Kitto KGibson A(2024)Places to intervene in complex learning systemsProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636938(929-935)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636938
Zhao LEcheverria VSwiecki ZYan LAlfredo RLi XGasevic DMartinez-Maldonado R(2024)Epistemic Network Analysis for End-users: Closing the Loop in the Context of Multimodal Analytics for Collaborative Team LearningProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636855(90-100)Online publication date: 18-Mar-2024
https://doi.org/10.1145/3636555.3636855
Show More Cited By

Index Terms

Lessons Learnt from a Multimodal Learning Analytics Deployment In-the-Wild
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing systems and tools
  2. Ubiquitous and mobile computing
    1. Empirical studies in ubiquitous and mobile computing

Recommendations

Multimodal learning analytics
LAK '13: Proceedings of the Third International Conference on Learning Analytics and Knowledge

New high-frequency data collection technologies and machine learning analysis techniques could offer new insights into learning, especially in tasks in which students have ample space to generate unique, personalized artifacts, such as a computer ...
Deploying multimodal learning analytics models to explore the impact of digital distraction and peer learning on student performance
Abstract
Social media have been extensively incorporated in higher education as an indispensable tool for learning. Nevertheless, research has conflicting findings about its effectiveness due to the highly reported digital distraction and poor ...
Highlights
- Multimodal data captures course performance in blended Problem-Based Learning.
- ...
Critical success factors for human-centred learning analytics systems design
AfriCHI '23: Proceedings of the 4th African Human Computer Interaction Conference

The benefits of learning analytics have not yet been fully harnessed in developing countries. This is due to low adoption levels, which are caused by various factors, including learning analytics systems design inherent challenges. This study ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer-Human Interaction

ACM Transactions on Computer-Human Interaction Volume 31, Issue 1

February 2024

517 pages

EISSN:1557-7325

DOI:10.1145/3613507

Editors:
Kristina Höök
KTH Royal Institute of Technology, Sweden
,
Kasper Hornbæk
University of Copenhagen, Denmark

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 November 2023

Online AM: 06 September 2023

Accepted: 24 August 2023

Revised: 24 August 2023

Received: 18 December 2022

Published in TOCHI Volume 31, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Australian Government through the Australian Research Counci
Roberto Martinez-Maldonado’s

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
1,021
Total Downloads

Downloads (Last 12 months)807
Downloads (Last 6 weeks)83

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Spikol DLi ZNolte AOhsaki ARapur K(2024)Investigating Hackathons with Collaboration AnalyticsProceedings of the 8th International Conference on Game Jams, Hackathons and Game Creation Events10.1145/3697789.3697797(1-8)Online publication date: 11-Oct-2024
https://dl.acm.org/doi/10.1145/3697789.3697797
Kitto KGibson A(2024)Places to intervene in complex learning systemsProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636938(929-935)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636938
Zhao LEcheverria VSwiecki ZYan LAlfredo RLi XGasevic DMartinez-Maldonado R(2024)Epistemic Network Analysis for End-users: Closing the Loop in the Context of Multimodal Analytics for Collaborative Team LearningProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636855(90-100)Online publication date: 18-Mar-2024
https://doi.org/10.1145/3636555.3636855
Yan LEcheverria VJin YFernandez‐Nieto GZhao LLi XAlfredo RSwiecki ZGašević DMartinez‐Maldonado R(2024)Evidence‐based multimodal learning analytics for feedback and reflection in collaborative learningBritish Journal of Educational Technology10.1111/bjet.1349855:5(1900-1925)Online publication date: 22-Jun-2024
https://doi.org/10.1111/bjet.13498
Zhao LGašević DSwiecki ZLi YLin JSha LYan LAlfredo RLi XMartinez‐Maldonado R(2024)Towards automated transcribing and coding of embodied teamwork communication through multimodal learning analyticsBritish Journal of Educational Technology10.1111/bjet.13476Online publication date: 30-May-2024
https://doi.org/10.1111/bjet.13476
Wang YLai YHuang X(2024)Innovations in Online Learning Analytics: A Review of Recent Research and Emerging TrendsIEEE Access10.1109/ACCESS.2024.349362112(166761-166775)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3493621
Krogstie JSharma K(2024)Enhancing Our Understanding of Business Process Model Comprehension Using Biometric DataEnterprise, Business-Process and Information Systems Modeling10.1007/978-3-031-61007-3_13(159-174)Online publication date: 31-May-2024
https://doi.org/10.1007/978-3-031-61007-3_13
Molnar AGarcia JKostkova P(2024)Learning Analytics for Assessment and Gamification in Digital StorytellingAssessment Analytics in Education10.1007/978-3-031-56365-2_16(313-326)Online publication date: 8-May-2024
https://doi.org/10.1007/978-3-031-56365-2_16

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents