More Web Proxy on the site http://driver.im/

research-article

Multimodal human attention detection for reading

Authors:

Stephen ChanAuthors Info & Claims

SAC '16: Proceedings of the 31st Annual ACM Symposium on Applied Computing

Pages 187 - 192

https://doi.org/10.1145/2851613.2851681

Published: 04 April 2016 Publication History

Abstract

Affective computing in human-computer interaction research enables computers to understand human affects or emotions to provide better service. In this paper, we investigate the detection of human attention useful in intelligent e-learning applications. Our principle is to use only ubiquitous hardware available in most computer systems, namely, webcam and mouse. Information from multiple modalities is fused together for effective human attention detection. We invite human subjects to carry out experiments in reading articles being subjected to different kinds of distraction to induce different attention levels. Machine-learning techniques are applied to identify useful features to recognize human attention level. Our results indicate improved performance with multimodal inputs, suggesting an interesting affective computing direction.

References

[1]

Calvo, R. and Mello, S. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1):18--37, 2010.

Digital Library

[2]

Picard, R. Affective Computing. The MIT Press, 1997.

Digital Library

[3]

Zeng, Z., Pantic, M., Roisman, G. and Huang, T. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1):39--58, 2009.

Digital Library

[4]

Oviatt, S. Multimodal interfaces. Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications, L. Erlbaum Assoc. Inc., pp. 286--304, 2007.

Digital Library

[5]

Cunningham, D. W., Kleiner, M., Bülthoff, H. H. and Wallraven, C. The components of conversational facial expressions. In Proc. of Symposium on Applied Perception in Graphics and Visualization, ACM, pp. 143--150, 2004.

Digital Library

[6]

Ekman, P. Universals and cultural differences in facial expression of emotion. In Proc. of Nebraska Symposium on Motivation, pp. 207--283, 1972.

[7]

Ambadar, Z., Cohn, J. F. and Reed, L. I. All smiles are not created equal: Morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous. Journal of Nonverbal Behavior, 33(1):17--34, 2009.

[8]

Anderson, A., Christenson, S., Sinclair, M. and Lehr, C. Check and connect: The importance of relationships for promoting engagement with school. Journal of School Psychology, 42:95--113, 2004.

[9]

Yamauchi, T. Mouse trajectories and state anxiety: Feature selection with random forest. In Proc. of ACII, pp. 399--404, IEEE, 2013.

Digital Library

[10]

Tsoulouhas, G., Georgiou, D. and Karakos, A. Detection of learners' affective state based on mouse movements. Journal of Computing, 3(11):9--18, 2011.

[11]

Bixler, R. and D'Mello, S. Detecting boredom and engagement during writing with keystroke analysis, task appraisals, and stable traits. In Proc. of IUI, pp. 225--, ACM, 2013.

Digital Library

[12]

Bolt, R. Put-that-there: Voice and gesture at the graphics interface. ACM SIGGRAPH Computer Graphics, 14(3):262--270, 1980.

Digital Library

[13]

Saragih, J. M., Lucey, S. and Cohn, J. F. Deformable model fitting by regularized landmark man-shift. International Journal of Computer Vision, 91(2):200--215, 2011.

Digital Library

[14]

Gross, R., Matthews, I., Cohn, J., Kanade, T. and Baker, S. Multi-PIE. In Proc. of IEEE International Conference on Automatic Face and Gesture Recognition, pp. 1--8, 2008.

[15]

Xiong, X. and De la Torre, F. Supervised descent method and its applications to face alignment. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 532--539, 2013.

Digital Library

[16]

Tsalakanidou, F. and Malassiotis, S. Real-time 2D+3D facial action and expression recognition. Pattern Recognition, 43(5):1763--1775, 2010.

Digital Library

[17]

Talavera, L. An evaluation of filter and wrapper methods for feature selection in categorical clustering. In Proc. of International Symposium on Intelligent Data Analysis, pp. 440--451, 2005.

Digital Library

[18]

Stroupe, G. Comprehension and time difference across no-music, lyrical music, and non-lyrical music groups. Department of Sociology, Emory University, 2005.

[19]

Whitehill, J., Serpell, Z., Lin, Y.-C., Foster, A. and Movellan, J. R. The faces of engagement: Automatic recognition of student engagement from facial expressions. IEEE Transactions on Affective Computing, 5(1):86--98, 2014.

[20]

Fu, Y., Leong, H. V., Ngai, G., Huang, M. X. and Chan, S. C. F. Physiological mouse: Towards an emotion-aware mouse. Universal Access in Information Society: An International Journal, Springer.

Cited By

Sahu PJain K(2024)Sustained attention detection in humans using a prefrontal theta-EEG rhythmCognitive Neurodynamics10.1007/s11571-024-10113-018:5(2675-2687)Online publication date: 3-May-2024
https://doi.org/10.1007/s11571-024-10113-0
Dai LLi YZhang M(2023)Detection of Operator Fatigue in the Main Control Room of a Nuclear Power Plant Based on Eye Blink Rate, PERCLOS and Mouse VelocityApplied Sciences10.3390/app1304271813:4(2718)Online publication date: 20-Feb-2023
https://doi.org/10.3390/app13042718
Heck MJeong JBecker C(2023)Evaluating the Potential of Caption Activation to Mitigate Confusion Inferred from Facial Gestures in Virtual MeetingsProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614142(243-252)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614142
Show More Cited By

Index Terms

Multimodal human attention detection for reading
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms

Recommendations

Multimodal human attention detection for reading from facial expression, eye gaze, and mouse dynamics

Affective computing has recently become an important area in human-computer interaction research. Techniques have been developed to enable computers to understand human affects or emotions, in order to predict human intention more precisely and provide ...
The role of spoken feedback in experiencing multimodal interfaces as human-like
ICMI '03: Proceedings of the 5th international conference on Multimodal interfaces

If user interfaces should be made human-like vs. tool-like has been debated in the HCI field, and this debate affects the development of multimodal interfaces. However, little empirical study has been done to support either view so far. Even if there is ...
Enabling Multimodal Human–Robot Interaction for the Karlsruhe Humanoid Robot

In this paper, we present our work in building technologies for natural multimodal human-robot interaction. We present our systems for spontaneous speech recognition, multimodal dialogue processing, and visual perception of a user, which includes ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '16: Proceedings of the 31st Annual ACM Symposium on Applied Computing

April 2016

2360 pages

ISBN:9781450337397

DOI:10.1145/2851613

Conference Chair:
Sascha Ossowski
University Rey Juan Carlos, Spain

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 April 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SAC 2016

Sponsor:

SIGAPP

SAC 2016: Symposium on Applied Computing

April 4 - 8, 2016

Pisa, Italy

Acceptance Rates

SAC '16 Paper Acceptance Rate 252 of 1,047 submissions, 24%;

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25

Sponsor:
sigapp

The 40th ACM/SIGAPP Symposium on Applied Computing

March 31 - April 4, 2025

Catania , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
220
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)2

Reflects downloads up to 18 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sahu PJain K(2024)Sustained attention detection in humans using a prefrontal theta-EEG rhythmCognitive Neurodynamics10.1007/s11571-024-10113-018:5(2675-2687)Online publication date: 3-May-2024
https://doi.org/10.1007/s11571-024-10113-0
Dai LLi YZhang M(2023)Detection of Operator Fatigue in the Main Control Room of a Nuclear Power Plant Based on Eye Blink Rate, PERCLOS and Mouse VelocityApplied Sciences10.3390/app1304271813:4(2718)Online publication date: 20-Feb-2023
https://doi.org/10.3390/app13042718
Heck MJeong JBecker C(2023)Evaluating the Potential of Caption Activation to Mitigate Confusion Inferred from Facial Gestures in Virtual MeetingsProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614142(243-252)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614142
Rahman SShaikh AMattila HLipping TSari MRaumonen PVirkki J(2023)A Hat-Integrated HCI System for Serious Games–Proof-of-Concept Applications in Focus Detection and Game ControllingGames and Learning Alliance10.1007/978-3-031-49065-1_36(373-382)Online publication date: 29-Nov-2023
https://dl.acm.org/doi/10.1007/978-3-031-49065-1_36
Choi YKim JHong J(2022)Immersion Measurement in Watching Videos Using Eye-tracking DataIEEE Transactions on Affective Computing10.1109/TAFFC.2022.320931113:4(1759-1770)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TAFFC.2022.3209311
Li FXu GFeng S(2021)Eye Tracking Analytics for Mental States Assessment – A Review*2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC52423.2021.9658674(2266-2271)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1109/SMC52423.2021.9658674

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents