More Web Proxy on the site http://driver.im/

research-article

Multimodal recognition of reading activity in transit using body-worn sensors

Authors:

Andreas Bulling,

Hans GellersenAuthors Info & Claims

ACM Transactions on Applied Perception (TAP), Volume 9, Issue 1

Article No.: 2, Pages 1 - 21

https://doi.org/10.1145/2134203.2134205

Published: 05 March 2012 Publication History

Abstract

Reading is one of the most well-studied visual activities. Vision research traditionally focuses on understanding the perceptual and cognitive processes involved in reading. In this work we recognize reading activity by jointly analyzing eye and head movements of people in an everyday environment. Eye movements are recorded using an electrooculography (EOG) system; body movements using body-worn inertial measurement units. We compare two approaches for continuous recognition of reading: String matching (STR) that explicitly models the characteristic horizontal saccades during reading, and a support vector machine (SVM) that relies on 90 eye movement features extracted from the eye movement data. We evaluate both methods in a study performed with eight participants reading while sitting at a desk, standing, walking indoors and outdoors, and riding a tram. We introduce a method to segment reading activity by exploiting the sensorimotor coordination of eye and head movements during reading. Using person-independent training, we obtain an average precision for recognizing reading of 88.9% (recall 72.3%) using STR and of 87.7% (recall 87.9%) using SVM over all participants. We show that the proposed segmentation scheme improves the performance of recognizing reading events by more than 24%. Our work demonstrates that the joint analysis of eye and body movements is beneficial for reading recognition and opens up discussion on the wider applicability of a multimodal recognition approach to other visual and physical activities.

References

[1]

Bannach, D., Lukowicz, P., and Amft, O. 2008. Rapid prototyping of activity recognition applications. IEEE Pervasive Comput. 7, 2, 22--31.

Digital Library

[2]

Bao, L. and Intille, S. S. 2004. Activity recognition from user-annotated acceleration data. In Proceedings of the 2nd International Conference on Pervasive Computing. Springer, Berlin, 1--17.

[3]

Barea, R., Boquete, L., Mazo, M., and Lopez, E. 2002. System for assisted mobility using eye movements based on electrooculography. IEEE Trans. Neural Syst. Rehab. Eng. 10, 4, 209--218.

[4]

Biedert, R., Buscher, G., Schwarz, S., Hees, J., and Dengel, A. 2010. Text 2.0. In Extended Abstracts of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, 4003--4008.

Digital Library

[5]

Brown, M., Marmor, M., and Vaegan. 2006. ISCEV standard for clinical electro-oculography (EOG). Documenta Ophthalmologica 113, 3, 205--212.

[6]

Bulling, A., Roggen, D., and Tröster, G. 2009. Wearable EOG goggles: Seamless sensing and context-awareness in everyday environments. J. Ambient Intell. Smart Environ.1, 2, 157--171.

[7]

Bulling, A., Roggen, D., and Tröster, G. 2011. What's in the eyes for context-awareness? IEEE Pervasive Comput. 10, 2, 48--57.

Digital Library

[8]

Bulling, A., Ward, J. A., Gellersen, H., and Tröster, G. 2008. Robust recognition of reading activity in transit using wearable electrooculography. In Proceedings of the 6th International Conference on Pervasive Computing, Springer, Berlin. 19--37.

Digital Library

[9]

Bulling, A., Ward, J. A., Gellersen, H., and Tröster, G. 2011. Eye movement analysis for activity recognition using electrooculography. IEEE Trans. Patt. Anal. Machine Intell. 33, 4, 741--753.

Digital Library

[10]

Campbell, C. S. and Maglio, P. P. 2001. A robust algorithm for reading detection. In Proceedings of the Workshop on Perceptive User Interfaces. ACM, New York, 1--7.

Digital Library

[11]

Canosa, R. L. 2009. Real-world vision: Selective perception and task. ACM Trans. Appl. Percept. 6, 2, 1--34.

Digital Library

[12]

Chen, Y. and Newman, W. S. 2004. A human-robot interface based on electro-oculography. In Proceedings of the International Conference on Robotics and Automation. Vol. 1, 243--248.

[13]

Crammer, K. and Singer, Y. 2003. Ultraconservative online algorithms for multiclass problems. J. Mach. Learn. Res. 3, 951--991.

Digital Library

[14]

Davies, N., Siewiorek, D. P., and Sukthankar, R. 2008. Special issue: Activity-based computing. IEEE Pervasive Comput. 7, 2.

Digital Library

[15]

Dempere-Marco, L., Hu, X., MacDonald, S. L. S., Ellis, S. M., Hansell, D. M., and Yang, G.-Z. 2002. The use of visual search for knowledge gathering in image decision support. IEEE Trans. Syst. Man. Cybern. 22, 3, 741--754.

[16]

Ding, Q., Tong, K., and Li, G. 2005. Development of an EOG (electro-oculography) based human-computer interface. In Proceedings of the 27th Annual International Conference of the Engineering in Medicine and Biology Society. 6829--6831.

[17]

Duchowski, A. T. 2007. Eye-Tracking Methodology: Theory and Practice. Springer, Berlin.

Digital Library

[18]

Elhelw, M., Nicolaou, M., Chung, A., Yang, G.-Z., and Atkins, M. S. 2008. A gaze-based study for investigating the perception of visual realism in simulated scenes. ACM Trans. Appl. Percept. 5, 1, 1--20.

Digital Library

[19]

Hacisalihzade, S. S., Stark, L. W., and Allen, J. S. 1992. Visual perception and sequences of eye movement fixations: A stochastic modeling approach. IEEE Trans. Syst. Man Cybern. 22, 3, 474--481.

[20]

Hayhoe, M. M. and Ballard, D. H. 2005. Eye movements in natural behavior. Trends Cognitive Sci. 9, 188--194.

[21]

Henderson, J. M. 2003. Human gaze control during real-world scene perception. Trends Cognitive Sci. 7, 11, 498--504.

[22]

Huynh, T., Fritz, M., and Schiele, B. 2008. Discovery of activity patterns using topic models. In Proceedings of the 10th International Conference on Ubiquitous Computing. ACM, New York, 10--19.

Digital Library

[23]

Ji, Q. and Yang, X. 2002. Real-time eye, gaze, and face pose tracking for monitoring driver vigilance. Real-Time Imaging 8, 5, 357--377.

Digital Library

[24]

Johansson, R. S., Westling, G., Backstrom, A., and Flanagan, J. R. 2001. Eye-hand coordination in object manipulation. J. Neurosci. 21, 17, 6917--6932.

[25]

Karson, C. N., Berman, K. F., Donnelly, E. F., Mendelson, W. B., Kleinman, J. E., and Wyatt, R. J. 1981. Speaking, thinking, and blinking. Psych. Res. 5, 3, 243--246.

[26]

Keat, F. T., Ranganath, S., and Venkatesh, Y. V. 2003. Eye gaze based reading detection. In Proceedings of the Conference on Convergent Technologies for Asia-Pacific Region. Vol. 2. 825--828.

[27]

Kern, N., Schiele, B., and Schmidt, A. 2007. Recognizing context for annotating a live life recording. Personal Ubiquit. Comput. 11, 4, 251--263.

Digital Library

[28]

Levenshtein, V. I. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Phys. Doklady 10, 8, 707--710.

[29]

Lin, C.-J. 2008. LIBLINEAR - A library for large linear classification. http://www.csie.ntu.edu.tw/~cjlin/liblinear/.

[30]

Liversedge, S. P. and Findlay, J. M. 2000. Saccadic eye movements and cognition. Trends Cognitive Sci. 4, 1, 6--14.

[31]

Logan, B., Healey, J., Philipose, M., Tapia, E., and Intille, S. S. 2007. A long-term evaluation of sensing modalities for activity recognition. In Proceedings of the 9th International Conference on Ubiquitous Computing. ACM, New York, 483--500.

Digital Library

[32]

Maglio, P. P., Matlock, T., Campbell, C. S., Zhai, S., and Smith, B. 2000. Gaze and speech in attentive user interfaces. In Proceedings of the 3rd International Conference on Advances in Multimodal Interfaces. Vol. 1948, 1--7.

Digital Library

[33]

Manabe, H. and Fukumoto, M. 2006. Full-time wearable headphone-type gaze detector. In Extended Abstracts of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, 1073--1078.

Digital Library

[34]

Manor, B. R. and Gordon, E. 2003. Defining the temporal threshold for ocular fixation in free-viewing visuo-cognitive tasks. J. Neurosci. Meth. 128, 1--2, 85--93.

[35]

Mitra, S. and Acharya, T. 2007. Gesture recognition: A survey. IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev. 37, 3, 311--324.

Digital Library

[36]

Najafi, B., Aminian, K., Paraschiv-Ionescu, A., Loew, F., Bula, C. J., and Robert, P. 2003. Ambulatory system for human motion analysis using a kinematic sensor: monitoring of daily physical activity in the elderly. IEEE Trans. Biomed. Eng. 50, 6, 711--723.

[37]

Pelz, J. B., Hayhoe, M. M., and Loeber, R. 2001. The coordination of eye, head, and hand movements in a natural task. Experim. Brain Res. 139, 3, 266--277.

[38]

Peng, H. 2008. mRMR feature selection toolbox for MATLAB. http://penglab.janelia.org/proj/mRMR/.

[39]

Peng, H., Long, F., and Ding, C. 2005. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans. Patt. Anal. Mach. Intell. 27, 8, 1226--1238.

Digital Library

[40]

Rayner, K. 1998. Eye movements in reading and information processing: 20 years of research. Psychol. Bull. 124, 3, 372--422.

[41]

Sailer, U., Flanagan, J. R., and Johansson, R. S. 2005. Eye-hand coordination during learning of a novel visuomotor task. J. Neuroscience 25, 39, 8833--8842.

[42]

Salvucci, D. D. and Anderson, J. R. 2001. Automated eye-movement protocol analysis. Human-Comput. Interact. 16, 1, 39--86.

Digital Library

[43]

Schiffman, H. R. 2001. Sensation and Perception: An Integrated Approach 5th Ed., Wiley, New York.

[44]

Schleicher, R., Galley, N., Briest, S., and Galley, L. 2008. Blinks and saccades as indicators of fatigue in sleepiness warnings: Looking tired? Ergonomics 51, 7, 982--1010.

[45]

Sibert, J. L., Gokturk, M., and Lavine, R. A. 2000. The reading assistant: Eye gaze triggered auditory prompting for reading remediation. In Proceedings of the 13th Symposium on User Interface Software and Technology. ACM, New York, 101--107.

Digital Library

[46]

Tinati, M. A. and Mozaffary, B. 2006. A wavelet packets approach to electrocardiograph baseline drift cancellation. Int. J. Biomed. Imaging Article ID 97157.

[47]

Turaga, P., Chellappa, R., Subrahmanian, V. S., and Udrea, O. 2008. Machine recognition of human activities: A survey. IEEE Trans. Circuits Syst. Video Technol. 18, 11, 1473--1488.

Digital Library

[48]

Vehkaoja, A. T., Verho, J. A., Puurtinen, M. M., Nojd, N. M., Lekkala, J. O., and Hyttinen, J. A. 2005. Wireless head cap for EOG and facial EMG measurements. In Proceedings of the 27th Annual International Conference of the Engineering in Medicine and Biology Society. 5865--5868.

[49]

Ward, J. A., Lukowicz, P., and Gellersen, H. 2011. Performance metrics for activity recognition. ACM Trans. Intell. Syst. Technol. 2, 1, Article 6.

Digital Library

[50]

Ward, J. A., Lukowicz, P., Tröster, G., and Starner, T. E. 2006. Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE Trans. Patt. Anal. Mach. Intell. 28, 10, 1553--1567.

Digital Library

[51]

Widdel, H. 1984., Operational problems in analysing eye movements. In Theoretical and Applied Aspects of Eye Movement Research, Elsevier, Amsterdam, 21--29.

[52]

Wijesoma, W. S., Wee, K. S., Wee, O. C., Balasuriya, A. P., San, K. T., and Soon, K. K. 2005. EOG based control of mobile assistive platforms for the severely disabled. In Proceedings of the International Conference on Robotics and Biomimetics. 490--494.

[53]

Young, S. 2010. Cognitive user interfaces. IEEE Signal Process. 27, 3, 128--140.

Cited By

Kumar GChinnapurapu NSrinivas K(2024)Power and area efficient FIR filter using Radix- 2r multiplier for de-noise the electrooculography (EOG) signalScientific Reports10.1038/s41598-024-73514-514:1Online publication date: 30-Sep-2024
https://doi.org/10.1038/s41598-024-73514-5
Unger MWedel MTuzhilin A(2024)Predicting consumer choice from raw eye-movement data using the RETINA deep learning architectureData Mining and Knowledge Discovery10.1007/s10618-023-00989-738:3(1069-1100)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1007/s10618-023-00989-7
Ye XWang K(2024)Cross-User Activity Recognition via Temporal Relation Optimal TransportMobile and Ubiquitous Systems: Computing, Networking and Services10.1007/978-3-031-63989-0_18(355-374)Online publication date: 19-Jul-2024
https://doi.org/10.1007/978-3-031-63989-0_18
Show More Cited By

Index Terms

Multimodal recognition of reading activity in transit using body-worn sensors

Recommendations

Eye movement analysis for activity recognition
UbiComp '09: Proceedings of the 11th international conference on Ubiquitous computing

In this work we investigate eye movement analysis as a new modality for recognising human activity. We devise 90 different features based on the main eye movement characteristics: saccades, fixations and blinks. The features are derived from eye ...
Smart Eyewear for Interaction and Activity Recognition
CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems

vice class with a lot of possibilities for user interac- tion design and unobtrusive activity tracking. In this paper we show applications using an early prototype of J!NS MEME, smart glasses with integrated electrodes to detect eye movements (...
Analysing EOG signal features for the discrimination of eye movements with wearable devices
PETMEI '11: Proceedings of the 1st international workshop on pervasive eye tracking & mobile eye-based interaction

Eye tracking research in human-computer interaction and experimental psychology traditionally focuses on stationary devices and a small number of common eye movements. The advent of pervasive eye tracking promises new applications, such as eye-based ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Applied Perception

ACM Transactions on Applied Perception Volume 9, Issue 1

March 2012

95 pages

ISSN:1544-3558

EISSN:1544-3965

DOI:10.1145/2134203

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 March 2012

Accepted: 01 January 2011

Revised: 01 January 2011

Received: 01 October 2010

Published in TAP Volume 9, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

79
Total Citations
View Citations
790
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)2

Reflects downloads up to 19 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kumar GChinnapurapu NSrinivas K(2024)Power and area efficient FIR filter using Radix- 2r multiplier for de-noise the electrooculography (EOG) signalScientific Reports10.1038/s41598-024-73514-514:1Online publication date: 30-Sep-2024
https://doi.org/10.1038/s41598-024-73514-5
Unger MWedel MTuzhilin A(2024)Predicting consumer choice from raw eye-movement data using the RETINA deep learning architectureData Mining and Knowledge Discovery10.1007/s10618-023-00989-738:3(1069-1100)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1007/s10618-023-00989-7
Ye XWang K(2024)Cross-User Activity Recognition via Temporal Relation Optimal TransportMobile and Ubiquitous Systems: Computing, Networking and Services10.1007/978-3-031-63989-0_18(355-374)Online publication date: 19-Jul-2024
https://doi.org/10.1007/978-3-031-63989-0_18
Thapa KSeo YYang SKim K(2023)Semi-Supervised Adversarial Auto-Encoder to Expedite Human Activity RecognitionSensors10.3390/s2302068323:2(683)Online publication date: 6-Jan-2023
https://doi.org/10.3390/s23020683
Lei YHe SKhamis MYe J(2023)An End-to-End Review of Gaze Estimation and its Interactive Applications on Handheld Mobile DevicesACM Computing Surveys10.1145/360694756:2(1-38)Online publication date: 15-Sep-2023
https://dl.acm.org/doi/10.1145/3606947
Große-bölting GGerstenberger DGildehaus LMühling ASchulte C(2023)Identity in Higher Computer Education Research: A Systematic Literature ReviewACM Transactions on Computing Education10.1145/360670723:3(1-35)Online publication date: 12-Sep-2023
https://dl.acm.org/doi/10.1145/3606707
Pennekamp JMatzutt RKlinkmüller CBader LSerror MWagner EMalik SSpiß MRahn JGürpinar TVlad ELeemans SKanhere SStich VWehrle K(2023)An Interdisciplinary Survey on Information Flows in Supply ChainsACM Computing Surveys10.1145/360669356:2(1-38)Online publication date: 14-Sep-2023
https://dl.acm.org/doi/10.1145/3606693
Cao YDhekne AAmmar M(2023)ViSigProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35807977:1(1-27)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3580797
Asl EEbadollahi SVahidnia RJalali A(2023)Statistical Database of Human Motion Recognition Using Wearable IoT—A ReviewIEEE Sensors Journal10.1109/JSEN.2023.328217123:14(15253-15304)Online publication date: 15-Jul-2023
https://doi.org/10.1109/JSEN.2023.3282171
Baray SAhmed MChowdhury MKise K(2023)EOG-Based Reading Detection in the Wild Using Spectrograms and Nested Classification ApproachIEEE Access10.1109/ACCESS.2023.331603211(105619-105632)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3316032
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents