short-paper

A classification method of cooking operations based on eye movement patterns

Authors:

Hiroshi MuraseAuthors Info & Claims

ETRA '16: Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications

Pages 205 - 208

https://doi.org/10.1145/2857491.2857500

Published: 14 March 2016 Publication History

Get Access

Abstract

We are developing a cooking support system that coaches beginners. In this work, we focus on eye movement patterns while cooking meals because gaze dynamics include important information for understanding human behavior. The system first needs to classify typical cooking operations. In this paper, we propose a gaze-based classification method and evaluate whether or not the eye movement patterns have a potential to classify the cooking operations. We improve the conventional N-gram model of eye movement patterns, which was designed to be applied for recognition of office work. Conventionally, only relative movement from the previous frame was used as a feature. However, since in cooking, users pay attention to cooking ingredients and equipments, we consider fixation as a component of the N-gram. We also consider eye blinks, which is related to the cognitive state. Compared to the conventional method, instead of focusing on statistical features, we consider the ordinal relations of fixation, blink, and the relative movement. The proposed method estimates the likelihood of the cooking operations by Support Vector Regression (SVR) using frequency histograms of N-grams as explanatory variables.

References

[1]

Bulling, A., Ward, J., Gellersen, H., and Troster, G. 2011. Eye movement analysis for activity recognition using electrooculography. Proc. IEEE Trans. on Pattern Analysis and Machine Intelligence 33, 4 (February), 741--751.

Digital Library

Google Scholar

[2]

Collobert, R., and Bengio, S. 2001. Support vector machines for large-scale regression problems. J. Machine Learning Research 1 (February), 143--160.

Digital Library

Google Scholar

[3]

Doman, K., Kuai, C., Takahashi, T., Ide, I., and Murase, H. 2011. Video CooKing: Towards the synthesis of multimedia cooking recipes. In Proc. 17th Int. Conf. on Multimedia Modeling, 135--145.

Digital Library

Google Scholar

[4]

Hayashi, Y., Doman, K., Ide, I., Deguchi, D., and Murase, H. 2013. Automatic authoring of domestic cooking video based on the description of cooking instructions. In Proc. 5th Int. Workshop on Multimedia for Cooking and Eating Activities, 21--26.

Digital Library

Google Scholar

[5]

Irwin, D. 2004. Fixation location and fixation duration as indices of cognitive processing. In The Interface of Language, Vision, and Action: Eye Movements and the Visual World, Psychological Press, 105--134.

Google Scholar

[6]

Land, M., and Hayhoe, M. 2001. In what ways do eye movements contribute to everyday activities? Vision Research 41, 25 (November), 3559--3565.

Crossref

Google Scholar

[7]

Li, Y., Fathi, A., and Rehg, J. 2013. Learning to predict gaze in egocentric video. In Proc. 2013 IEEE Int. Conf. on Computer Vision, 3216--3223.

Digital Library

Google Scholar

[8]

Matsumura, Y., Hashimoto, A., Mori, S., Mukunoki, M., and Minoh, M. 2015. Clustering scenes in cooking video guided by object access. In Proc. 7th Int. Workshop on Multimedia for Cooking and Eating Activities, 1--6.

Google Scholar

[9]

NAC. http://www.nacinc.com/datasheets/archive/EMR9-Data-Sheet-June-09.pdf. Accessed Jan. 2016.

Google Scholar

[10]

Ogaki, K., Kitani, K., and Sugano, Y. 2012. Coupling eye-motion and ego-motion features for first-person activity recognition. In Proc. IEEE Workshop on Egocentric Vision in Conjunction with CVPR2012, 1--7.

Google Scholar

[11]

Schleicher, R., Galley, N., Briest, S., and Galley, L. 2008. Blinks and saccades as indicators of fatigue in sleepiness warnings: Looking tired? Ergonomics 51, 7 (July), 982--1010.

Crossref

Google Scholar

[12]

Steil, J., and Bulling, A. 2015. Discovery of everyday human activities from long-term visual behavior using topic models. In Proc. 2015 ACM Int. Joint Conf. on Pervasive and Ubiquitous Computing, 75--85.

Digital Library

Google Scholar

[13]

Vinod, D. 1969. Integer programming and the theory of grouping. J. American Statistical Assoc. 64, 326 (June), 506--519.

Crossref

Google Scholar

Cited By

View all

Sauer YAgarwala RLenhart PLendway RSeveritt BNeugebauer AHosp BCastner NWahl S(2024)Eye tracking data set of academics making an omelette: An egg-breaking workProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3655657(1-3)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649902.3655657
Higa SYamada KKamisato S(2023)Intelligent Eye-Controlled Electric Wheelchair Based on Estimating Visual Intentions Using One-Dimensional Convolutional Neural Network and Long Short-Term MemorySensors10.3390/s2308402823:8(4028)Online publication date: 16-Apr-2023
https://doi.org/10.3390/s23084028
Zhang GBortoletto MHu ZShi LBâce MBulling A(2023)Exploring Natural Language Processing Methods for Interactive Behaviour ModellingHuman-Computer Interaction – INTERACT 202310.1007/978-3-031-42286-7_1(3-26)Online publication date: 25-Aug-2023
https://doi.org/10.1007/978-3-031-42286-7_1
Show More Cited By

Index Terms

A classification method of cooking operations based on eye movement patterns
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Activity recognition and understanding

Recommendations

Hierarchical HMM for Eye Movement Classification
Computer Vision – ECCV 2020 Workshops
Abstract
In this work, we tackle the problem of ternary eye movement classification, which aims to separate fixations, saccades and smooth pursuits from the raw eye positional data. The efficient classification of these different types of eye movements ...
Rule-based learning for eye movement type detection
MCPMD '18: Proceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data

Eye movements hold information about human perception, intention, and cognitive state. Various algorithms have been proposed to identify and distinguish eye movements, particularly fixations, saccades, and smooth pursuits. A major drawback of existing ...
Gaze-based interest detection on newspaper articles
PETMEI '18: Proceedings of the 7th Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction

Eye tracking measures have been used to recognize cognitive states involving mental workload, comprehension, and self-confidence in the task of reading. In this paper, we present how these measures can be used to detect the interest of a reader. From the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ETRA '16: Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications

March 2016

378 pages

ISBN:9781450341257

DOI:10.1145/2857491

Conference Chairs:
Pernilla Qvarfordt
FX Palo Alto Laboratory
,
Dan Witzner Hansen
IT University of Copenhagen, Denmark

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 March 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

ETRA '16

Sponsor:

ETRA '16: 2016 Symposium on Eye Tracking Research and Applications

March 14 - 17, 2016

South Carolina, Charleston

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
204
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)4

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Sauer YAgarwala RLenhart PLendway RSeveritt BNeugebauer AHosp BCastner NWahl S(2024)Eye tracking data set of academics making an omelette: An egg-breaking workProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3655657(1-3)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649902.3655657
Higa SYamada KKamisato S(2023)Intelligent Eye-Controlled Electric Wheelchair Based on Estimating Visual Intentions Using One-Dimensional Convolutional Neural Network and Long Short-Term MemorySensors10.3390/s2308402823:8(4028)Online publication date: 16-Apr-2023
https://doi.org/10.3390/s23084028
Zhang GBortoletto MHu ZShi LBâce MBulling A(2023)Exploring Natural Language Processing Methods for Interactive Behaviour ModellingHuman-Computer Interaction – INTERACT 202310.1007/978-3-031-42286-7_1(3-26)Online publication date: 25-Aug-2023
https://doi.org/10.1007/978-3-031-42286-7_1
Ikegami NArnold SNagahama KYamazaki K(2020)Active Learning of the Cutting of Cooking Ingredients using Simulation with Object Splitting2020 IEEE/SICE International Symposium on System Integration (SII)10.1109/SII46433.2020.9025925(1-8)Online publication date: Jan-2020
https://doi.org/10.1109/SII46433.2020.9025925

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Hierarchical HMM for Eye Movement Classification

Rule-based learning for eye movement type detection

Gaze-based interest detection on newspaper articles