More Web Proxy on the site http://driver.im/

research-article

Learning to find eye region landmarks for remote gaze estimation in unconstrained settings

Authors:

Andreas Bulling,

Otmar HilligesAuthors Info & Claims

ETRA '18: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications

Article No.: 21, Pages 1 - 10

https://doi.org/10.1145/3204493.3204545

Published: 14 June 2018 Publication History

Abstract

Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras. In unconstrained real-world settings, however, such methods are surpassed by recent appearance-based methods due to difficulties in modeling factors such as illumination changes and other visual artifacts. We present a novel learning-based method for eye region landmark localization that enables conventional methods to be competitive to latest appearance-based methods. Despite having been trained exclusively on synthetic data, our method exceeds the state of the art for iris localization and eye shape registration on real-world imagery. We then use the detected landmarks as input to iterative model-fitting and lightweight learning-based gaze estimation methods. Our approach outperforms existing model-fitting and appearance-based methods in the context of person-independent and personalized gaze estimation.

Supplementary Material

MP4 File (a21-park.mp4)

Download
31.78 MB

References

[1]

Nuri Murat Arar and Jean-Philippe Thiran. 2017. Robust Real-Time Multi-View Eye Tracking. CoRR Arxiv preprint abs/1711.05444 (2017). arXiv:1711.05444 http://arxiv.org/abs/1711.05444

[2]

Shumeet Baluja and Dean Pomerleau. 1994. Non-Intrusive Gaze Tracking Using Artificial Neural Networks. Technical Report. Pittsburgh, PA, USA.

[3]

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum Learning. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML '09). ACM, New York, NY, USA, 41--48.

Digital Library

[4]

Ralf Biedert, Georg Buscher, Sven Schwarz, Jörn Hees, and Andreas Dengel. 2010. Text 2.0. In CHI'10 Extended Abstracts on Human Factors in Computing Systems. ACM, 4003--4008.

Digital Library

[5]

Wolfgang Fuhl, David Geisler, Thiago Santini, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2016a. Evaluation of State-of-the-art Pupil Detection Algorithms on Remote Eye Images. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp '16). ACM, New York, NY, USA, 1716--1725.

Digital Library

[6]

Wolfgang Fuhl, Thomas Kübler, Katrin Sippel, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2015. ExCuSe: Robust Pupil Detection in Real-World Scenarios. Springer International Publishing, Cham, 39--51.

[7]

Wolfgang Fuhl, Thiago C. Santini, Thomas Kübler, and Enkelejda Kasneci. 2016b. ElSe: Ellipse Selection for Robust Pupil Detection in Real-world Environments. In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications (ETRA '16). ACM, New York, NY, USA, 123--130.

Digital Library

[8]

Kenneth Alberto Funes Mora, Florent Monay, and Jean-Marc Odobez. 2014. EYEDIAP: A Database for the Development and Evaluation of Gaze Estimation Algorithms from RGB and RGB-D Cameras. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '14). ACM, New York, NY, USA, 255--258.

Digital Library

[9]

Kenneth A. Funes-Mora and Jean-Marc Odobez. 2016. Gaze Estimation in the 3D Space Using RGB-D Sensors. International Journal of Computer Vision 118, 2 (01 Jun 2016), 194--216.

Digital Library

[10]

Sina Honari, Pavlo Molchanov, Stephen Tyree, Pascal Vincent, Christopher Pal, and Jan Kautz. 2018. Improving Landmark Localization with Semi-Supervised Learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]

Jia-Bin Huang, Qin Cai, Zicheng Liu, Narendra Ahuja, and Zhengyou Zhang. 2014a. Towards accurate and robust cross-ratio based gaze trackers through learning from simulation. In Eye Tracking Research and Applications, ETRA '14, Safety Harbor, FL, USA, March 26-28, 2014. 75--82.

Digital Library

[12]

Michael Xuelin Huang, Tiffany C.K. Kwok, Grace Ngai, Hong Va Leong, and Stephen C.F. Chan. 2014b. Building a Self-Learning Eye Gaze Model from User Interaction Data. In Proceedings of the 22Nd ACM International Conference on Multimedia (MM '14). ACM, New York, NY, USA, 1017--1020.

Digital Library

[13]

Qiong Huang, Ashok Veeraraghavan, and Ashutosh Sabharwal. 2017. TabletGaze: Dataset and Analysis for Unconstrained Appearance-based Gaze Estimation in Mobile Tablets. Mach. Vision Appl. 28, 5-6 (Aug. 2017), 445--461.

Digital Library

[14]

Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37 (ICML'15). JMLR.org, 448--456. http://dl.acm.org/citation.cfm?id=3045118.3045167

Digital Library

[15]

Davis E. King. 2009. Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 10 (2009), 1755--1758.

Digital Library

[16]

Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014). arXiv:1412.6980 http://arxiv.org/abs/1412.6980

[17]

Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojciech Matusik, and Antonio Torralba. 2016. Eye Tracking for Everyone. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]

Kai Kunze, Andreas Bulling, Yuzuko Utsumi, Shiga Yuki, and Koichi Kise. 2013. I know what you are reading - Recognition of document types using mobile eye tracking. In Proc. IEEE International Symposium on Wearable Computers (ISWC). 113--116.

Digital Library

[19]

Dongheng Li, David Winfield, and Derrick J. Parkhurst. 2005. Starburst: A Hybrid Algorithm for Video-based Eye Tracking Combining Feature-based and Model-based Approaches. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops - Volume 03 (CVPR '05). IEEE Computer Society, Washington, DC, USA, 79--.

Digital Library

[20]

Feng Lu, Takahiro Okabe, Yusuke Sugano, and Yoichi Sato. 2011a. A Head Pose-free Approach for Appearance-based Gaze Estimation. In Proceedings of the British Machine Vision Conference. BMVA Press, 126.1--126.11.

[21]

Feng Lu, Yusuke Sugano, Takahiro Okabe, and Yoichi Sato. 2011b. Inferring Human Gaze from Appearance via Adaptive Linear Regression. In Proceedings of the 2011 International Conference on Computer Vision (ICCV '11). IEEE Computer Society, Washington, DC, USA, 153--160.

Digital Library

[22]

Päivi Majaranta and Andreas Bulling. 2014. Eye Tracking and Eye-Based Human-Computer Interaction. Springer, 39--65.

[23]

Alejandro Newell, Kaiyu Yang, and Jia Deng. 2016. Stacked hourglass networks for human pose estimation. In European Conference on Computer Vision. Springer, 483--499.

[24]

Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic. 2013. 300 faces in-the-wild challenge: The first facial landmark localization challenge. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 397--403.

Digital Library

[25]

Laura Sesma, Arantxa Villanueva, and Rafael Cabeza. 2012. Evaluation of Pupil Center-eye Corner Vector for Gaze Estimation Using a Web Cam. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '12). ACM, New York, NY, USA, 217--220.

Digital Library

[26]

Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. 2017. Learning From Simulated and Unsupervised Images Through Adversarial Training. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]

Brian A. Smith, Qi Yin, Steven K. Feiner, and Shree K. Nayar. 2013. Gaze Locking: Passive Eye Contact Detection for Human-object Interaction. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 271--280.

Digital Library

[28]

Yusuke Sugano, Yasuyuki Matsushita, and Yoichi Sato. 2014. Learning-by-Synthesis for Appearance-Based 3D Gaze Estimation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition. 1821--1828.

Digital Library

[29]

Yusuke Sugano, Xucong Zhang, and Andreas Bulling. 2016. AggreGaze: Collective Estimation of Audience Attention on Public Displays. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 821--831.

Digital Library

[30]

Li Sun, Zicheng Liu, and Ming-Ting Sun. 2015. Real Time Gaze Estimation with a Consumer Depth Camera. Inf. Sci. 320, C (Nov. 2015), 346--360.

Digital Library

[31]

Yi Sun, Xiaogang Wang, and Xiaoou Tang. 2013. Deep Convolutional Network Cascade for Facial Point Detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Digital Library

[32]

Kar-Han Tan, David J. Kriegman, and Narendra Ahuja. 2002. Appearance-based Eye Gaze Estimation. In Proceedings of the Sixth IEEE Workshop on Applications of Computer Vision (WACV '02). IEEE Computer Society, Washington, DC, USA, 191--. http://dl.acm.org/citation.cfm?id=832302.836853

Digital Library

[33]

Fabian Timm and Erhardt Barth. 2011. Accurate Eye Centre Localisation by Means of Gradients. In VISAPP. SciTePress, 125--130. http://dblp.uni-trier.de/db/conf/visapp/visapp2011.html#TimmB11

[34]

Jonathan Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler. 2014. Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1 (NIPS'14). MIT Press, Cambridge, MA, USA, 1799--1807. http://dl.acm.org/citation.cfm?id=2968826.2969027

Digital Library

[35]

Alexander Toshev and Christian Szegedy. 2014. DeepPose: Human Pose Estimation via Deep Neural Networks. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR '14). IEEE Computer Society, Washington, DC, USA, 1653--1660.

Digital Library

[36]

Roberto Valenti, Nicu Sebe, and Theo Gevers. 2012. Combining Head Pose and Eye Location Information for Gaze Estimation. Trans. Img. Proc. 21, 2 (Feb. 2012), 802--815.

Digital Library

[37]

Arantxa Villanueva, Victoria Ponz, Laura Sesma-Sanchez, Mikel Ariz, Sonia Porta, and Rafael Cabeza. 2013. Hybrid Method Based on Topography for Robust Detection of Iris Center and Eye Corners. ACM Trans. Multimedia Comput. Commun. Appl. 9, 4, Article 25 (Aug. 2013), 20 pages.

Digital Library

[38]

Jian-Gang Wang, Eric Sung, and Ronda Venkateswarlu. 2003. Eye gaze estimation from a single image of one eye. In Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on. IEEE, 136--143.

Digital Library

[39]

Kang Wang and Qiang Ji. 2017. Real Time Eye Gaze Tracking with 3D Deformable Eye-Face Model. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV) (ICCV '17). IEEE Computer Society, Washington, DC, USA.

[40]

Erroll Wood, Tadas Baltruaitis, Xucong Zhang, Yusuke Sugano, Peter Robinson, and Andreas Bulling. 2015. Rendering of Eyes for Eye-Shape Registration and Gaze Estimation. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV) (ICCV '15). IEEE Computer Society, Washington, DC, USA, 3756--3764.

Digital Library

[41]

Erroll Wood, Tadas Baltrušaitis, Louis-Philippe Morency, Peter Robinson, and Andreas Bulling. 2016a. A 3D morphable eye region model for gaze estimation. In European Conference on Computer Vision. Springer, 297--313.

[42]

Erroll Wood, Tadas Baltrušaitis, Louis-Philippe Morency, Peter Robinson, and Andreas Bulling. 2016b. Learning an Appearance-based Gaze Estimator from One Million Synthesised Images. In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications (ETRA '16). ACM, New York, NY, USA, 131--138.

Digital Library

[43]

Erroll Wood and Andreas Bulling. 2014. EyeTab: Model-based Gaze Estimation on Unmodified Tablet Computers. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '14). ACM, New York, NY, USA, 207--210.

Digital Library

[44]

Xuehan Xiong, Zicheng Liu, Qin Cai, and Zhengyou Zhang. 2014. Eye Gaze Tracking Using an RGBD Camera: A Comparison with a RGB Solution. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '14 Adjunct). ACM, New York, NY, USA, 1113--1121.

Digital Library

[45]

Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. 2015. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755 (2015).

[46]

Jing Yang, Qingshan Liu, and Kaihua Zhang. 2017. Stacked hourglass network for robust facial landmark localisation. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on. IEEE, 2025--2033.

[47]

Dong Hyun Yoo, Bang Rae Lee, and Myoung Jin Chung. 2002. Non-Contact Eye Gaze Tracking System by Mapping of Corneal Reflections. In Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition (FGR '02). IEEE Computer Society, Washington, DC, USA, 101--.

Digital Library

[48]

Stefanos Zafeiriou, George Trigeorgis, Grigorios Chrysos, Jiankang Deng, and Jie Shen. 2017. The Menpo Facial Landmark Localisation Challenge: A Step Towards the Solution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.

[49]

Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. 2015. Appearance-Based Gaze Estimation in the Wild. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4511--4520.

[50]

Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. 2017. It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 2299--2308.

[51]

Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. 2018. MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2018).

[52]

Yanxia Zhang, Andreas Bulling, and Hans Gellersen. 2013. SideWays: A Gaze Interface for Spontaneous Interaction with Situated Displays. In Proc. of the 31st SIGCHI International Conference on Human Factors in Computing Systems (CHI 2013) (2013-04-27). ACM, New York, NY, USA, 851--860.

Digital Library

[53]

Yanxia Zhang, Andreas Bulling, and Hans Gellersen. 2014. Pupil-canthi-ratio: a calibration-free method for tracking horizontal gaze direction. In Proc. of the 2014 International Working Conference on Advanced Visual Interfaces (AVI 14) (2014-05-27). ACM, New York, NY, USA, 129--132.

Digital Library

Cited By

Shen KLi YGuo ZGao JWu Y(2024)Model-Based 3D Gaze Estimation Using a TOF CameraSensors10.3390/s2404107024:4(1070)Online publication date: 6-Feb-2024
https://doi.org/10.3390/s24041070
Liu JChi JYang Z(2024)A review on personal calibration issues for video-oculographic-based gaze trackingFrontiers in Psychology10.3389/fpsyg.2024.130904715Online publication date: 20-Mar-2024
https://doi.org/10.3389/fpsyg.2024.1309047
Cho CPark SMa SLee HLim EHong S(2024)Feasibility of video-based real-time nystagmus tracking: a lightweight deep learning model approach using ocular object segmentationFrontiers in Neurology10.3389/fneur.2024.134210815Online publication date: 21-Feb-2024
https://doi.org/10.3389/fneur.2024.1342108
Show More Cited By

Index Terms

Learning to find eye region landmarks for remote gaze estimation in unconstrained settings
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques
      1. Pointing

Recommendations

Eye-Model-Based Gaze Estimation by RGB-D Camera
CVPRW '14: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops

This paper proposes a method of eye-model-based gaze estimation by RGB-D camera, Kinect sensor. Different from other methods, our method sets up a model to calibrate the eyeball center by gazing at a target in 3D space, not predefined. And then by ...
Facial Landmarks Based Region-Level Data Augmentation for Gaze Estimation
Advances in Computer Graphics
Abstract
Data augmentation (DA) is an effective technique and is widely used in various deep learning tasks (including gaze estimation). Appearance-based gaze estimation aims to directly learn a mapping from face images to gaze directions. Since subtle ...
Eye-gaze estimation under various head positions and iris states

A new method for eye-gaze estimation under normal head movement is investigated.Head position and orientations are acquired by Kinect depth data and eye direction is obtained from high resolution images.Bayesian multinomial logistic regression is used ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ETRA '18: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications

June 2018

595 pages

ISBN:9781450357067

DOI:10.1145/3204493

Conference Chairs:
Bonita Sharif
Youngstown State University
,
Krzysztof Krejtz
University of Social Sciences and Humanities, Warsaw, Poland

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Pictorial

Author Tags

Qualifiers

Research-article

Funding Sources

JST CREST
ERC

Conference

ETRA '18

Sponsor:

ETRA '18: 2018 Symposium on Eye Tracking Research and Applications

June 14 - 17, 2018

Warsaw, Poland

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

106
Total Citations
View Citations
1,079
Total Downloads

Downloads (Last 12 months)102
Downloads (Last 6 weeks)11

Reflects downloads up to 18 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Shen KLi YGuo ZGao JWu Y(2024)Model-Based 3D Gaze Estimation Using a TOF CameraSensors10.3390/s2404107024:4(1070)Online publication date: 6-Feb-2024
https://doi.org/10.3390/s24041070
Liu JChi JYang Z(2024)A review on personal calibration issues for video-oculographic-based gaze trackingFrontiers in Psychology10.3389/fpsyg.2024.130904715Online publication date: 20-Mar-2024
https://doi.org/10.3389/fpsyg.2024.1309047
Cho CPark SMa SLee HLim EHong S(2024)Feasibility of video-based real-time nystagmus tracking: a lightweight deep learning model approach using ocular object segmentationFrontiers in Neurology10.3389/fneur.2024.134210815Online publication date: 21-Feb-2024
https://doi.org/10.3389/fneur.2024.1342108
Khaleel AAbbas TSami Ibrahim A(2024)Best low-cost methods for real-time detection of the eye and gaze trackingi-com10.1515/icom-2023-002623:1(79-94)Online publication date: 8-Jan-2024
https://doi.org/10.1515/icom-2023-0026
Wang HZhang ZCheng YChang HCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)TextGaze: Gaze-Controllable Face Generation with Natural LanguageProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681252(7143-7151)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681252
Park YManduchi R(2024)A Functional Usability Analysis of Appearance-Based Gaze Tracking for AccessibilityProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3656363(1-7)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649902.3656363
Joshi IGrimmer MRathgeb CBusch CBremond FDantcheva A(2024)Synthetic Data in Human Analysis: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.336282146:7(4957-4976)Online publication date: Jul-2024
https://doi.org/10.1109/TPAMI.2024.3362821
Ghosh SDhall AHayat MKnibbe JJi Q(2024)Automatic Gaze Analysis: A Survey of Deep Learning Based ApproachesIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.332133746:1(61-84)Online publication date: Jan-2024
https://doi.org/10.1109/TPAMI.2023.3321337
Mahmud ZHungler PEtemad A(2024)Multistream Gaze Estimation With Anatomical Eye Region Isolation by Synthetic to Real Transfer LearningIEEE Transactions on Artificial Intelligence10.1109/TAI.2024.33661745:8(4232-4246)Online publication date: Aug-2024
https://doi.org/10.1109/TAI.2024.3366174
Hassan MYin XZhuang YAldridge CMcMurry TSoutherland ARohde G(2024)A Digital Camera-Based Eye Movement Assessment Method for NeuroEye ExaminationIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2023.328594028:2(655-665)Online publication date: Feb-2024
https://doi.org/10.1109/JBHI.2023.3285940
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents