More Web Proxy on the site http://driver.im/

article

Display-camera calibration using eye reflections and geometry constraints

Authors:

Christian Nitschke,

Atsushi Nakazawa,

Haruo TakemuraAuthors Info & Claims

Computer Vision and Image Understanding, Volume 115, Issue 6

Pages 835 - 853

https://doi.org/10.1016/j.cviu.2011.02.008

Published: 01 June 2011 Publication History

Abstract

In this paper, we describe a novel method for calibrating display-camera setups from reflections in a user's eyes. Combining both devices creates a capable controlled illumination system that enables a range of interesting vision applications in non-professional environments, including object/face reconstruction and human-computer interaction. One major issue barring such systems from average homes is the geometric calibration to obtain the pose of the display which requires special hardware and tedious user interaction. Our proposed approach eliminates this requirement by introducing the novel idea of analyzing screen reflections in the cornea of the human eye, a mirroring device that is always available. We employ a simple shape model to recover pose and reflection characteristics of the eye. Thorough experimental evaluation shows that the basic strategy results in a large error and discusses possible reasons. Based on the findings, a non-linear optimization strategy is developed that exploits geometry constraints within the system to considerably improve the initial estimate. It further allows to automatically resolve an inherent ambiguity that arises in image-based eye pose estimation. The strategy may also be integrated to improve spherical mirror calibration. We describe several comprehensive experimental studies which show that the proposed method performs stably with respect to varying subjects, display poses, eye positions, and gaze directions. The results are feasible and should be sufficient for many applications. In addition, the findings provide general insight on the application of eye reflections for geometric reconstruction.

References

[1]

Abate, A.F., Nappi, M., Riccio, D. and Sabatino, G., 2D and 3D face recognition: a survey. Pattern Recognit. Lett. v28 i14. 1885-1906.

Digital Library

[2]

M. Backes, T. Chen, M. D&#252;rmuth, H.P.A. Lensch, M. Welk, Tempest in a teapot: compromising reflections revisited, in: Proceedings of the IEEE Symposium on Security and Privacy (SP), 2009, pp. 315-327.

Digital Library

[3]

M. Backes, M. D&#252;rmuth, D. Unruh, Compromising reflections - or - how to read LCD monitors around the corner, in: Proceedings of the IEEE Symposium on Security and Privacy (SP), 2008, pp. 158-169.

Digital Library

[4]

Baker, S. and Nayar, S.K., A theory of single-viewpoint catadioptric image formation. Int. J. Comput. Vis. v35 i2. 175-196.

Digital Library

[5]

O. Bimber, G. Wetzstein, A. Emmerling, C. Nitschke, Enabling view-dependent stereoscopic projection in real environments, in: Proceedings of the IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), 2005, pp. 14-23.

Digital Library

[6]

Bogan, S., Waring, G., Ibrahim, O., Drews, C. and Curtis, L., Classification of normal corneal topography based on computer-assisted videokeratography. Arch. Ophthalmol. v108 i7. 945-949.

[7]

T. Bonfort, P. Sturm, P. Gargallo, General specular surface triangulation, in: Proceedings of the Asian Conference on Computer Vision (ACCV), 2006, pp. 872-881.

Digital Library

[8]

J.-Y. Bouguet, Camera Calibration Toolbox for Matlab, 2010. <http://www.vision.caltech.edu/bouguetj/calib_doc/> (last accessed on 18.02.11).

[9]

Bowyer, K.W., Hollingsworth, K. and Flynn, P.J., Image understanding for iris biometrics: a survey. Comput. Vis. Image Underst. v110 i2. 281-307.

Digital Library

[10]

Bradski, G. and Kaehler, A., Learning OpenCV. 2008. first ed. O'Reilly Media, Inc., Sebastopol.

[11]

Clark, J.J., Photometric stereo using LCD displays. Image Vision Comput. v28 i4. 704-714.

Digital Library

[12]

F.L. Coutinho, C.H. Morimoto, Free head motion eye gaze tracking using a single camera and multiple light sources, in: Proceedings of the Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI), 2006, pp. 171-178.

[13]

F.L. Coutinho, C.H. Morimoto, A depth compensation method for cross-ratio based eye tracking, in: Proceedings of the ACM Symposium on Eye Tracking Research and Applications (ETRA), 2010, pp. 137-140.

Digital Library

[14]

Daugman, J., How iris recognition works. IEEE Trans. Circuits Syst. Video Technol. v14 i1. 21-30.

Digital Library

[15]

Duchowski, A.T., A breadth-first survey of eye tracking applications. Beh. Res. Methods Instr. Comput. v34 i4. 455-470.

[16]

Duchowski, A.T., Eye Tracking Methodology: Theory and Practice. 2007. second ed. Springer-Verlag, New York.

Digital Library

[17]

Y. Francken, T. Cuypers, T. Mertens, J. Gielis, P. Bekaert, High quality mesostructure acquisition using specularities, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1-7.

[18]

Y. Francken, C. Hermans, P. Bekaert, Screen-camera calibration using a spherical mirror, in: Proceedings of the IEEE Canadian Conference on Computer and Robot Vision (CRV), 2007, pp. 11-20.

Digital Library

[19]

Y. Francken, C. Hermans, P. Bekaert, Screen-camera calibration using gray codes, in: Proceedings of the IEEE Canadian Conference on Computer and Robot Vision (CRV), 2009, pp. 155-161.

Digital Library

[20]

Y. Francken, C. Hermans, T. Cuypers, P. Bekaert, Fast normal map acquisition using an LCD screen emitting gradient patterns, in: Proceedings of the IEEE Canadian Conference on Computer and Robot Vision (CRV), 2008, pp. 189-195.

Digital Library

[21]

N. Funk, Y.-H. Yang, Using a raster display for photometric stereo, in: Proceedings of the IEEE Canadian Conference on Computer and Robot Vision (CRV), 2007, pp. 201-207.

Digital Library

[22]

Geyer, C. and Daniilidis, K., Catadioptric projective geometry. Int. J. Comput. Vis. v45 i3. 223-243.

Digital Library

[23]

E.D. Guestrin, M. Eizenman, Remote point-of-gaze estimation requiring a single-point calibration for applications with infants, in: Proceedings of the ACM Symposium on Eye Tracking Research &amp; Applications (ETRA), 2008, pp. 267-274.

Digital Library

[24]

M.A. Halstead, B.A. Barsky, S.A. Klein, R.B. Mandell, Reconstructing curved surfaces from specular reflection patterns using spline surface fitting of normals, in: Proceedings of the ACM SIGGRAPH, 1996, pp. 335-342.

Digital Library

[25]

Hammoud, R.I., . 2008. Applications and Experiments, 2008.Springer-Verlag, Berlin/Heidelberg.

Digital Library

[26]

Hansen, D. and Ji, Q., In the eye of the beholder: a survey of models for eyes and gaze. IEEE Trans. Pattern Anal. Mach. Intell. v32 i3. 478-500.

Digital Library

[27]

Hartley, R. and Zisserman, A., Multiple View Geometry in Computer Vision. 2003. second ed. Cambridge University Press, New York.

Digital Library

[28]

Hoffman, D.M., Girshick, A.R., Akeley, K. and Banks, M.S., Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. J. Vis. v8 i3. 1-30.

[29]

Ihrke, I., Kutulakos, K.N., Lensch, H.P.A., Magnor, M. and Heidrich, W., State of the art in transparent and specular object reconstruction. STAR Proc. Eurogr. 87-108.

[30]

N. Iqbal, S.-Y.A. Lee, A study on human gaze estimation using screen reflection, in: Proceedings of the International Conference on Intelligent Data Engineering and Automated Learning (IDEAL), 2008, pp. 104-111.

Digital Library

[31]

Iskander, D.R.A., A parametric approach to measuring limbus corneae from digital images. IEEE Trans. Biomed. Eng. v53 i6. 1134-1140.

[32]

Jaimes, A. and Sebe, N., Multimodal human-computer interaction: a survey. Comput. Vis. Image Underst. v108 i1-2. 116-134.

Digital Library

[33]

M.K. Johnson, H. Farid, Exposing digital forgeries through specular highlights on the eye, in: Proceedings of the Information Hiding: International Workshop (IH), 2007, pp. 311-325.

Digital Library

[34]

Kang, J.J., Eizenman, M., Guestrin, E.D. and Eizenman, E., Investigation of the cross-ratios method for point-of-gaze estimation. IEEE Trans. Biomed. Eng. v55 i9. 2293-2302.

[35]

P.L. Kaufman, A. Alm, Adler's Physiology of the Eye: Clinical Application, 10th ed., Mosby, St. Louis, 2003.

[36]

R.K. Kumar, A. Ilie, J.-M. Frahm, M. Pollefeys, Simple calibration of non-overlapping cameras with a mirror, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1-7.

[37]

Ko, Y.J., Lee, E.C. and Park, K.R., A robust gaze detection method by compensating for facial movements based on corneal specularities. Pattern Recognit. Lett. v29 i10. 1474-1485.

Digital Library

[38]

S. Kuthirummal, S.K. Nayar, Multiview radial catadioptric imaging for scene capture, in: Proceedings of the ACM SIGGRAPH, 2006, pp. 916-923.

Digital Library

[39]

Kutulakos, K.N. and Steger, E., A theory of refractive and specular 3D shape by light-path triangulation. Int. J. Comput. Vis. v76 i1. 13-29.

Digital Library

[40]

Lambooij, M., IJsselsteijn, W., Fortuin, M. and Heynderickx, I., Visual discomfort and visual fatigue of stereoscopic displays: a review. J. Imag. Sci. Technol. v53 i3.

[41]

Mandell, R.B., A guide to videokeratography. Int. Contact Lens Clin. v23 i6. 205-228.

[42]

Morimoto, C.H. and Mimica, M.R., Eye gaze tracking techniques for interactive applications. Comput. Vis. Image Underst. v98 i1. 4-24.

Digital Library

[43]

N.J.W. Morris, K.N. Kutulakos, Reconstructing the surface of inhomogeneous transparent scenes by scatter-trace photography, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2007, pp. 1-8.

[44]

D. Nehab, T. Weyrich, S. Rusinkiewicz, Dense 3D reconstruction from specularity consistency, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1-7.

[45]

K. Nishino, P.N. Belhumeur, S.K. Nayar, Using eye reflections for face recognition under varying illumination, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2005, pp. 519-526.

Digital Library

[46]

K. Nishino, S.K. Nayar, Eyes for relighting, in: Proceedings of the ACM SIGGRAPH, 2004, pp. 704-711.

Digital Library

[47]

Nishino, K. and Nayar, S.K., Corneal imaging system: environment from eyes. Int. J. Comput. Vis. v70 i1. 23-40.

Digital Library

[48]

C. Nitschke, A. Nakazawa, H. Takemura, Display-camera calibration from eye reflections, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2009, pp. 1226-1232.

[49]

C. Nitschke, A. Nakazawa, H. Takemura, Eye reflection analysis and application to display-camera calibration, in: Proceedings of the IEEE International Conference on Image Processing (ICIP), 2009, pp. 3449-3452.

Digital Library

[50]

Porta, M., Vision-based user interfaces: methods and applications. Int. J. Hum. Comput. Stud. v57 i1. 27-73.

[51]

Press, W.H., Vetterling, W.T., Teukolsky, S.A. and Flannery, B.P., Numerical Recipes in C++: The Art of Scientific Computing. 2002. second ed. Cambridge University Press, New York.

Digital Library

[52]

G. Schindler, Photometric stereo via computer screen lighting for real-time surface reconstruction, in: Proceedings of the International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT), 2008.

[53]

D. Schnieders, X. Fu, K.-Y.K. Wong, Reconstruction of display and eyes from a single image, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010, pp. 1442-1449.

[54]

Snell, R.S. and Lemp, M.A., Clinical Anatomy of the Eye. 1998. second ed. Blackwell Publishing, Malden.

[55]

P. Sturm, T. Bonfort, How to compute the pose of an object without a direct view? in: Proceedings of the Asian Conference on Computer Vision (ACCV), 2006, pp. 21-31.

Digital Library

[56]

Sun, B. and Ramamoorthi, R., Affine double- and triple-product wavelet integrals for rendering. ACM Trans. Graph. v28 i2. 1-17.

Digital Library

[57]

Swaminathan, R., Grossberg, M.D. and Nayar, S.K., Non-single viewpoint catadioptric cameras: geometry and analysis. Int. J. Comput. Vis. v66 i3. 211-229.

Digital Library

[58]

Tarini, M., Lensch, H.P., Goesele, M. and Seidel, H., 3D acquisition of mirroring objects using striped patterns. Graph. Models. v67 i4. 233-259.

Digital Library

[59]

N. Tsumura, M.N. Dang, T. Makino, Y. Miyake, Estimating the directions to light sources using images of eye for reconstructing 3D human face, in: Proceedings of the IS&amp;T/SID's Color Imaging Conference, Color Science, Systems and Application, 2003, pp. 77-81.

[60]

Turk, M., Computer vision in the interface. Commun. ACM. v47 i1. 60-67.

Digital Library

[61]

Villanueva, A. and Cabeza, R., Models for gaze tracking systems. J. Image Video Process. v4. 1-16.

[62]

Wang, H., Lin, S., Ye, X. and Gu, W., Separating corneal reflections for illumination estimation. Neurocomputing. v71 i10-12. 1788-1797.

Digital Library

[63]

J.-G. Wang, E. Sung, R. Venkateswarlu, Eye gaze estimation from a single image of one eye, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2003, pp. 136-143.

Digital Library

[64]

Woodham, R.J., Photometric method for determining surface orientation from multiple images. Opt. Eng. v19 i1. 139-144.

[65]

Yoo, D.H. and Chung, M.J., A novel non-intrusive eye gaze estimation using cross-ratio under large head motion. Comput. Vis. Image Underst. v98 i1. 25-51.

Digital Library

[66]

Young, L. and Sheena, D., Survey of eye movement recording methods. Behav. Res. Methods Instrum. v7 i5. 397-429.

[67]

Zhao, W., Chellappa, R., Phillips, P.J. and Rosenfeld, A., Face recognition: a literature survey. ACM Comput. Surv. v35 i4. 399-458.

Digital Library

[68]

D.E. Zongker, D.M. Werner, B. Curless, D.H. Salesin, Environment matting and compositing, in: Proceedings of the ACM SIGGRAPH, (1999) 205-214.

Digital Library

Cited By

Uchida NTakatuka KYamaba HNakazawa AMukunoki MOkazaki N(2019)HMD-based cover test system for the diagnosis of ocular misalignmentArtificial Life and Robotics10.1007/s10015-018-0520-424:3(332-337)Online publication date: 1-Sep-2019
https://dl.acm.org/doi/10.1007/s10015-018-0520-4
Lander Clöchtefeld MKrüger A(2018)hEYEbridProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/31611661:4(1-29)Online publication date: 8-Jan-2018
https://dl.acm.org/doi/10.1145/3161166
Lander CGehring SLöchtefeld MBulling AKrüger AHenze NWoźniak PVäänänen KWilliamson JSchneegass S(2017)EyemirrorProceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia10.1145/3152832.3152839(279-291)Online publication date: 26-Nov-2017
https://dl.acm.org/doi/10.1145/3152832.3152839
Show More Cited By

Display-camera calibration using eye reflections and geometry constraints
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks

Recommendations

Eye reflection analysis and application to display-camera calibration
ICIP'09: Proceedings of the 16th IEEE international conference on Image processing

We present a novel technique for calibrating display-camera systems from reflections in the user's eyes. Display-camera systems enable a range of vision applications that need controlled illumination, including 3D object reconstruction, facial modeling ...
Single camera remote eye gaze tracking under natural head movements
ICAT'06: Proceedings of the 16th international conference on Advances in Artificial Reality and Tele-Existence

Most available gaze tracking systems based on corneal reflection have low tolerance for head movements. In this paper, an eye-gaze tracking method, which can make this system be adaptive to external lighting changes and to the user’s head movements, is ...
Simple, Accurate, and Robust Projector-Camera Calibration
3DIMPVT '12: Proceedings of the 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission

Structured-light systems are simple and effective tools to acquire 3D models. Built with off-the-shelf components, a data projector and a camera, they are easy to deploy and compare in precision with expensive laser scanners. But such a high precision ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Computer Vision and Image Understanding

Computer Vision and Image Understanding Volume 115, Issue 6

June, 2011

182 pages

ISSN:1077-3142

Issue’s Table of Contents

Copyright © Elsevier Inc. © 2011.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 June 2011

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Uchida NTakatuka KYamaba HNakazawa AMukunoki MOkazaki N(2019)HMD-based cover test system for the diagnosis of ocular misalignmentArtificial Life and Robotics10.1007/s10015-018-0520-424:3(332-337)Online publication date: 1-Sep-2019
https://dl.acm.org/doi/10.1007/s10015-018-0520-4
Lander Clöchtefeld MKrüger A(2018)hEYEbridProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/31611661:4(1-29)Online publication date: 8-Jan-2018
https://dl.acm.org/doi/10.1145/3161166
Lander CGehring SLöchtefeld MBulling AKrüger AHenze NWoźniak PVäänänen KWilliamson JSchneegass S(2017)EyemirrorProceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia10.1145/3152832.3152839(279-291)Online publication date: 26-Nov-2017
https://dl.acm.org/doi/10.1145/3152832.3152839
Schneider DGrubert JSubramanian SSteimle JDachselt RMartinez Plasencia DGrossman T(2017)Towards Around-Device Interaction using Corneal ImagingProceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces10.1145/3132272.3134127(287-293)Online publication date: 17-Oct-2017
https://dl.acm.org/doi/10.1145/3132272.3134127
Huang MLi JNgai GLeong HMark GFussell SLampe Cschraefel mHourcade JAppert CWigdor D(2017)ScreenGlintProceedings of the 2017 CHI Conference on Human Factors in Computing Systems10.1145/3025453.3025794(2546-2557)Online publication date: 2-May-2017
https://dl.acm.org/doi/10.1145/3025453.3025794
Plopski AItoh YNitschke CKiyokawa KKlinker GTakemura H(2015)Corneal-Imaging Calibration for Optical See-Through Head-Mounted DisplaysIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2015.239185721:4(481-490)Online publication date: 18-Apr-2015
https://dl.acm.org/doi/10.1109/TVCG.2015.2391857
Takemura KKimura SSuda SQvarfordt PHansen D(2014)Estimating point-of-regard using corneal surface imageProceedings of the Symposium on Eye Tracking Research and Applications10.1145/2578153.2578197(251-254)Online publication date: 26-Mar-2014
https://dl.acm.org/doi/10.1145/2578153.2578197
Nakazawa ANitschke C(2012)Point of Gaze Estimation through Corneal Surface Reflection in an Active Illumination EnvironmentProceedings, Part II, of the 12th European Conference on Computer Vision --- ECCV 2012 - Volume 757310.5555/2964398.2964411(159-172)Online publication date: 7-Oct-2012
https://dl.acm.org/doi/10.5555/2964398.2964411

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents