More Web Proxy on the site http://driver.im/

research-article

Corneal-Imaging Calibration for Optical See-Through Head-Mounted Displays

Authors:

Alexander Plopski,

Christian Nitschke,

Kiyoshi Kiyokawa,

Gudrun Klinker,

Haruo TakemuraAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 21, Issue 4

Pages 481 - 490

https://doi.org/10.1109/TVCG.2015.2391857

Published: 18 April 2015 Publication History

Abstract

In recent years optical see-through head-mounted displays (OST-HMDs) have moved from conceptual research to a market of mass-produced devices with new models and applications being released continuously. It remains challenging to deploy augmented reality (AR) applications that require consistent spatial visualization. Examples include maintenance, training and medical tasks, as the view of the attached scene camera is shifted from the user's view. A calibration step can compute the relationship between the HMD-screen and the user's eye to align the digital content. However, this alignment is only viable as long as the display does not move, an assumption that rarely holds for an extended period of time. As a consequence, continuous recalibration is necessary. Manual calibration methods are tedious and rarely support practical applications. Existing automated methods do not account for user-specific parameters and are error prone. We propose the combination of a pre-calibrated display with a per-frame estimation of the user's cornea position to estimate the individual eye center and continuously recalibrate the system. With this, we also obtain the gaze direction, which allows for instantaneous uncalibrated eye gaze tracking, without the need for additional hardware and complex illumination. Contrary to existing methods, we use simple image processing and do not rely on iris tracking, which is typically noisy and can be ambiguous. Evaluation with simulated and real data shows that our approach achieves a more accurate and stable eye pose estimation, which results in an improved and practical calibration with a largely improved distribution of projection error.

References

[1]

A. Agrawal and S. Ramalingam. Single image calibration of multi-axial imaging systems. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1399–1406, 2013.

[2]

M. Axholt, M. A. Skoglund, S. D. O'Connell, M. D. Cooper, S. R. Ellis, and A. Ynnerman. Parameter estimation variance of the single point active alignment method in optical see-through head mounted display calibration. In Proc. IEEE Virtual Reality Conference (VR), pages 27–34, 2011.

[3]

R. T. Azuma. Predictive Tracking for Augmented Reality. PhD thesis, University of North Carolina, 1995.

[4]

M. Backes, T. Chen, M. Duermuth, H. Lensch, and M. Welk. Tempest in a teapot: Compromising reflections revisited. In Proc. IEEE Symposium on Security and Privacy (SP), pages 315–327, 2009.

[5]

M. Backes, M. Durmuth, and D. Unruh. Compromising reflections-or-how to read lcd monitors around the corner. In Proc. IEEE Symposium on Security and Privacy (SP), pages 158–169, 2008.

[6]

A. Bulling, D. Roggen and G. Tröster. Wearable eog goggles: Seamless sensing and context-awareness in everyday environments. Journal of Ambient Intelligence and Smart Environments, 1(2): 157–171, 2009.

Digital Library

[7]

T. Caudell and D. Mizell. Augmented reality: An application of heads-up display technology to manual manufacturing processes. In Proc. Hawaii International Conference on System Sciences, volume 2, pages 659–669, 1992.

[8]

A. Geiger, F. Moosmann, O. Car, and B. Schuster. Automatic calibration of range and camera sensors using a single shot. In Proc. International Conference on Robotics and Automation (ICRA), pages 3936–3943, 2012.

[9]

Y. Genc F. Sauer, F. Wenzel, M. Tuceryan, and N. Navab. Optical see-through hmd calibration: A stereo method validated with a video see-through system. In Proc. IEEE/ACM International Symposium on Augmented Reality (ISAR), pages 165–174, 2000.

[10]

Y. Gene, M. Tuceryan, and N. Navab. Practical solutions for calibration of optical see-through devices. In Proc. IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR), pages 169–175, 2002.

[11]

E. D. Guestrin and M. Eizenman. General theory of remote gaze estimation using the pupil center and corneal reflections. IEEE Transactions on Biomedical Engineering, 53(6):1124–1133, 2006.

[12]

M. A. Halstead, B. A. Barsky, S. A. Klein, and R. B. Mandell. Reconstructing curved surfaces from specular reflection patterns using spline surface fitting of normals. In Proc. ACM Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pages 335–342, 1996.

[13]

D. Hansen and Q. Ji. In the eye of the beholder: A survey of models for eyes and gaze. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(3):478–500, 2010.

Digital Library

[14]

R. L. Holloway. Registration error analysis for augmented reality. Presence: Teleoperators and Virtual Environments, 6(4):413–432, 1997.

Digital Library

[15]

Y. Ishiguro, A. Mujibiya, T. Miyaki, and J. Rekimoto. Aided eyes: Eye activity sensing for daily life. In Proc. Augmented Human International Conference (AH), pages 25:1–25:7, 2010.

[16]

Y. Itoh and G. Klinker. Interaction-free calibration for optical see-through head-mounted displays based on 3d eye localization. In Proc. IEEE Symposium on 3D User Interfaces (3DUI), pages 75–82, 2014.

[17]

Y. Itoh and G. Klinker. Performance and sensitivity analysis of indica: Interaction-free display calibration for optical see-through head-mounted displays. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 171–176, 2014.

[18]

Y. Itoh and G. Klinker. Light-field correction for spatial calibration of optical see-through head-mounted displays. IEEE Transactions on Visualization and Computer Graphics (Proceedings Virtual Reality 2015), 21(4):471–480, April 2015.

Digital Library

[19]

N. Kishishita, K. Kiyokawa, J. Orlosky, T. Mashita, H. Takemura, and E. Kruijff. Analysing the effects of a wide field of view augmented reality display on search performance in divided attention tasks. In Proc. IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR), pages 177–186, 2014.

[20]

M. Klemm, H. Hoppe, and F. Seebacher. Non-parametric camera-based calibration of optical see-through glasses for augmented reality applications. In Proc. IEEE International Symposium on Mixed and Augmented Reality - Arts, Media, and Humanities (ISMAR-AMH), pages 273–274, 2014.

[21]

G. Liestol and A. Morrison. Views, alignment and incongruity in indirect augmented reality. In Proc. IEEE International Symposium on Mixed and Augmented Reality - Arts, Media, and Humanities (ISMAR-AMH), pages 23–28, 2013.

[22]

P. Maier, A. Dey, C. A. L. Waechter, C. Sandor, M. Tonnis, and G. Klinker. An empiric evaluation of confirmation methods for optical see-through head-mounted display calibration. In Proc. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 267–268, 2011.

[23]

A. Nakazawa and C. Nitschke. Point of gaze estimation through corneal surface reflection in an active illumination environment. In Proc. European Conference on Computer Vision (ECCV), pages 159–172, 2012.

[24]

N. Navab, S. Zokai, Y. Genc and E. M. Coelho. An on-line evaluation system for optical see-through augmented reality. In Proc. IEEE Virtual Reality Conference (VR), pages 245–246, 2004.

[25]

K. Nishino, P. N. Belhumeur, and S. K. Nayar. Using eye reflections for face recognition under varying illumination. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 519–526, 2005.

[26]

K. Nishino and S. K. Nayar. Eyes for relighting. In Proc. ACM Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pages 704–711, 2004.

[27]

K. Nishino and S. K. Nayar. Corneal imaging system: Environment from eyes. International Journal of Computer Vision, 70(1):23–40, 2006.

[28]

C. Nitschke and A. Nakazawa. Super-resolution from corneal images. In Proc. British Machine Vision Conference (BMVC), pages 22.1–22.12, 2012.

[29]

C. Nitschke, A. Nakazawa, and H. Takemura. Display-camera calibration using eye reflections and geometry constraints. Computer Vision and Image Understanding, 115(6):835–853, 2011.

Digital Library

[30]

C. Nitschke, A. Nakazawa, and H. Takemura. Corneal imaging revis-ited: An overview of corneal reflection analysis and applications. IPSJ Transactions on Computer Vision and Applications, 5:1–18, 2013.

[31]

C. B. Owen, J. Zhou, A. Tang, and F. Xiao. Display-relative calibration for optical seethrough head-mounted displays. In Proc. IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR), pages 70–78, 2004.

Digital Library

[32]

H. M. Park, S. H. Lee, and J. S. Choi. Wearable augmented reality system using gaze interaction. In Proc. IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR), pages 175–176, 2008.

[33]

J. P. Rolland, R. L. Holloway, and H. Fuchs. Comparison of optical and video see-through, head-mounted displays. In Proc. Photonics for Industrial Applications (SPIE), pages 293–307, 1995.

[34]

E. Schneider, T. Villgrattner, J. Vockeroth, K. Bartl, S. Kohlbecher, S. Bardins, H. Ulbrich, and T. Brandt. Eyeseecam: An eye movement-driven head camera for the examination of natural visual exploration. Annals of the New York Academy of Sciences, 1164(1):461–467, 2009.

[35]

D. Schnieders, X. Fu, and K.-Y. K. Wong. Reconstruction of display and eyes from a single image. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1442–1449, 2010.

[36]

S.-W. Shih, Y.-T. Wu, and J. Liu. A calibration-free gaze tracking technique. In Proc. International Conference on Pattern Recognition (ICPR), pages 201–204, 2000.

[37]

A. Tsukada and T. Kanade. Automatic acquisition of a 3d eye model for a wearable first-person vision device. In Proc. ACM Symposium on Eye Tracking Research and Applications (ETRA), pages 213–216, 2012.

[38]

A. Tsukada, M. Shino, M. S. Devyver, and T. Kanade. Illumination-free gaze estimation method for first-person vision wearable device. In Proc. IEEE International Conference on Computer Vision Workshops (ICCVW), pages 2084–2091, 2011.

[39]

N. Tsumura, M. N. Dang, T. Makino, and Y. Miyake. Estimating the directions to light sources using images of eye for reconstructing 3D human face. In Proc. IS&TISID Color Imaging Conference (CIC), pages 77–81, 2003.

[40]

M. Tuceryan, Y. Genc and N. Navab. Single-point active alignment method (spaam) for optical see-through hmd calibration for augmented reality. Presence, 11(3):259–276, 2002.

Digital Library

[41]

A. Villanueva and R. Cabeza. A novel gaze estimation system with one calibration point. IEEE Transactions on Systems, Man, and Cybernetics Part B: Cybernetics, 38(4):1123–1138, 2008.

Digital Library

[42]

K.-Y. K. Wong, D. Schnieders, and S. Li. Recovering light directions and camera poses from a single sphere. In Proc. European Conference on Computer Vision (ECCV), pages 631–642, 2008.

[43]

H. Wu, Q. Chen, and T. Wada. Visual direction estimation from a monocular image. IEICE Transactions on Information and Systems, E88-D(10):2277–2285, 2005.

Digital Library

Cited By

Covolan JOliveira CSanches SSementille A(2024)Non-deterministic method for semi-automatic calibration of smartphone-based OST HMDsVirtual Reality10.1007/s10055-024-00978-128:2Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1007/s10055-024-00978-1
Sutton JLanglotz TPlopski A(2022)Seeing Colours: Addressing Colour Vision Deficiency with Vision Augmentations using Computational GlassesACM Transactions on Computer-Human Interaction10.1145/348689929:3(1-53)Online publication date: 14-Jan-2022
https://dl.acm.org/doi/10.1145/3486899
Deng NYe JChen NYang X(2021)Towards Stereoscopic On-vehicle AR-HUDThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-021-02209-z37:9-11(2527-2538)Online publication date: 1-Sep-2021
https://dl.acm.org/doi/10.1007/s00371-021-02209-z
Show More Cited By

Index Terms

Corneal-Imaging Calibration for Optical See-Through Head-Mounted Displays
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
  2. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Virtual reality

Index terms have been assigned to the content through auto-classification.

Recommendations

Calibrating an Optical See-Through Rig with Two Non-overlapping Cameras: The Virtual Camera Framework
3DIMPVT '12: Proceedings of the 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission

We present a novel extrinsic calibration method for optical see-through systems. It is primarily aimed at tablet-like systems with a semi transparent screen, a camera tracking the user position and another camera analyzing the scene but easily ...
An Affordable Solution for Binocular Eye Tracking and Calibration in Head-mounted Displays
MM '15: Proceedings of the 23rd ACM international conference on Multimedia

Immersion is the ultimate goal of head-mounted displays (HMD) for Virtual Reality (VR) in order to produce a convincing user experience. Two important aspects in this context are motion sickness, often due to imprecise calibration, and the integration ...
A Method for Calibrating See-Through Head-Mounted Displays for AR
IWAR '99: Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality

In order to have a working AR system, the see-through system must be calibrated such that the internal models of objects match their physical counterparts. By match, we mean they should have the same position, orientation, and size information as well ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 21, Issue 4

April 2015

119 pages

ISSN:1077-2626

Issue’s Table of Contents

Copyright © 2015.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 18 April 2015

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Covolan JOliveira CSanches SSementille A(2024)Non-deterministic method for semi-automatic calibration of smartphone-based OST HMDsVirtual Reality10.1007/s10055-024-00978-128:2Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1007/s10055-024-00978-1
Sutton JLanglotz TPlopski A(2022)Seeing Colours: Addressing Colour Vision Deficiency with Vision Augmentations using Computational GlassesACM Transactions on Computer-Human Interaction10.1145/348689929:3(1-53)Online publication date: 14-Jan-2022
https://dl.acm.org/doi/10.1145/3486899
Deng NYe JChen NYang X(2021)Towards Stereoscopic On-vehicle AR-HUDThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-021-02209-z37:9-11(2527-2538)Online publication date: 1-Sep-2021
https://dl.acm.org/doi/10.1007/s00371-021-02209-z
Krajancich BKellnhofer PWetzstein G(2020)Optimizing depth perception in virtual and augmented reality through gaze-contingent stereo renderingACM Transactions on Graphics10.1145/3414685.341782039:6(1-10)Online publication date: 27-Nov-2020
https://dl.acm.org/doi/10.1145/3414685.3417820
Utsu TTakemura KKrejtz KSharif B(2019)Remote corneal imaging by integrating a 3D face model and an eyeball modelProceedings of the 11th ACM Symposium on Eye Tracking Research & Applications10.1145/3314111.3319817(1-5)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3314111.3319817
Mahmood ZBibi NUsman MKhan UMuhammad N(2019)Mobile cloud based-framework for sports applicationsMultidimensional Systems and Signal Processing10.1007/s11045-019-00639-630:4(1991-2019)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1007/s11045-019-00639-6
Fountain JSmith S(2019)Detecting rigid links between sensors for automatic sensor space alignment in virtual environmentsVirtual Reality10.1007/s10055-018-0341-823:1(71-84)Online publication date: 5-Mar-2019
https://dl.acm.org/doi/10.1007/s10055-018-0341-8
Langlotz TSutton JZollmann SItoh YRegenbrecht HMandryk RHancock MPerry MCox A(2018)ChromaGlassesProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3173964(1-12)Online publication date: 21-Apr-2018
https://dl.acm.org/doi/10.1145/3173574.3173964
Wagemakers AFafard DStavness IMark GFussell SLampe Cschraefel mHourcade JAppert CWigdor D(2017)Interactive Visual Calibration of Volumetric Head-Tracked 3D DisplaysProceedings of the 2017 CHI Conference on Human Factors in Computing Systems10.1145/3025453.3025685(3943-3953)Online publication date: 2-May-2017
https://dl.acm.org/doi/10.1145/3025453.3025685
Langlotz TCook MRegenbrecht H(2016)Real-Time Radiometric Compensation for Optical See-Through Head-Mounted DisplaysIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2016.259378122:11(2385-2394)Online publication date: 1-Nov-2016
https://dl.acm.org/doi/10.1109/TVCG.2016.2593781
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents