Semantic 3D gaze mapping for estimating focused objects

Reviewer: Steve Carson

Knowing where users focus their gaze in real, virtual, augmented, or mixed environments is essential for creating immersive applications in areas like gaming, product design, psychology, and advertising. In multiuser environments, knowing the 3D point-of-gaze (PoG) of each user is a key enabler for immersive interaction. Creating such systems is difficult if they require detailed modeling of a part of the real world and associating its elements with a virtual world, making systems that construct their own models from sensory data highly desirable. Previous research has used a wide variety of hardware (virtual reality headsets with eye trackers, head/body position trackers, video cameras) and software techniques (structure from motion, simultaneous localization and mapping, and key frames) to solve aspects of determining 3D PoG focused objects at the expense of complex and time-consuming setup. The authors cleverly leverage this previous research, combining results from multiple open-source software packages to derive semantic (that is, user attention to objects in the local environment) 3D gaze mapping for estimating user focus without prior information. The main contributions of this paper are its excellent descriptive figures, well-defined algorithms, logical expository flow, and presentation of the mathematical details of computing 3D PoG from multiple 2D PoGs projected into key frames by Delaunay triangulation. The authors also conduct an experiment that shows their method can compute approximate user fixation time on objects, making it suitable for marketing and advertising research.