More Web Proxy on the site http://driver.im/

research-article

Open access

TipTrack: Precise, Low-Latency, Robust Optical Pen Tracking on Arbitrary Surfaces Using an IR-Emitting Pen Tip

Authors:

Vitus Maierhöfer,

Andreas Schmid,

Raphael WimmerAuthors Info & Claims

TEI '24: Proceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction

Article No.: 18, Pages 1 - 13

https://doi.org/10.1145/3623509.3633366

Published: 11 February 2024 Publication History

All formats PDF

Abstract

Tables are focus points for social interactions and support everyday activities, such as learning, crafting, or dining. These physical interactions on and around the table may be augmented with digital information and tools projected onto the tabletop. For interaction with such projected information, touch input suffers from technical and interactional limitations. Pen input is a more robust alternative that does not suffer from Midas-touch problems. We developed a system for tracking the position of an IR-emitting pen tip on a planar surface with sub-millimeter resolution and an end-to-end latency of less than 30 ms. Distinguishing between drawing and hovering states is done by combining a stereoscopic camera setup and a machine-learning classifier. We demonstrate practical performance, uses and limitations through multiple studies and examples.

Supplemental Material

MP4 File

Project Video

Download
64.49 MB

References

[1]

Ananta Narayanan Balaji, Clayton Kimber, David Li, Shengzhi Wu, Ruofei Du, and David Kim. 2022. RetroSphere: Self-Contained Passive 3D Controller Tracking for Augmented Reality. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 4 (Dec. 2022), 1–36. https://doi.org/10.1145/3569479

Digital Library

[2]

The Holy Bible. n.d. Exodus 25,23. URL: https://www.kingjamesbibleonline.org/Exodus-25-23/.

[3]

Gary Bradski. 2000. The openCV library. Dr. Dobb’s Journal: Software Tools for the Professional Programmer 25, 11 (2000), 120–123. Publisher: Miller Freeman Inc.

[4]

Peter Brandl, Michael Haller, Michael Hurnaus, Verena Lugmayr, Juergen Oberngruber, Claudia Oster, Christian Schafleitner, and Mark Billinghurst. 2007. An Adaptable Rear-Projection Screen Using Digital Pens And Hand Gestures. In Proceedings of the 17th International Conference on Artificial Reality and Telexistence (ICAT 2007). 49–54. https://doi.org/10.1109/ICAT.2007.12

Digital Library

[5]

Rui Chen, Po-Jui (Ray) Chen, Rui Feng, Yilin (Elaine) Liu, Andy Wu, and Ali Mazalek. 2014. SciSketch: a tabletop collaborative sketching system. In Proceedings of the 8th International Conference on Tangible, Embedded and Embodied Interaction - TEI ’14. ACM Press, Munich, Germany, 247–250. https://doi.org/10.1145/2540930.2540973

Digital Library

[6]

M. R. Davis and T. O. Ellis. 1964. The RAND tablet: a man-machine graphical communication device. In Proceedings of the October 27-29, 1964, fall joint computer conference, part I on XX - AFIPS ’64 (Fall, part I). ACM Press, San Francisco, California, 325. https://doi.org/10.1145/1464052.1464080

Digital Library

[7]

Mustafa Doga Dogan, Raul Garcia-Martin, Patrick William Haertel, Jamison John O’Keefe, Ahmad Taka, Akarsh Aurora, Raul Sanchez-Reillo, and Stefanie Mueller. 2023. BrightMarker: 3D Printed Fluorescent Markers for Object Tracking. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. ACM, San Francisco CA USA, 1–13. https://doi.org/10.1145/3586183.3606758

Digital Library

[8]

Mustafa Doga Dogan, Ahmad Taka, Michael Lu, Yunyi Zhu, Akshat Kumar, Aakar Gupta, and Stefanie Mueller. 2022. InfraredTags: Embedding Invisible AR Markers and Barcodes Using Low-Cost, Infrared-Based 3D Printing and Imaging Tools. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems(CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3491102.3501951

Digital Library

[9]

R. G. Enticknap and E. F. Schuster. 1959. Sage data system considerations. Transactions of the American Institute of Electrical Engineers, Part I: Communication and Electronics 77, 6 (Jan. 1959), 824–832. https://doi.org/10.1109/TCE.1959.6372899 Conference Name: Transactions of the American Institute of Electrical Engineers, Part I: Communication and Electronics.

[10]

Andreas Rene Fender, Thomas Roberts, Tiffany Luong, and Christian Holz. 2023. InfinitePaint: Painting in Virtual Reality with Passive Haptics Using Wet Brushes and a Physical Proxy Canvas. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1–13. https://doi.org/10.1145/3544548.3580927

Digital Library

[11]

S. Garrido-Jurado, R. Muñoz-Salinas, F.J. Madrid-Cuevas, and M.J. Marín-Jiménez. 2014. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition 47, 6 (June 2014), 2280–2292. https://doi.org/10.1016/j.patcog.2014.01.005 01071.

Digital Library

[12]

Christian Holz and Patrick Baudisch. 2010. The generalized perceived input point model and how to double touch accuracy by extracting fingerprints. In Proceedings of the 28th international conference on Human factors in computing systems - CHI ’10. ACM Press, Atlanta, Georgia, USA, 581. https://doi.org/10.1145/1753326.1753413

Digital Library

[13]

Fadi Imad. 2018. Real-Time Pen Input System for Writing Utilizing Stereo Vision. Journal of Computers (2018), 1000–1009. https://doi.org/10.17706/jcp.13.9.1000-1009

[14]

Martin Kaltenbrunner and Ross Bencina. 2007. reacTIVision: a computer-vision framework for table-based tangible interaction. In Proceedings of the 1st international conference on Tangible and embedded interaction - TEI ’07. ACM Press, Baton Rouge, Louisiana, 69. https://doi.org/10.1145/1226969.1226983

Digital Library

[15]

David Kim, Shahram Izadi, Jakub Dostal, Christoph Rhemann, Cem Keskin, Christopher Zach, Jamie Shotton, Timothy Large, Steven Bathiche, Matthias Nießner, D. Alex Butler, Sean Fanello, and Vivek Pradeep. 2014. RetroDepth: 3D silhouette sensing for high-precision input on and above physical surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Toronto Ontario Canada, 1377–1386. https://doi.org/10.1145/2556288.2557336

Digital Library

[16]

Johnny Chung Lee. 2008. Hacking the Nintendo Wii Remote. IEEE Pervasive Computing 7, 3 (July 2008), 39–45. https://doi.org/10.1109/MPRV.2008.53 Conference Name: IEEE Pervasive Computing.

Digital Library

[17]

Kuk-Seon Lee, Sang-Heon Oh, Kuk-Hui Jeon, Seong-Soo Kang, Dong-Hee Ryu, and Byung-Gyu Kim. 2012. A Study on Smart Touch Projector System Technology Using Infrared (IR) Imaging Sensor. Journal of Korea Multimedia Society 15, 7 (July 2012), 870–878. https://doi.org/10.9717/KMMS.2012.15.7.870

[18]

Guy Lüthi, Andreas Rene Fender, and Christian Holz. 2022. DeltaPen: A Device with Integrated High-Precision Translation and Rotation Sensing on Passive Surfaces. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. ACM, Bend OR USA, 1–12. https://doi.org/10.1145/3526113.3545655

Digital Library

[19]

I. Scott MacKenzie and R. William Soukoreff. 2003. Phrase sets for evaluating text entry techniques. In CHI ’03 extended abstracts on Human factors in computing systems - CHI ’03. ACM Press, Ft. Lauderdale, Florida, USA, 754. https://doi.org/10.1145/765891.765971

Digital Library

[20]

Vitus Maierhöfer, Andreas Schmid, and Raphael Wimmer. 2022. Demonstration of an Infrared Pen as an Input Device for Projected Augmented Reality Tabletops. In Proceedings of Mensch und Computer 2022(MuC ’22). Association for Computing Machinery, New York, NY, USA, 584–586. https://doi.org/10.1145/3543758.3547519

Digital Library

[21]

G. Margetis, A. Ntelidakis, X. Zabulis, S. Ntoa, P. Koutlemanis, and C. Stephanidis. 2013. Augmenting physical books towards education enhancement. In 2013 1st IEEE Workshop on User-Centered Computer Vision (UCCV). 43–49. https://doi.org/10.1109/UCCV.2013.6530807

[22]

Fabrice Matulic, Riku Arakawa, Brian Vogel, and Daniel Vogel. 2020. PenSight: Enhanced Interaction with a Pen-Top Camera. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376147

Digital Library

[23]

Leo Miyashita, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2018. MIDAS projection: markerless and modelless dynamic projection mapping for material representation. ACM Transactions on Graphics 37, 6 (Dec. 2018), 196:1–196:12. https://doi.org/10.1145/3272127.3275045

Digital Library

[24]

Albert Ng, Michelle Annett, Paul Dietz, Anoop Gupta, and Walter F. Bischof. 2014. In the blink of an eye: investigating latency perception during stylus interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Toronto Ontario Canada, 1103–1112. https://doi.org/10.1145/2556288.2557037

Digital Library

[25]

Andriy Pavlovych and Wolfgang Stuerzlinger. 2009. The tradeoff between spatial jitter and latency in pointing tasks. In Proceedings of the 1st ACM SIGCHI symposium on Engineering interactive computing systems. ACM, Pittsburgh PA USA, 187–196. https://doi.org/10.1145/1570433.1570469

Digital Library

[26]

Gustavo Thebit Pfeiffer, Ricardo Guerra Marroquim, and Antonio Alberto Fernandes de Oliveira. 2014. WebcamPaperPen: A Low-Cost Graphics Tablet. In 2014 27th SIBGRAPI Conference on Graphics, Patterns and Images. 87–94. https://doi.org/10.1109/SIBGRAPI.2014.54 ISSN: 2377-5416.

Digital Library

[27]

Hugo Romat, Andreas Fender, Manuel Meier, and Christian Holz. 2021. Flashpen: A High-Fidelity and High-Precision Multi-Surface Pen for Virtual Reality. In Proceedings of the 2021 IEEE Virtual Reality and 3D User Interfaces (VR). IEEE, Lisboa, Portugal, 306–315. https://doi.org/10.1109/VR50410.2021.00053

[28]

Andreas Schmid and Raphael Wimmer. 2021. Yet Another Latency Measuring Device. preprint. Open Science Framework. https://doi.org/10.31219/osf.io/tkghj

[29]

Andreas Schmid and Raphael Wimmer. 2023. Measuring the Latency of Graphics Frameworks on X11-Based Systems. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems(CHI EA ’23). ACM, Hamburg Germany, 1–7. https://doi.org/10.1145/3544549.3585779

Digital Library

[30]

Hyunyoung Song, Tovi Grossman, George Fitzmaurice, François Guimbretiere, Azam Khan, Ramtin Attar, and Gordon Kurtenbach. 2009. PenLight: combining a mobile projector and a digital pen for dynamic visual overlay. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Boston MA USA, 143–152. https://doi.org/10.1145/1518701.1518726

Digital Library

[31]

Ivan E. Sutherland. 1964. Sketchpad a Man-Machine Graphical Communication System. SIMULATION 2, 5 (May 1964), R–3–R–20. https://doi.org/10.1177/003754976400200514

[32]

Andrey Vakunov, Chuo-Ling Chang, Fan Zhang, George Sung, Matthias Grundmann, and Valentin Bazarevsky. 2020. MediaPipe Hands: On-device Real-time Hand Tracking. In CVPR Workshop on Computer Vision for Augmented and Virtual Reality (CV4ARVR). Seattle, WA, USA. https://doi.org/10.48550/arXiv.2006.10214

[33]

Pierre Wellner. 1993. Interacting with paper on the DigitalDesk. Commun. ACM 36, 7 (July 1993), 87–96. https://doi.org/10.1145/159544.159630

Digital Library

[34]

Raphael Wimmer. 2017. Some Thoughts on Ergonomic, Practical, and Economic Properties of Interactive Tabletops. In Workshop "The Disappearing Tabletop" at ISS 2017. https://epub.uni-regensburg.de/36401/

[35]

Raphael Wimmer, Andreas Schmid, and Florian Bockes. 2019. On the Latency of USB-Connected Input Devices. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland Uk, 1–12. https://doi.org/10.1145/3290605.3300650

Digital Library

[36]

Po-Chen Wu, Robert Wang, Kenrick Kin, Christopher Twigg, Shangchen Han, Ming-Hsuan Yang, and Shao-Yi Chien. 2017. DodecaPen: Accurate 6DoF Tracking of a Passive Stylus. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, Québec City QC Canada, 365–374. https://doi.org/10.1145/3126594.3126664

Digital Library

[37]

Robert Xiao, Scott Hudson, and Chris Harrison. 2016. DIRECT: Making Touch Tracking on Ordinary Surfaces Practical with Hybrid Depth-Infrared Sensing. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces. ACM, Niagara Falls Ontario Canada, 85–94. https://doi.org/10.1145/2992154.2992173

Digital Library

[38]

Tim zum Hoff, Sabrina Großkopp, Robin Neuhaus, Marc Hassenzahl, and Majella Mirjam Lilith Vincent. 2022. Interactive Tables for Social Experiences at Home. In Proceedings of the 16th International Conference on Tangible, Embedded, and Embodied Interaction(TEI ’22). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3490149.3501325

Digital Library

Cited By

Fender AKari M(2024)OptiBasePen: Mobile Base+Pen Input on Passive Surfaces by Sensing Relative Base Motion Plus Close-Range Pen PositionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676467(1-9)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676467

Index Terms

TipTrack: Precise, Low-Latency, Robust Optical Pen Tracking on Arbitrary Surfaces Using an IR-Emitting Pen Tip
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Demonstration of an Infrared Pen as an Input Device for Projected Augmented Reality Tabletops
MuC '22: Proceedings of Mensch und Computer 2022

Interactive tabletops do not only offer a large surface for collaborative interaction. They also offer quick access to digital tools directly at the table - where a large number of everyday activities take place. Tabletops with an embedded display are ...
Pen + Mid-Air: An Exploration of Mid-Air Gestures to Complement Pen Input on Tablets
NordiCHI '16: Proceedings of the 9th Nordic Conference on Human-Computer Interaction

In this paper, we report on a series of studies, exploring the potential of pen and mid-air input on tablets. We describe a field study with an early prototype of a drawing application and follow-up inquiries, such as the development and comparison of ...
Motion and context sensing techniques for pen computing
GI '13: Proceedings of Graphics Interface 2013

We explore techniques for a slender and untethered stylus prototype enhanced with a full suite of inertial sensors (three-axis accelerometer, gyroscope, and magnetometer). We present a taxonomy of enhanced stylus input techniques and consider a number ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

TEI '24: Proceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction

February 2024

1058 pages

ISBN:9798400704024

DOI:10.1145/3623509

General Chairs:
Luigina Ciolfi
University College Cork
,
Trevor Hogan
Munster Technological University
,
Program Chairs:
Tanja Döring
University of Bremen
,
Tom Jenkins
IT University of Copenhagen
,
Jelle van Dijk
University of Twente
,
Samuel Huron
Télécom Paris
,
Zhuying Li
Southeast University
,
Publications Chairs:
David Coyle
University College Dublin
,
Beat Signer
Vrije Universiteit Brussel

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 February 2024

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Data Availability

Project Video https://dl.acm.org/doi/10.1145/3623509.3633366#TipTrack_tei_24.mp4

Funding Sources

Bundesministerium für Bildung und Forschung

Conference

TEI '24

Sponsor:

SIGCHI

TEI '24: Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction

February 11 - 14, 2024

Cork, Ireland

Acceptance Rates

Overall Acceptance Rate 393 of 1,367 submissions, 29%

Upcoming Conference

TEI '25

Sponsor:
sigchi

Nineteenth International Conference on Tangible, Embedded, and Embodied Interaction

March 4 - 7, 2025

Bordeaux / Talence , France

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
469
Total Downloads

Downloads (Last 12 months)469
Downloads (Last 6 weeks)110

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Fender AKari M(2024)OptiBasePen: Mobile Base+Pen Input on Passive Surfaces by Sensing Relative Base Motion Plus Close-Range Pen PositionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676467(1-9)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676467

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents