short-paper

Dynamic projection mapping onto a deformable object with occlusion based on high-speed tracking of dot marker array

Authors:

Gaku Narita,

Yoshihiro Watanabe,

Masatoshi IshikawaAuthors Info & Claims

VRST '15: Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology

Pages 149 - 152

https://doi.org/10.1145/2821592.2821618

Published: 13 November 2015 Publication History

Get Access

Abstract

In recent years, projection mapping has attracted much attention in a variety of fields. Generally, however, the objects in projection mapping are limited to rigid and static or quasistatic objects. Dynamic projection mapping onto a deformable object could remarkably expand the possibilities. In order to achieve such a projection mapping, it is necessary to recognize the deformation of the object even when it is occluded. However, it is still a challenging problem to achieve this task in real-time with low latency. In this paper, we propose an efficient, high-speed tracking method utilizing high-frame-rate imaging. Our method is able to track an array of dot markers arranged on a deformable object even when there is external occlusion caused by the user interaction and self-occlusion caused by the deformation of the object itself. Additionally, our method can be applied to a stretchable object. Dynamic projection mapping with our method showed robust and consistent display onto a sheet of paper and cloth with a tracking performance of about 0.2 ms per frame, with the result that the projected pattern appeared to be printed on the deformable object.

Supplementary Material

ZIP File (p149-narita.zip)

Supplemental files.

Download
11.45 MB

References

[1]

Bartoli, A., and Zisserman, A. 2004. Direct estimation of non-rigid registrations. In BMVA '04, 899--908.

Google Scholar

[2]

Fujimoto, Y., Smith, R., Taketomi, T., Yamamoto, G., Miyazaki, J., Kato, H., and Thomas, B. 2014. Geometrically-correct projection-based texture mapping onto a deformable object. IEEE TVCG 20, 4, 540--549.

Digital Library

Google Scholar

[3]

Hilsmann, A., and Eisert, P. 2008. Tracking deformable surfaces with optical flow in the presence of self occlusion in monocular image sequences. In IEEE CVPRW '08, 1--6.

Google Scholar

[4]

Iwai, D., and Sato, K. 2006. Limpid desk: See-through access to disorderly desktop in projection-based mixed reality. In Proc. ACM VRST '06, 112--115.

Digital Library

Google Scholar

[5]

Kato, H., and Billinghurst, M. 1999. Marker tracking and hmd calibration for a video-based augmented reality conferencing system. In IEEE and ACM IWAR '99, 85--94.

Digital Library

Google Scholar

[6]

Khalilbeigi, M., Lissermann, R., Kleine, W., and Steimle, J. 2012. Foldme: Interacting with double-sided foldable displays. In Proc. ACM TEI '12, 33--40.

Digital Library

Google Scholar

[7]

Lin, W. C., and Liu, Y. 2007. A lattice-based mrf model for dynamic near-regular texture tracking. IEEE PAMI 29, 5, 777--792.

Digital Library

Google Scholar

[8]

Ng, A., Lepinski, J., Wigdor, D., Sanders, S., and Dietz, P. 2012. Designing for low-latency direct-touch input. In Proc. ACM UIST '12, 453--464.

Digital Library

Google Scholar

[9]

Okumura, K., Oku, H., and Ishikawa, M. 2012. Lumipen: Projection-based mixed reality for dynamic objects. In IEEE ICME '12, 699--704.

Digital Library

Google Scholar

[10]

Pilet, J., Lepetit, V., and Fua, P. 2008. Fast non-rigid surface detection, registration and realistic augmentation. IJCV 76, 2, 109--122.

Digital Library

Google Scholar

[11]

Punpongsanon, P., Iwai, D., and Sato, K. 2013. Deforme: Projection-based visualization of deformable surfaces using invisible textures. In ACM SIGGRAPH Asia '13 Emerging Technologies, 8:1--8:3.

Digital Library

Google Scholar

[12]

Steimle, J., Jordt, A., and Maes, P. 2013. Flexpad: Highly flexible bending interactions for projected handheld displays. In Proc. ACM CHI '13, 237--246.

Digital Library

Google Scholar

[13]

Uchiyama, H., and Marchand, E. 2011. Deformable random dot markers. In IEEE ISMAR '11, 237--238.

Digital Library

Google Scholar

Cited By

View all

Ibrahim MGopi MMajumder A(2024)Real-Time Seamless Multi-Projector Displays on Deformable SurfacesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337209730:5(2527-2537)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372097
Sueyoshi TMorimoto Y(2023)Automatic Generation of Dynamic Projection Mapping for a Large Number of Elliptic Leaves大量の楕円形の葉を対象とした動的プロジェクションマッピングの自動生成The Journal of the Society for Art and Science10.3756/artsci.22.17_122:5(17_1-17_7)Online publication date: 2023
https://doi.org/10.3756/artsci.22.17_1
Ibrahim MMajumder A(2023)Spatially Augmented Reality on Non-rigid Dynamic Surfaces2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00350(1009-1010)Online publication date: Mar-2023
https://doi.org/10.1109/VRW58643.2023.00350
Show More Cited By

Index Terms

Dynamic projection mapping onto a deformable object with occlusion based on high-speed tracking of dot marker array
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Tracking
      2. Computer vision tasks
        Scene understanding
  2. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
      2. Virtual reality

Recommendations

Dynamic Projection Mapping onto Deforming Non-Rigid Surface Using Deformable Dot Cluster Marker

Dynamic projection mapping for moving objects has attracted much attention in recent years. However, conventional approaches have faced some issues, such as the target objects being limited to rigid objects, and the limited moving speed of the targets. ...
Hand-Eye Camera Calibration with an Optical Tracking System
ICDSC '18: Proceedings of the 12th International Conference on Distributed Smart Cameras

This paper presents a method for hand-eye camera calibration via an optical tracking system (OTS) faciltating robotic applications. The camera pose cannot be directly tracked via the OTS. Because of this, a transformation matrix between a marker-plate ...
Simultaneous Localization, Mapping and Moving Object Tracking

Simultaneous localization, mapping and moving object tracking (SLAMMOT) involves both simultaneous localization and mapping (SLAM) in dynamic environments and detecting and tracking these dynamic objects. In this paper, a mathematical framework is ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

VRST '15: Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology

November 2015

237 pages

ISBN:9781450339902

DOI:10.1145/2821592

Conference Chairs:
Qinping Zhao
Beihang Univerisity
,
Daniel Thalmann
Nanyang Technological University
,
Editor:
Stephen N. Spencer
University of Washington
,
Program Chairs:
Enhua Wu
University of Macau & Institute of Software, Chinese Academy of Sciences
,
Ming C. Lin
University of North Carolina at Chapel Hill
,
Lili Wang
Beihang University

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 November 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

VRST '15

Sponsor:

VRST '15: 21th ACM Symposium on Virtual Reality Software and Technology

November 13 - 15, 2015

Beijing, China

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
705
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)4

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Ibrahim MGopi MMajumder A(2024)Real-Time Seamless Multi-Projector Displays on Deformable SurfacesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337209730:5(2527-2537)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372097
Sueyoshi TMorimoto Y(2023)Automatic Generation of Dynamic Projection Mapping for a Large Number of Elliptic Leaves大量の楕円形の葉を対象とした動的プロジェクションマッピングの自動生成The Journal of the Society for Art and Science10.3756/artsci.22.17_122:5(17_1-17_7)Online publication date: 2023
https://doi.org/10.3756/artsci.22.17_1
Ibrahim MMajumder A(2023)Spatially Augmented Reality on Non-rigid Dynamic Surfaces2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00350(1009-1010)Online publication date: Mar-2023
https://doi.org/10.1109/VRW58643.2023.00350
Ibrahim MGopi MMajumder A(2023)Projector-Camera Calibration on Dynamic, Deformable Surfaces2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00295(905-906)Online publication date: Mar-2023
https://doi.org/10.1109/VRW58643.2023.00295
Macedo MApolinario A(2023)Occlusion Handling in Augmented Reality: Past, Present and FutureIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.311786629:2(1590-1609)Online publication date: 1-Feb-2023
https://doi.org/10.1109/TVCG.2021.3117866
Ibrahim MGopi MMajumder A(2023)Self-Calibrating Dynamic Projection Mapping System for Dynamic, Deformable Surfaces with Jitter Correction and Occlusion Handling2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00044(293-302)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00044
Ibrahim MMajumder AGopi M(2022)Dynamic projection mapping on deformable stretchable materials using boundary trackingComputers and Graphics10.1016/j.cag.2022.01.004103:C(61-74)Online publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1016/j.cag.2022.01.004
Cole AZiauddin SMalcolm JGreenspan M(2022)Efficient Range Sensing Using Imperceptible Structured LightComputer Vision, Imaging and Computer Graphics Theory and Applications10.1007/978-3-030-94893-1_11(241-267)Online publication date: 22-Jan-2022
https://doi.org/10.1007/978-3-030-94893-1_11
Sueyoshi TMorimoto Y(2021)Automatic Generation of Dynamic Projection Mapping for Leaves葉を対象とした動的プロジェクションマッピングの自動生成The Journal of the Society for Art and Science10.3756/artsci.20.2120:1(21-29)Online publication date: 30-Mar-2021
https://doi.org/10.3756/artsci.20.21
Sueyoshi TMorimoto Y(2021)Interactive DPM for Thin Plants with the Latency MeasurementACM SIGGRAPH 2021 Posters10.1145/3450618.3469151(1-2)Online publication date: 5-Aug-2021
https://dl.acm.org/doi/10.1145/3450618.3469151
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Dynamic Projection Mapping onto Deforming Non-Rigid Surface Using Deformable Dot Cluster Marker

Hand-Eye Camera Calibration with an Optical Tracking System

Simultaneous Localization, Mapping and Moving Object Tracking