More Web Proxy on the site http://driver.im/

research-article

Open access

Pmomo: Projection Mapping on Movable 3D Object

Authors:

Shuangjiu Xiao,

Xu ChenAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Pages 781 - 790

https://doi.org/10.1145/2858036.2858329

Published: 07 May 2016 Publication History

Abstract

We introduce Pmomo (acronym of projection mapping on movable object), a dynamic projection mapping system that tracks the 6-DOF position of real-world object, and shades it with virtual 3D contents by projection. The system can precisely lock the projection on the moving object in real-time, even the one with complex geometry. Based on depth camera, we developed a novel and robust tracking method that samples the structure of the object into low-density point cloud, then performs an adaptive searching scheme for the registration procedure. As a fully interactive system, our method can handle both internal and external complex occlusions, and can quickly track back the object even when losing track. In order to further improve the realism of the projected virtual textures, our system innovatively culls occlusions away from projection, which is achieved by a facet-covering method. As a result, the Pmomo system enables the possibility of new interactive Augmented Reality applications that require high-quality dynamic projection effect.

Supplementary Material

MP4 File (p781-zhou.mp4)

Download
244.53 MB

References

[1]

Daniel G Aliaga, Yu Hong Yeung, Alvin Law, Behzad Sajadi, and Aditi Majumder. 2012. Fast high-resolution appearance editing using superimposed projections. ACM Transactions on Graphics (TOG) 31, 2 (2012), 13.

Digital Library

[2]

Kuwahara Asai, Sakata Lacroix, Koyaku Kobayashi, Yamato Noda, Hiraoka Nakamura, and Doi Nagashio. 2014. "Face Hacking. (2014). http://www.face-hacking.com

[3]

Deepak Bandyopadhyay, Ramesh Raskar, and Henry Fuchs. 2001. Dynamic shader lamps: Painting on movable objects. In Augmented Reality, 2001. Proceedings. IEEE and ACM International Symposium on. IEEE, 207--216.

Digital Library

[4]

Hrvoje Benko, Ricardo Jota, and Andrew Wilson. 2012. MirageTable: freehand interaction on a projected augmented reality tabletop. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 199--208.

Digital Library

[5]

Hrvoje Benko, Andrew D Wilson, Federico Zannier, and Hrvoje Benko. 2014. Dyadic projected spatial augmented reality. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 645--655.

Digital Library

[6]

Paul J Besl and Neil D McKay. 1992. Method for registration of 3-D shapes. In Robotics-DL tentative. International Society for Optics and Photonics, 586--606.

[7]

Robert Bridson. 2007. Fast Poisson disk sampling in arbitrary dimensions. In ACM SIGGRAPH, Vol. 2007. ACM, 5.

Digital Library

[8]

Daniel Cotting, Martin Naef, Markus Gross, and Henry Fuchs. 2004. Embedding imperceptible patterns into projected images for simultaneous acquisition and display. In Mixed and Augmented Reality, 2004. ISMAR 2004. Third IEEE and ACM International Symposium on. IEEE, 100--109.

Digital Library

[9]

Ali O Ercan, Abbas El Gamal, and Leonidas J Guibas. 2013. Object tracking in the presence of occlusions using multiple cameras: A sensor network approach. ACM Transactions on Sensor Networks (TOSN) 9, 2 (2013), 16.

Digital Library

[10]

Vincent Gay-Bellile, Adrien Bartoli, and Patrick Sayd. 2010. Direct estimation of nonrigid registrations with image-based self-occlusion reasoning. Pattern Analysis and Machine Intelligence, IEEE Transactions on 32, 1 (2010), 87--104.

Digital Library

[11]

Chris Harrison, Hrvoje Benko, and Andrew D Wilson. 2011. OmniTouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 441--450.

Digital Library

[12]

Robert Held, Ankit Gupta, Brian Curless, and Maneesh Agrawala. 2012. 3D puppetry: a kinect-based interface for 3D animation. In UIST. Citeseer, 423--434.

Digital Library

[13]

David Holman, Roel Vertegaal, Mark Altosaar, Nikolaus Troje, and Derek Johns. 2005. Paper windows: interaction techniques for digital paper. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 591--599.

Digital Library

[14]

Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and others. 2011. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 559--568.

Digital Library

[15]

Tyler Johnson and Henry Fuchs. 2007. Real-time projector tracking on complex geometry using ordinary imagery. In Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on. IEEE, 1--8.

[16]

Andreas Jordt and Reinhard Koch. 2013. Direct model-based tracking of 3d object deformations in depth and color video. International journal of computer vision 102, 1--3 (2013), 239--255.

Digital Library

[17]

Shingo Kagami and Koichi Hashimoto. 2015. Sticky projection mapping: 450-fps tracking projection onto a moving planar surface. In SIGGRAPH Asia 2015 Emerging Technologies. ACM, 23.

Digital Library

[18]

Mohammadreza Khalilbeigi, Roman Lissermann, Wolfgang Kleine, and Jurgen Steimle. 2012. FoldMe: interacting with double-sided foldable displays. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction. ACM, 33--40.

Digital Library

[19]

Mohammadreza Khalilbeigi, Roman Lissermann, Max Muhlhauser, and Jurgen Steimle. 2011. Xpaaand: interaction techniques for rollable displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2729--2732.

Digital Library

[20]

Alvin J Law, Daniel G Aliaga, Behzad Sajadi, Aditi Majumder, and Zygmunt Pizlo. 2011. Perceptually based appearance modification for compliant appearance editing. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 2288--2300.

[21]

Johnny C Lee, Scott E Hudson, and Edward Tse. 2008. Foldable interactive displays. In Proceedings of the 21st annual ACM symposium on User interface software and technology. ACM, 287--290.

Digital Library

[22]

David G Lowe. 1999. Object recognition from local scale-invariant features. In Computer vision, 1999. The proceedings of the seventh IEEE international conference on, Vol. 2. Ieee, 1150--1157.

Digital Library

[23]

David Molyneaux, Hans Gellersen, and Joe Finney. 2013. Cooperative augmentation of mobile smart objects with projected displays. ACM Transactions on Interactive Intelligent Systems (TiiS) 3, 2 (2013), 7.

Digital Library

[24]

Daniel Moreno and Gabriel Taubin. 2012. Simple, accurate, and robust projector-camera calibration. In 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), 2012 Second International Conference on. IEEE, 464--471.

Digital Library

[25]

Richard A Newcombe, Dieter Fox, and Steven M Seitz. 2015. DynamicFusion: Reconstruction and Tracking of Non-rigid Scenes in Real-Time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 343--352.

[26]

Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In Mixed and augmented reality (ISMAR), 2011 10th IEEE international symposium on. IEEE, 127--136.

Digital Library

[27]

Kohei Okumura, Hiromasa Oku, and Masatoshi Ishikawa. 2012. Lumipen: Projection-based mixed reality for dynamic objects. In Multimedia and Expo (ICME), 2012 IEEE International Conference on. IEEE, 699--704.

Digital Library

[28]

Christoph Resch, Peter Keitler, and Gudrun Klinker. 2014. Sticky projections: a new approach to interactive shader lamp tracking. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on. IEEE, 151--156.

[29]

Szymon Rusinkiewicz and Marc Levoy. 2001. Efficient variants of the ICP algorithm. In 3-D Digital Imaging and Modeling, 2001. Proceedings. Third International Conference on. IEEE, 145--152.

[30]

Behzad Sajadi, Mahdi Abbaspour Tehrani, Mehdi Rahimzadeh, and Aditi Majumder. 2015. High-resolution lighting of 3D reliefs using a network of projectors and cameras. In 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), 2015. IEEE, 1--4.

[31]

Christian Siegl, Matteo Colaianni, Lucas Thies, Justus Thies, Michael Zollhofer, Shahram Izadi, Marc Stamminger, and Frank Bauer. 2015. Real-time pixel luminance optimization for dynamic multi-projection mapping. ACM Transactions on Graphics (TOG) 34, 6 (2015), 237.

Digital Library

[32]

Martin Spindler and Raimund Dachselt. 2009. PaperLens: advanced magic lens interaction above the tabletop. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces. ACM, 7.

Digital Library

[33]

Jurgen Steimle, Andreas Jordt, and Pattie Maes. 2013. Flexpad: highly flexible bending interactions for projected handheld displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 237--246.

Digital Library

[34]

Jay Summet and Rahul Sukthankar. 2005. Tracking locations of moving hand-held displays using projected light. In Pervasive Computing. Springer, 37--46.

Digital Library

[35]

Junchen Wang, Hiromichi Suenaga, Keika Hoshi, Liangjing Yang, Etsuko Kobayashi, Ichiro Sakuma, and Hongen Liao. 2014. Augmented reality navigation with automatic marker-free image registration using 3-D image overlay for dental surgery. Biomedical Engineering, IEEE Transactions on 61, 4 (2014), 1295--1304.

[36]

Andrew Wilson, Hrvoje Benko, Shahram Izadi, and Otmar Hilliges. 2012. Steerable augmented reality with the beamatron. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 413--422.

Digital Library

[37]

Ruigang Yang and Greg Welch. 2001. Automatic and continuous projector display surface calibration using every-day imagery. In Proc. of 9th International Conf. in Central Europe in Computer Graphics, Visualization, and Computer Vision. Citeseer.

[38]

Zhengyou Zhang. 2000. A flexible new technique for camera calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on 22, 11 (2000), 1330--1334.

Digital Library

[39]

Feng Zheng, Ryan Schubert, and Greg Weich. 2013. A general approach for closed-loop registration in AR. In Virtual Reality (VR), 2013 IEEE. IEEE, 47--50.

Cited By

Ibrahim MGopi MMajumder A(2024)Real-Time Seamless Multi-Projector Displays on Deformable SurfacesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337209730:5(2527-2537)Online publication date: May-2024
https://doi.org/10.1109/TVCG.2024.3372097
Wang JCui HCheng CZhao XYang RYang F(2024)An externally guided spatial augmented reality assembly assistance system in the aviation manufacturing industryThe International Journal of Advanced Manufacturing Technology10.1007/s00170-024-13906-z133:9-10(4893-4906)Online publication date: 26-Jun-2024
https://doi.org/10.1007/s00170-024-13906-z
Lim ANg LGriffin CKryer NBaghernezhad F(2023)Reverse Projection: Real-Time Local Space Texture MappingACM SIGGRAPH 2023 Posters10.1145/3588028.3603653(1-2)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588028.3603653
Show More Cited By

Index Terms

Pmomo: Projection Mapping on Movable 3D Object
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Hand-Eye Camera Calibration with an Optical Tracking System
ICDSC '18: Proceedings of the 12th International Conference on Distributed Smart Cameras

This paper presents a method for hand-eye camera calibration via an optical tracking system (OTS) faciltating robotic applications. The camera pose cannot be directly tracked via the OTS. Because of this, a transformation matrix between a marker-plate ...
Tangible projection mapping: dynamic appearance augmenting of objects in hands
SA '18: SIGGRAPH Asia 2018 Emerging Technologies

We propose a technique of dynamic appearance augmentation by projection to object in user's hands, named Tangible Projection Mapping. This technique allows users to hold a target object freely, and augments appearance of that in various postures by user'...
3D object surface tracking using partial shape templates trained from a depth camera for spatial augmented reality environments
AUIC '13: Proceedings of the Fourteenth Australasian User Interface Conference - Volume 139

We present a 3D object tracking method using a single depth camera for Spatial Augmented Reality (SAR). The drastic change of illumination in a SAR environment makes object tracking difficult. Our method uses a depth camera to train and track the 3D ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

35
Total Citations
View Citations
3,731
Total Downloads

Downloads (Last 12 months)346
Downloads (Last 6 weeks)41

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ibrahim MGopi MMajumder A(2024)Real-Time Seamless Multi-Projector Displays on Deformable SurfacesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337209730:5(2527-2537)Online publication date: May-2024
https://doi.org/10.1109/TVCG.2024.3372097
Wang JCui HCheng CZhao XYang RYang F(2024)An externally guided spatial augmented reality assembly assistance system in the aviation manufacturing industryThe International Journal of Advanced Manufacturing Technology10.1007/s00170-024-13906-z133:9-10(4893-4906)Online publication date: 26-Jun-2024
https://doi.org/10.1007/s00170-024-13906-z
Lim ANg LGriffin CKryer NBaghernezhad F(2023)Reverse Projection: Real-Time Local Space Texture MappingACM SIGGRAPH 2023 Posters10.1145/3588028.3603653(1-2)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588028.3603653
Yanagi RHashimoto AChiba NUshiku YEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Reference-based Dense Pose Estimation via Partial 3D Point Cloud MatchingProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612679(9411-9413)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612679
Macedo MApolinario A(2023)Occlusion Handling in Augmented Reality: Past, Present and FutureIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.311786629:2(1590-1609)Online publication date: 1-Feb-2023
https://doi.org/10.1109/TVCG.2021.3117866
Peng HNishida SWatanabe Y(2023)Studying User Perceptible Misalignment in Simulated Dynamic Facial Projection Mapping2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00064(493-502)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00064
Ibrahim MGopi MMajumder A(2023)Self-Calibrating Dynamic Projection Mapping System for Dynamic, Deformable Surfaces with Jitter Correction and Occlusion Handling2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00044(293-302)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00044
Hu YHu WQuigley A(2023)Towards Using Generative AI for Facilitating Image Creation in Spatial Augmented Reality2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00095(441-443)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00095
Yang FDing XLiu YMa F(2023)Inter-reflection compensation for immersive projection displayMultimedia Tools and Applications10.1007/s11042-023-15973-y83:4(10427-10443)Online publication date: 22-Jun-2023
https://doi.org/10.1007/s11042-023-15973-y
Morosi FCaruso GBecattini NCascini G(2023)Projected Augmented Reality for Industrial Design: Challenges and OpportunitiesTowards a Smart, Resilient and Sustainable Industry10.1007/978-3-031-38274-1_6(61-73)Online publication date: 4-Aug-2023
https://doi.org/10.1007/978-3-031-38274-1_6
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents