More Web Proxy on the site http://driver.im/

research-article

You as a Puppet: Evaluation of Telepresence User Interface for Puppetry

Authors:

Mose Sakashita,

Tatsuya Minagawa,

Keisuke Kawahara,

Yoichi OchiaiAuthors Info & Claims

UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

Pages 217 - 228

https://doi.org/10.1145/3126594.3126608

Published: 20 October 2017 Publication History

Abstract

We propose an immersive telepresence system for puppetry that transmits a human performer's body and facial movements into a puppet with audiovisual feedback to the performer. The cameras carried in place of puppet's eyes stream live video to the HMD worn by the performer, so that performers can see the images from the puppet's eyes with their own eyes and have a visual understanding of the puppet's ambience. In conventional methods to manipulate a puppet (a hand-puppet, a string-puppet, and a rod-puppet), there is a need to practice manipulating puppets, and there is difficulty carrying out interactions with the audience. Moreover, puppeteers must be positioned exactly where the puppet is. The proposed system addresses these issues by enabling a human performer to manipulate the puppet remotely using his or her body and facial movements. We conducted several user studies with both beginners and professional puppeteers. The results show that, unlike the conventional method, the proposed system facilitates the manipulation of puppets especially for beginners. Moreover, this system allows performers to enjoy puppetry and fascinate audiences.

Supplementary Material

suppl.mov (uistf1785-file3.mp4)

Supplemental video

Download
55.35 MB

MP4 File (p217-sakashita.mp4)

Download
267.42 MB

References

[1]

E. Blumenthal. 2005. Puppetry and Puppets: An Illustrated World Survey. Thames & Hudson. https://books.google.co.uk/books?id=VkV- QgAACAAJ

[2]

Leonardo Bonanni, Cati Vaucelle, Jeff Lieberman, and Orit Zuckerman. 2006. PlayPals: Tangible Interfaces for Remote Communication and Play. In CHI '06 Extended Abstracts on Human Factors in Computing Systems (CHI EA '06). ACM, NY, NY, USA, 574--579.

Digital Library

[3]

Matthew Brand. 1999. Voice Puppetry. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '99). ACM Press/Addison-Wesley Publishing Co., NY, NY, USA, 21--28.

Digital Library

[4]

L. Fritsche, F. Unverzagt, J. Peters, and R. Calandra. 2015. First-Person Tele-Operation of a Humanoid Robot. In 15th IEEE-RAS International Conference on Humanoid Robots. 997--1002. http://www.ausy.tu-darmstadt.de/ uploads/Site/EditPublication/Fritsche_Humanoids15.pdf

[5]

J.C. Garlen and A.M. Graham. 2009. Kermit Culture: Critical Perspectives on Jim Henson's Muppets. McFarland, Incorporated, Publishers. https://books.google.co.jp/books?id=Rac9xis0BIgC

[6]

A. Gruebler and K. Suzuki. 2010. Measurement of distal EMG signals using a wearable device for reading facial expressions. In 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology. 4594--4597.

[7]

Robert Held, Ankit Gupta, Brian Curless, and Maneesh Agrawala. 2012. 3D Puppetry: A Kinect-based Interface for 3D Animation. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, NY, NY, USA, 423--434.

Digital Library

[8]

Tamara Hunt and Nancy Renfro. 1982. Puppetry in early childhood education. N. Renfro Studios.

[9]

Keisuke Kawahara, Mose Sakashita, Amy Koike, Ippei Suzuki, Kenta Suzuki, and Yoichi Ochiai. 2016. Transformed Human Presence for Puppetry. In Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology (ACE2016). ACM, NY, NY, USA, Article 38, 6 pages.

Digital Library

[10]

Joseph J. LaViola, Jr. 2000. A Discussion of Cybersickness in Virtual Environments. SIGCHI Bull. 32, 1 (Jan. 2000), 47--56.

Digital Library

[11]

Jun Ki Lee, R. L. Toscano, W. D. Stiehl, and C. Breazeal. 2008. The design of a semi-autonomous robot avatar for family communication and education. In RO-MAN 2008 - The 17th IEEE International Symposium on Robot and Human Interactive Communication. 166--173.

[12]

Lui's Leite and Veronica Orvalho. 2012. Shape Your Body: Control a Virtual Silhouette Using Body Motion. In CHI '12 Extended Abstracts on Human Factors in Computing Systems (CHI EA '12). ACM, NY, NY, USA, 1913--1918.

Digital Library

[13]

Luis LEite and Veronica Orvalho. 2017. Mani-Pull-Action: Hand-based Digital Puppetry. Proc. ACM Hum.-Comput. Interact. 1, 1, Article 2 (June 2017), 16 pages.

Digital Library

[14]

B. Y. L. Li, A. S. Mian, W. Liu, and A. Krishna. 2013. Using Kinect for face recognition under varying poses, expressions, illumination and disguise. In Applications of Computer Vision (WACV), 2013 IEEE Workshop on. 186--192.

Digital Library

[15]

H. Li, P. Roivainen, and R. Forcheimer. 1993. 3DD Motion Estimation in Model-Based Facial Image Coding.

[16]

IEEE Trans. Pattern Anal. Mach. Intell. 15, 6 (June 1993), 545--555.

Digital Library

[17]

Hao Li, Laura Trutoiu, Kyle Olszewski, Lingyu Wei, Tristan Trutna, Pei-Lun Hsieh, Aaron Nicholls, and Chongyang Ma. 2015. Facial Performance Sensing Head-mounted Display. ACM Trans. Graph. 34, 4, Article 47 (July 2015), 9 pages.

Digital Library

[18]

Peter Lincoln, Greg Welch, Andrew Nashel, Adrian Ilie, Andrei State, and Henry Fuchs. 2009. Animatronic Shader Lamps Avatars. In Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality (ISMAR '09). IEEE Computer Society, Washington, DC, USA, 27--33.

Digital Library

[19]

Katsutoshi Masai, Yuta Sugiura, Masa Ogata, Katsuhiro Suzuki, Fumihiko Nakamura, Sho Shimamura, Kai Kunze, Masahiko Inami, and Maki Sugimoto. 2015. AffectiveWear: Toward Recognizing Facial Expression. In ACM SIGGRAPH 2015 Posters (SIGGRAPH '15). ACM, NY, NY, USA, Article 16, 1 pages.

Digital Library

[20]

Marvin Minsky. 1980. Telepresence. OMNI magazin, June 1980. (1980).

[21]

Kana Misawa and Jun Rekimoto. 2015. ChameleonMask: Embodied Physical and Social Telepresence Using Human Surrogates. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15). ACM, NY, NY, USA, 401--411.

Digital Library

[22]

Shohei Nagai, Shunichi Kasahara, and Jun Rekimoto. 2015. LiveSphere: Sharing the Surrounding Visual Environment for Immersive Experience in Remote Collaboration. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, NY, NY, USA, 113--116.

Digital Library

[23]

Oculus.inc. 2014. Oculus Rift Develop kit 2 (last accessed August 27, 2016). (2014). https://www.oculus.com/en- us/dk2/

[24]

E.M.B.J. O'Hare. 2005. Puppetry in Education and Therapy: Unlocking Doors to the Mind and Heart. AuthorHouse. https://books.google.co.jp/books?id=7X0CodZfjrcC

[25]

Masaki Oshita, Yuta Senju, and Syun Morishige. 2013. Character Motion Control Interface with Hand Manipulation Inspired by Puppet Mechanism. In Proceedings of the 12th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (VRCAI '13). ACM, NY, NY, USA, 131--138.

Digital Library

[26]

Emil Polyak. 2012. Virtual Impersonation Using Interactive Glove Puppets. In SIGGRAPH Asia 2012 Posters (SA '12). ACM, NY, NY, USA, Article 31, 1 pages.

Digital Library

[27]

Daisuke Sakamoto, Takayuki Kanda, Tetsuo Ono, Hiroshi Ishiguro, and Norihiro Hagita. 2007. Android As a Telecommunication Medium with a Human-like Presence. In Proceedings of the ACM/IEEE International Conference on Human-robot Interaction (HRI '07). ACM, NY, NY, USA, 193--200.

Digital Library

[28]

Mose Sakashita, Keisuke Kawahara, Amy Koike, Kenta Suzuki, Ippei Suzuki, and Yoichi Ochiai. 2016. Yadori: Mask-type User Interface for Manipulation of Puppets. In ACM SIGGRAPH 2016 Emerging Technologies (SIGGRAPH '16). ACM, NY, NY, USA, Article 23, 1 pages.

Digital Library

[29]

C.E. Schaefer and L.J. Carey. 1994. Family Play Therapy. J. Aronson.

[30]

https://books.google.co.jp/books?id=kElcV84iH- 4C

[31]

Dairoku Sekiguchi, Masahiko Inami, and Susumu Tachi. 2001. RobotPHONE: RUI for interpersonal communication. In CHI'01 Extended Abstracts on Human Factors in Computing Systems. ACM, 277--278.

Digital Library

[32]

Rika Shoji, Toshiki Yoshiike, Yuya Kikukawa, Tadahiro Nishikawa, Taigetsu Saori, Suketomo Ayaka, Tetsuaki Baba, and Kumiko Kushiyama. 2012. Mimicat: Face Input Interface Supporting Animatronics Costume Performer's Facial Expression. In ACM SIGGRAPH 2012 Posters (SIGGRAPH '12). ACM, NY, NY, USA, Article 72, 1 pages.

Digital Library

[33]

Ronit Slyper, Guy Hoffman, and Ariel Shamir. 2015. Mirror Puppeteering: Animating Toy Robots in Front of a Webcam. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, NY, NY, USA, 241--248.

Digital Library

[34]

Ronit Slyper, Jill Lehman, Jodi Forlizzi, and Jessica Hodgins. 2011. A Tongue Input Device for Creating Conversations. In Proceedings of the 24th annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, 117--126.

Digital Library

[35]

Yuta Sugiura, Calista Lee, Masayasu Ogata, Anusha Withana, Yasutoshi Makino, Daisuke Sakamoto, Masahiko Inami, and Takeo Igarashi. 2012. PINOKY: A Ring That Animates Your Plush Toys. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, NY, NY, USA, 725--734.

Digital Library

[36]

Thibaut Weise, Hao Li, Luc Van Gool, and Mark Pauly. 2009. Face/Off: Live Facial Puppetry. In Proceedings of

Digital Library

[37]

the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA '09). ACM, NY, NY, USA, 7--16.

[38]

John P Whitney, Tianyao Chen, John Mars, and Jessica K Hodgins. 2016. A hybrid hydrostatic transmission and human-safe haptic telepresence robot. In Robotics and Automation (ICRA), 2016 IEEE International Conference on. IEEE, 690--695.

Digital Library

[39]

Tomoko Yonezawa and Kenji Mase. 2002. Musically Expressive Doll in Face-to-Face Communication. In Proceedings of the 4th IEEE International Conference on Multimodal Interfaces (ICMI '02). IEEE Computer Society, Washington, DC, USA, 417--.

Digital Library

Cited By

Wang YSeaborn K(2024)Kawaii Computing: Scoping Out the Japanese Notion of Cute in User Experiences with Interactive SystemsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651001(1-9)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651001
Zhou QLedo DFitzmaurice GAnderson F(2024)TimeTunnel: Integrating Spatial and Temporal Motion Editing for Character Animation in Virtual RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641927(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641927
Van Haastregt JWelle MZhang YKragic D(2024)Puppeteer Your Robot: Augmented Reality Leader-Follower Teleoperation2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids)10.1109/Humanoids58906.2024.10769593(1019-1026)Online publication date: 22-Nov-2024
https://doi.org/10.1109/Humanoids58906.2024.10769593
Show More Cited By

Index Terms

You as a Puppet: Evaluation of Telepresence User Interface for Puppetry
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Transformed Human Presence for Puppetry
ACE '16: Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology

We propose a system for transmitting a human performer's body and facial movements to a puppet with audiovisual feedback to the performer. The system consists of a head-mounted display (HMD) that shows the performer the video recording of the puppet's ...
Puppet playing: an interactive character animation system with hand motion control
Transactions on computational science XII

Puppetry is a popular art form involving the process of animating the inanimate performing puppets. Puppet playing not only is controlled by artist's hand motions but also follows physical laws and engineering principles. To implement a puppet playing ...
Networked Virtual Marionette Theater
KES '08: Proceedings of the 12th international conference on Knowledge-Based Intelligent Information and Engineering Systems, Part I

This paper describes a system that allows users to control virtual marionette characters based on computer graphics (CG marionette characters) with their hand and finger movements and thus perform a marionette theatrical play. The system consists of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

October 2017

870 pages

ISBN:9781450349819

DOI:10.1145/3126594

General Chair:
Krzysztof Gajos
Harvard University
,
Program Chairs:
Jennifer Mankoff
Carnegie Mellon University
,
Chris Harrison
Carnegie Mellon University

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

FUJITSU SOCIAL SCIENCE LABORATORY LIMITED

Conference

UIST '17

Sponsor:

UIST '17: The 30th Annual ACM Symposium on User Interface Software and Technology

October 22 - 25, 2017

QC, Québec City, Canada

Acceptance Rates

UIST '17 Paper Acceptance Rate 73 of 324 submissions, 23%;

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
858
Total Downloads

Downloads (Last 12 months)106
Downloads (Last 6 weeks)7

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang YSeaborn K(2024)Kawaii Computing: Scoping Out the Japanese Notion of Cute in User Experiences with Interactive SystemsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651001(1-9)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651001
Zhou QLedo DFitzmaurice GAnderson F(2024)TimeTunnel: Integrating Spatial and Temporal Motion Editing for Character Animation in Virtual RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641927(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641927
Van Haastregt JWelle MZhang YKragic D(2024)Puppeteer Your Robot: Augmented Reality Leader-Follower Teleoperation2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids)10.1109/Humanoids58906.2024.10769593(1019-1026)Online publication date: 22-Nov-2024
https://doi.org/10.1109/Humanoids58906.2024.10769593
Sakashita MKim HWoodard BZhang RGuimbretière F(2023)VRoxy: Wide-Area Collaboration From an Office Using a VR-Driven Robotic ProxyProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606743(1-13)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606743
Ihara KFaridan MIchikawa AKawaguchi ISuzuki R(2023)HoloBots: Augmenting Holographic Telepresence with Mobile Robots for Tangible Remote Collaboration in Mixed RealityProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606727(1-12)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606727
Faridan MKumari BSuzuki R(2023)ChameleonControl: Teleoperating Real Human Surrogates through Mixed Reality Gestural Guidance for Remote Hands-on ClassroomsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581381(1-13)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581381
Nakamura FMurakami MSuzuki KFukuoka MMasai KSugimoto M(2023)Analyzing the Effect of Diverse Gaze and Head Direction on Facial Expression Recognition With Photo-Reflective Sensors Embedded in a Head-Mounted DisplayIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.317976629:10(4124-4139)Online publication date: 1-Oct-2023
https://doi.org/10.1109/TVCG.2022.3179766
Ishigaki SIsmail A(2023)Real-Time 3D Reconstruction for Mixed Reality Telepresence Using Multiple Depth SensorsAdvanced Communication and Intelligent Systems10.1007/978-3-031-25088-0_5(67-80)Online publication date: 15-Feb-2023
https://doi.org/10.1007/978-3-031-25088-0_5
Hung CChang RChen HLiang CChan LChen B(2022)Puppeteer: Exploring Intuitive Hand Gestures and Upper-Body Postures for Manipulating Human Avatar ActionsProceedings of the 28th ACM Symposium on Virtual Reality Software and Technology10.1145/3562939.3565609(1-11)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3562939.3565609
Echeverri DMaiden NSas C(2022)The Non-myth of the Noble Red: Exploring Brechtian Principles of Storytelling and Performance in the Authoring of a Tangible NarrativeProceedings of the 14th Conference on Creativity and Cognition10.1145/3527927.3535207(520-525)Online publication date: 20-Jun-2022
https://dl.acm.org/doi/10.1145/3527927.3535207
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents