Wizard of Props: Mixed Reality Prototyping with Physical Props to Design Responsive Environments
Authors: 
Yuzhen Zhang, Ruixiang Han, 
Ran Zhou, Peter Gyory, Clement Zheng, 
Patrick C. Shih, 
Ellen Yi-Luen Do, Malte F Jung, 
Wendy Ju, 
Daniel LeithingerAuthors Info & Claims
Article No.: 47, Pages 1 - 15
Abstract
Driven by the vision of future responsive environments, where everyday surroundings can perceive human behaviors and respond through intelligent robotic actuation, we propose Wizard of Props (WoP): a human-centered design workflow for creating expressive, implicit, and meaningful interactions. This collaborative experience prototyping approach integrates full-scale physical props with Mixed Reality (MR) to support ideation, prototyping, and rapid testing of responsive environments. We present two design explorations that showcase our investigations of diverse design solutions based on varying technology resources, contextual considerations, and target audiences. Design Exploration One focuses on mixed environment building, where we observe fluid prototyping methods. In Design Exploration Two, we explore how novice designers approach WoP, and illustrate their design ideas and behaviors. Our findings reveal that WoP complements conventional design methods, enabling intuitive body-storming, supporting flexible prototyping fidelity, and fostering expressive environment-human interactions through in-situ improvisational performance.
Supplemental Material
MP4 File
Presentation Video
- Download
- 223.50 MB
MP4 File
Demo of WoP Design Exploration One
- Download
- 389.57 MB
MP4 File
Demo of WoP Design Exploration Two
- Download
- 285.45 MB
References
[1]
Binh Vinh Duc Nguyen, Jihae Han, and Andrew Vande Moere. 2022. Towards Responsive Architecture that Mediates Place: Recommendations on How and When an Autonomously Moving Robotic Wall Should Adapt a Spatial Layout. Proc. ACM Hum.-Comput. Interact. 6, CSCW2, Article 467 (November 2022), 27 pages. https://doi.org/10.1145/3555568
[2]
“Ori Expandable Apartments.” Ori Expandable Apartments. Accessed July 22, 2023. https://www.oriliving.com/.
[3]
Elena Sabinson, Isha Pradhan, and Keith Evan Green. 2021. Plant-Human Embodied Biofeedback (pheB): A Soft Robotic Surface for Emotion Regulation in Confined Physical Space. In Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI ‘21). Association for Computing Machinery, New York, NY, USA, Article 89, 1-14. https://doi.org/10.1145/3430524.3446065
[4]
Miriam Sturdee, John Hardy, Nick Dunn, and Jason Alexander. 2015. A Public Ideation of Shape-Changing Applications. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces (ITS ‘15). Association for Computing Machinery, New York, NY, USA, 219–228. https://doi.org/10.1145/2817721.2817734
[5]
Myron W. Krueger. 1977. Responsive environments. In Proceedings of the June 13-16, 1977, national computer conference (AFIPS ‘77). Association for Computing Machinery, New York, NY, USA, 423–433. https://doi.org/10.1145/1499402.1499476
[6]
Majken K. Rasmussen, Giovanni M. Troiano, Marianne G. Petersen, Jakob G. Simonsen, and Kasper Hornbæk. 2016. Sketching Shape-changing Interfaces: Exploring Vocabulary, Metaphors Use, and Affordances. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ‘16). Association for Computing Machinery, New York, NY, USA, 2740–2751. https://doi.org/10.1145/2858036.2858183
[7]
S. Dow, B. MacIntyre, J. Lee, C. Oezbek, J. D. Bolter and M. Gandy, “Wizard of Oz support throughout an iterative design process,” in IEEE Pervasive Computing, vol. 4, no. 4, pp. 18-26, Oct.-Dec. 2005.
[8]
Mayra Donaji Barrera Machuca, Wolfgang Stuerzlinger, and Paul Asente. 2019. The Effect of Spatial Ability on Immersive 3D Drawing. In Proceedings of the 2019 Conference on Creativity and Cognition (C&C ‘19). Association for Computing Machinery, New York, NY, USA, 173–186. https://doi.org/10.1145/3325480.3325489
[9]
Marion Buchenau and Jane Fulton Suri. 2000. Experience prototyping. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques (DIS ‘00). Association for Computing Machinery, New York, NY, USA, 424–433. https://doi.org/10.1145/347642.347802
[10]
Sang-Gyun An, Yongkwan Kim, Joon Hyub Lee, and Seok-Hyung Bae. 2017. Collaborative Experience Prototyping of Automotive Interior in VR with 3D Sketching and Haptic Helpers. In Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI ‘17). Association for Computing Machinery, New York, NY, USA, 183–192. https://doi.org/10.1145/3122986.3123002
[11]
Jose Fermoso. 2008. Kinetic Structure at BMW Museum Interprets Car Design Process. Wired. com. July (2008).
[12]
Margaret Rhodes. 2014. An Interactive Ceiling That Undulates as You Pass Under. (Jan 2014). https://www.wired.com/2014/09/an-interactive-ceiling-that-undulates-as-you-pass-under/
[13]
Wen Yen Tang, Sheng Kai Tang, and Yuzn Zone Lee. 2011. Tangible Pixels. (2011).
[14]
Ryo Suzuki, Ryosuke Nakayama, Dan Liu, Yasuaki Kakehi, Mark D. Gross, and Daniel Leithinger. 2020. LiftTiles: Constructive Building Blocks for Prototyping Room-scale Shape-changing Interfaces. In Proceedings of the Fourteenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI ‘20). Association for Computing Machinery, New York, NY, USA, 143–151. https://doi.org/10.1145/3374920.3374941
[15]
Yuki Onishi, Kazuki Takashima, Shoi Higashiyama, Kazuyuki Fujita, and Yoshifumi Kitamura. 2022. WaddleWalls: Room-scale Interactive Partitioning System using a Swarm of Robotic Partitions. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST ‘22). Association for Computing Machinery, New York, NY, USA, Article 29, 1–15. https://doi.org/10.1145/3526113.3545615
[16]
Karmen Franinović, Luke Franzke, Florian Wille, and Andrés Villa Torres. 2019. Interacting with Electroactive Polymers in Responsive Environments. In Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI ‘19). Association for Computing Machinery, New York, NY, USA, 505–512. https://doi.org/10.1145/3294109.3301268
[17]
Nick Taylor, Jon Rogers, Loraine Clarke, Martin Skelly, Jayne Wallace, Pete Thomas, Babitha George, Romit Raj, Mike Shorter, and Michelle Thorne. 2021. Prototyping Things: Reflecting on Unreported Objects of Design Research for IoT. In Proceedings of the 2021 ACM Designing Interactive Systems Conference (DIS ‘21). Association for Computing Machinery, New York, NY, USA, 1807–1816. https://doi.org/10.1145/3461778.3462037
[18]
Danielle Wilde, Anna Vallgårda, and Oscar Tomico. 2017. Embodied Design Ideation Methods: Analysing the Power of Estrangement. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ‘17). Association for Computing Machinery, New York, NY, USA, 5158–5170. https://doi.org/10.1145/3025453.3025873
[19]
Sirkin, David, and Wendy Ju. “Using embodied design improvisation as a design research tool.” Proceedings of the international conference on Human Behavior in Design (HBiD 2014), Ascona, Switzerland. 2014.
[20]
Wikimedia Foundation. (2023, July 6). Mechanical turk. Wikipedia. https://en.wikipedia.org/wiki/Mechanical_Turk
[21]
John D. Gould, John Conti, and Todd Hovanyecz. 1983. Composing letters with a simulated listening typewriter. Commun. ACM 26, 4 (April 1983), 295–308. https://doi.org/10.1145/2163.358100
[22]
S. Dow, B. MacIntyre, J. Lee, C. Oezbek, J. D. Bolter and M. Gandy, “Wizard of Oz support throughout an iterative design process,” in IEEE Pervasive Computing, vol. 4, no. 4, pp. 18-26, Oct.-Dec. 2005.
[23]
David Maulsby, Saul Greenberg, and Richard Mander. 1993. Prototyping an intelligent agent through Wizard of Oz. In Proceedings of the INTERACT ‘93 and CHI ‘93 Conference on Human Factors in Computing Systems (CHI ‘93). Association for Computing Machinery, New York, NY, USA, 277–284. https://doi.org/10.1145/169059.169215
[24]
Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz studies: why and how. In Proceedings of the 1st international conference on Intelligent user interfaces (IUI ‘93). Association for Computing Machinery, New York, NY, USA, 193–200. https://doi.org/10.1145/169891.169968
[25]
Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz studies: why and how. In Proceedings of the 1st international conference on Intelligent user interfaces (IUI ‘93). Association for Computing Machinery, New York, NY, USA, 193–200. https://doi.org/10.1145/169891.169968
[26]
Laurel D. Riek. 2012. Wizard of Oz studies in HRI: a systematic review and new reporting guidelines. J. Hum.-Robot Interact. 1, 1 (July 2012), 119–136. https://doi.org/10.5898/JHRI.1.1.Riek
[27]
J.D. Zamfirescu-Pereira, David Sirkin, David Goedicke, Ray LC, Natalie Friedman, Ilan Mandel, Nikolas Martelaro, and Wendy Ju. 2021. Fake It to Make It: Exploratory Prototyping in HRI. In Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction (HRI ‘21 Companion). Association for Computing Machinery, New York, NY, USA, 19–28. https://doi.org/10.1145/3434074.3446909
[28]
Weilun Gong, Stephanie Santosa, Tovi Grossman, Michael Glueck, Daniel Clarke, and Frances Lai. 2023. Affordance-Based and User-Defined Gestures for Spatial Tangible Interaction. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (DIS ‘23). Association for Computing Machinery, New York, NY, USA, 1500–1514. https://doi.org/10.1145/3563657.3596032
[29]
David Sirkin, Brian Mok, Stephen Yang, and Wendy Ju. 2015. Mechanical Ottoman: How Robotic Furniture Offers and Withdraws Support. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI ‘15). Association for Computing Machinery, New York, NY, USA, 11–18. https://doi.org/10.1145/2696454.2696461
[30]
Y. Wang and K. Evan Green, “Designing Socially Interactive, Robotic Environments through Pattern Languages,” 2022 18th International Conference on Intelligent Environments (IE), Biarritz, France, 2022, pp. 1-8.
[31]
Marco Spadafora, Victor Chahuneau, Nikolas Martelaro, David Sirkin, and Wendy Ju. 2016. Designing the Behavior of Interactive Objects. In Proceedings of the TEI ‘16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI ‘16). Association for Computing Machinery, New York, NY, USA, 70–77. https://doi.org/10.1145/2839462.2839502
[32]
Houde, Stephanie, and Charles Hill. “What do prototypes prototype?.” Handbook of human-computer interaction. North-Holland, 1997. 367-381.
[33]
Binh Vinh Duc Nguyen, Adalberto L. Simeone, and Andrew Vande Moere. 2021. Exploring an Architectural Framework for Human-Building Interaction via a Semi-Immersive Cross-Reality Methodology. In Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction (HRI ‘21). Association for Computing Machinery, New York, NY, USA, 252–261. https://doi.org/10.1145/3434073.3444643
[34]
Pacchierotti, Claudio, “Wearable haptic systems for the fingertip and the hand: taxonomy, review, and perspectives.” IEEE transactions on haptics 10.4 (2017): 580-600.
[35]
Ryo Suzuki, Hooman Hedayati, Clement Zheng, James L. Bohn, Daniel Szafir, Ellen Yi-Luen Do, Mark D. Gross, and Daniel Leithinger. 2020. RoomShift: Room-scale Dynamic Haptics for VR with Furniture-moving Swarm Robots. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ‘20). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3313831.3376523
[36]
S. Mortezapoor, K. Vasylevska, E. Vonach and H. Kaufmann, “CoboDeck: A Large-Scale Haptic VR System Using a Collaborative Mobile Robot,” 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR), Shanghai, China, 2023, pp. 297-307.
[37]
Adalberto L. Simeone, Eduardo Velloso, and Hans Gellersen. 2015. Substitutional Reality: Using the Physical Environment to Design Virtual Reality Experiences. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ‘15). Association for Computing Machinery, New York, NY, USA, 3307–3316. https://doi.org/10.1145/2702123.2702389
[38]
Cathy Mengying Fang, Ryo Suzuki, and Daniel Leithinger. 2023. VR Haptics at Home: Repurposing Everyday Objects and Environment for Casual and On-Demand VR Haptic Experiences. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA ‘23). Association for Computing Machinery, New York, NY, USA, Article 312, 1–7. https://doi.org/10.1145/3544549.3585871
[39]
Lung-PanCheng, Patrick Luhne, Pedro Lopes,Christoph Sterz, and Patrick Baudisch. 2014. Haptic turk: a motion platform based on people. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘14). Association for Computing Machinery, New York, NY, USA, 3463–3472. https://doi.org/10.1145/2556288.2557101
[40]
Lung-Pan Cheng, Thijs Roumen, Hannes Rantzsch, Sven Köhler, Patrick Schmidt, Robert Kovacs, Johannes Jasper, Jonas Kemper, and Patrick Baudisch. 2015. TurkDeck: Physical Virtual Reality Based on People. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST ‘15). Association for Computing Machinery, New York, NY, USA, 417–426. https://doi.org/10.1145/2807442.2807463
[41]
Konstantina Kilteni, Raphaela Groten, Mel Slater; The Sense of Embodiment in Virtual Reality. Presence: Teleoperators and Virtual Environments 2012; 21 (4): 373–387.
[42]
Dennis Schleicher, Peter Jones, and Oksana Kachur. 2010. Bodystorming as embodied designing. interactions 17, 6 (November + December 2010), 47–51. https://doi.org/10.1145/1865245.1865256
[43]
Oulasvirta, Antti, Esko Kurvinen, and Tomi Kankainen. “Understanding contexts by being there: case studies in bodystorming.” Personal and ubiquitous computing 7 (2003): 125-134.
[44]
David Porfirio, Evan Fisher, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2019. Bodystorming Human-Robot Interactions. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST ‘19). Association for Computing Machinery, New York, NY, USA, 479–491. https://doi.org/10.1145/3332165.3347957
[45]
Sarah Fdili Alaoui. 2019. Making an Interactive Dance Piece: Tensions in Integrating Technology in Art. In Proceedings of the 2019 on Designing Interactive Systems Conference (DIS ‘19). Association for Computing Machinery, New York, NY, USA, 1195–1208. https://doi.org/10.1145/3322276.3322289
[46]
Joris Weijdom. 2022. Performative prototyping in collaborative mixed reality environments: an embodied design method for ideation and development in virtual reality. In Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction (TEI ‘22). Association for Computing Machinery, New York, NY, USA, Article 10, 1–13. https://doi.org/10.1145/3490149.3501316
[47]
José Manuel Vega-Cebrián, Elena Márquez Segura, Laia Turmo Vidal, Omar Valdiviezo-Hernández, Annika Waern, Robby Van Delden, Joris Weijdom, Lars Elbæk, Rasmus Vestergaard Andersen, Søren Stigkær Lekbo, and Ana Tajadura-Jiménez. 2023. Design Resources in Movement-based Design Methods: a Practice-based Characterization. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (DIS ‘23). Association for Computing Machinery, New York, NY, USA, 871–888. https://doi.org/10.1145/3563657.3596036
[48]
Barbieri, Loris, “Mixed prototyping with configurable physical archetype for usability evaluation of product interfaces.” Computers in Industry 64.3 (2013): 310-323.
[49]
Michele Fiorentino, Raffaele de Amicis, Giuseppe Monno, and Andre Stork. 2002. Spacedesign: A Mixed Reality Workspace for Aesthetic Industrial Design. In Proceedings of the 1st International Symposium on Mixed and Augmented Reality (ISMAR ‘02). IEEE Computer Society, USA, 86.
[50]
Ju, Wendy, and Larry Leifer. “The design of implicit interactions: Making interactive systems less obnoxious.” Design Issues 24.3 (2008): 72-84.
[51]
M. Sra, V. Danry and P. Maes, “Situated VR: Toward a Congruent Hybrid Reality Without Experiential Artifacts,” in IEEE Computer Graphics and Applications, vol. 42, no. 3, pp. 7-18, 1 May-June 2022.
[52]
Placemaking and the Human Scale City. RSS. (n.d.). https://www.pps.org/article/placemaking-and-the-human-scale-city
[53]
Camburn, B., Viswanathan, V., Linsey, J., Anderson, D., Jensen, D., Crawford, R., … Wood, K. (2017). Design prototyping methods: State of the art in strategies, techniques, and guidelines. Design Science, 3, E13.
[54]
Hoffman, Guy, and Wendy Ju. “Designing robots with movement in mind.” Journal of Human-Robot Interaction 3.1 (2014): 91-122.
Recommendations
The Role of Physical Props in VR Climbing Environments
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsDealing with fear of falling is a challenge in sport climbing. Virtual reality (VR) research suggests that using physical and reality-based interaction increases the presence in VR. In this paper, we present a study that investigates the influence of ...
Enabling Reusable Haptic Props for Virtual Reality by Hand Displacement
MuC '21: Proceedings of Mensch und Computer 2021Virtual Reality (VR) enables compelling visual experiences. However, providing haptic feedback is still challenging. Previous work suggests utilizing haptic props to overcome such limitations and presents evidence that props could function as a single ...
Props and Rocks: Passive Haptic Mixed Reality for Navigating Far-off Worlds
SIGGRAPH '24: ACM SIGGRAPH 2024 Immersive PavilionOur project immerses participants in a world blending virtual and physical realities. Users navigate a divided society, interacting with props via haptic feedback. Employing innovative techniques, including redirected walking, our project seamlessly ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
February 2024
1058 pages
ISBN:9798400704024
DOI:10.1145/3623509
- General Chairs:
- Luigina Ciolfi,
- Trevor Hogan,
- Program Chairs:
- Tanja Döring,
- Tom Jenkins,
- Jelle van Dijk,
- Samuel Huron,
- Zhuying Li,
- Publications Chairs:
- David Coyle,
- Beat Signer
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 February 2024
Check for updates
Qualifiers
- Research-article
- Research
- Refereed limited
Data Availability
Presentation Video https://dl.acm.org/doi/10.1145/3623509.3633395#C0306-002 (compressed and trimmed).mp4
Demo of WoP Design Exploration One https://dl.acm.org/doi/10.1145/3623509.3633395#WoP1.mp4
Demo of WoP Design Exploration Two https://dl.acm.org/doi/10.1145/3623509.3633395#WoP2.mp4
Conference
TEI '24
Sponsor:
TEI '24: Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction
February 11 - 14, 2024
Cork, Ireland
Acceptance Rates
Overall Acceptance Rate 393 of 1,367 submissions, 29%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 755Total Downloads
- Downloads (Last 12 months)755
- Downloads (Last 6 weeks)95
Reflects downloads up to 13 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in