[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3491102.3517689acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

ProGesAR: Mobile AR Prototyping for Proxemic and Gestural Interactions with Real-world IoT Enhanced Spaces

Published: 28 April 2022 Publication History

Abstract

Real-world IoT enhanced spaces involve diverse proximity- and gesture-based interactions between users and IoT devices/objects. Prototyping such interactions benefits various applications like the conceptual design of ubicomp space. AR (Augmented Reality) prototyping provides a flexible way to achieve early-stage designs by overlaying digital contents on real objects or environments. However, existing AR prototyping approaches have focused on prototyping AR experiences or context-aware interactions from the first-person view instead of full-body proxemic and gestural (pro-ges for short) interactions of real users in the real world. In this work, we conducted interviews to figure out the challenges of prototyping pro-ges interactions in real-world IoT enhanced spaces. Based on the findings, we present ProGesAR, a mobile AR tool for prototyping pro-ges interactions of a subject in a real environment from a third-person view, and examining the prototyped interactions from both the first- and third- person views. Our interface supports the effects of virtual assets dynamically triggered by a single subject, with the triggering events based on four features: location, orientation, gesture, and distance. We conduct a preliminary study by inviting participants to prototype in a freeform manner using ProGesAR. The early-stage findings show that with ProGesAR, users can easily and quickly prototype their design ideas about pro-ges interactions.

Supplementary Material

Supplemental Materials (3491102.3517689-supplemental-materials.zip)
MP4 File (3491102.3517689-video-figure.mp4)
Video Figure
MP4 File (3491102.3517689-video-preview.mp4)
Video Preview
MP4 File (3491102.3517689-talk-video.mp4)
Talk Video

References

[1]
Apple. 2019. Reality Composer. https://apps.apple.com/us/app/reality-composer/id1462358802
[2]
Narges Ashtari, Andrea Bunt, Joanna McGrenere, Michael Nebeling, and Parmit K Chilana. 2020. Creating augmented and virtual reality applications: Current practices, challenges, and opportunities. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.
[3]
Till Ballendat, Nicolai Marquardt, and Saul Greenberg. 2010. Proxemic interaction: designing for a proximity and orientation-aware environment. In ACM International Conference on Interactive Tabletops and Surfaces. 121–130.
[4]
Andrea Bellucci, Aneesh P Tarun, Ahmed Sabbir Arif, and Ali Mazalek. 2016. Developing Responsive and Interactive Environments with the ROSS Toolkit. In Proceedings of the TEI’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction. 782–785.
[5]
Yuanzhi Cao, Tianyi Wang, Xun Qian, Pawan S Rao, Manav Wadhawan, Ke Huo, and Karthik Ramani. 2019. GhostAR: A time-space editor for embodied authoring of human-robot collaborative task with augmented reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 521–534.
[6]
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2018. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. arXiv preprint arXiv:1812.08008(2018).
[7]
Kathy Charmaz. 2008. Constructionism and the grounded theory method. Handbook of constructionist research 1, 1 (2008), 397–412.
[8]
Kathy Charmaz. 2014. Constructing grounded theory. sage.
[9]
Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz studies—why and how. Knowledge-based systems 6, 4 (1993), 258–266.
[10]
Mark Fiala. 2005. ARTag, a fiducial marker system using digital techniques. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 2. IEEE, 590–596.
[11]
James Fogarty, Jodi Forlizzi, and Scott E Hudson. 2002. Specifying behavior and semantic meaning in an unmodified layered drawing package. In Proceedings of the 15th annual ACM symposium on User interface software and technology. 61–70.
[12]
Maxime Garcia, Rémi Ronfard, and Marie-Paule Cani. 2019. Spatial Motion Doodles: Sketching Animation in VR Using Hand Gestures and Laban Motion Analysis. In Motion, Interaction and Games. 1–10.
[13]
Danilo Gasques, Janet G Johnson, Tommy Sharkey, and Nadir Weibel. 2019. What you sketch is what you get: Quick and easy augmented reality prototyping with pintar. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1–6.
[14]
Saul Greenberg, Nicolai Marquardt, Till Ballendat, Rob Diaz-Marino, and Miaosen Wang. 2011. Proxemic interactions: the new ubicomp?interactions 18, 1 (2011), 42–50.
[15]
Sinem Güven and Steven Feiner. 2003. Authoring 3D hypermedia for wearable augmented and virtual reality. In Proceedings of IEEE International Symposium on Wearable Computers (ISWC’03). 21–23.
[16]
Edward Twitchell Hall. 1966. The hidden dimension. Vol. 609. Garden City, NY: Doubleday.
[17]
Björn Hartmann, Scott R Klemmer, Michael Bernstein, Leith Abdulla, Brandon Burr, Avi Robinson-Mosher, and Jennifer Gee. 2006. Reflective physical prototyping through integrated design, test, and analysis. In Proceedings of the 19th annual ACM symposium on User interface software and technology. 299–308.
[18]
Rex Hartson and Pardha Pyla. 2012. The UX Book: Process and guidelines for ensuring a quality user experience. Elsevier.
[19]
Ke Huo, Yuanzhi Cao, Sang Ho Yoon, Zhuangying Xu, Guiming Chen, and Karthik Ramani. 2018. Scenariot: spatially mapping smart things within augmented reality scenes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.
[20]
Adobe Inc.2019. Adobe Aero. https://apps.apple.com/us/app/adobe-aero/id1401748913
[21]
Sujin Jang, Niklas Elmqvist, and Karthik Ramani. 2014. GestureAnalyzer: visual analytics for pattern analysis of mid-air hand gestures. In Proceedings of the 2nd ACM symposium on Spatial user interaction. 30–39.
[22]
Runchang Kang, Anhong Guo, Gierad Laput, Yang Li, and Xiang’Anthony’ Chen. 2019. Minuet: Multimodal interaction with an Internet of Things. In Symposium on Spatial User Interaction. 1–10.
[23]
Hirokazu Kato and Mark Billinghurst. 1999. Marker tracking and hmd calibration for a video-based augmented reality conferencing system. In Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR’99). IEEE, 85–94.
[24]
Annie Kelly, R Benjamin Shapiro, Jonathan de Halleux, and Thomas Ball. 2018. ARcadia: A rapid prototyping platform for real-time tangible interfaces. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–8.
[25]
Daehwan Kim and Daijin Kim. 2006. An intelligent smart home control using body gestures. In 2006 International Conference on Hybrid Information Technology, Vol. 2. IEEE, 439–446.
[26]
Han-Jong Kim, Chang Min Kim, and Tek-Jin Nam. 2018. Sketchstudio: Experience prototyping with 2.5-dimensional animated design scenarios. In Proceedings of the 2018 Designing Interactive Systems Conference. 831–843.
[27]
Han-Jong Kim, Ju-Whan Kim, and Tek-Jin Nam. 2016. miniStudio: Designers’ Tool for Prototyping Ubicomp Space with Interactive Miniature. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 213–224.
[28]
Scott R Klemmer, Jack Li, James Lin, and James A Landay. 2004. Papier-Mache: toolkit support for tangible input. In Proceedings of the SIGCHI conference on Human factors in computing systems. 399–406.
[29]
Barry Kollee, Sven Kratz, and Anthony Dunnigan. 2014. Exploring gestural interaction in smart spaces using head mounted devices with ego-centric sensing. In Proceedings of the 2nd ACM symposium on Spatial user interaction. 40–49.
[30]
Veronika Krauß, Alexander Boden, Leif Oppermann, and René Reiners. 2021. Current practices, challenges, and design implications for collaborative AR/VR application development. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.
[31]
Kin Chung Kwan and Hongbo Fu. 2019. Mobi3DSketch: 3D Sketching in Mobile AR. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM.
[32]
David Ledo, Saul Greenberg, Nicolai Marquardt, and Sebastian Boring. 2015. Proxemic-aware controls: Designing remote controls for ubiquitous computing ecologies. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. 187–198.
[33]
David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquardt, Lora Oehlberg, and Saul Greenberg. 2018. Evaluation strategies for HCI toolkit research. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–17.
[34]
David Ledo, Jo Vermeulen, Sheelagh Carpendale, Saul Greenberg, Lora Oehlberg, and Sebastian Boring. 2019. Astral: Prototyping Mobile and Smart Object Interactive Behaviours Using Familiar Applications. In Proceedings of the 2019 on Designing Interactive Systems Conference. 711–724.
[35]
Sang-Su Lee, Jeonghun Chae, Hyunjeong Kim, Youn-kyung Lim, and Kun-pyo Lee. 2013. Towards more natural digital content manipulation via user freehand gestural interaction in a living room. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. 617–626.
[36]
Wei-Po Lee, Che Kaoli, and Jhih-Yuan Huang. 2014. A smart TV system with body-gesture control, tag-based rating and context-aware recommendation. Knowledge-Based Systems 56 (2014), 167–178.
[37]
Germán Leiva and Michel Beaudouin-Lafon. 2018. Montage: A video prototyping system to reduce re-shooting and increase re-usability. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 675–682.
[38]
Germán Leiva, Jens Emil Grønbæk, Clemens Nylandsted Klokmose, Cuong Nguyen, Rubaiat Habib Kazi, and Paul Asente. 2021. Rapido: Prototyping Interactive AR Experiences through Programming by Demonstration. In The 34th Annual ACM Symposium on User Interface Software and Technology. 626–637.
[39]
Germán Leiva, Cuong Nguyen, Rubaiat Habib Kazi, and Paul Asente. 2020. Pronto: Rapid Augmented Reality Video Prototyping Using Sketches and Enaction. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.
[40]
Yang Li, Jason I Hong, and James A Landay. 2004. Topiary: a tool for prototyping location-enhanced applications. In Proceedings of the 17th annual ACM symposium on User interface software and technology. 217–226.
[41]
Hao Lü and Yang Li. 2012. Gesture coder: a tool for programming multi-touch gestures by demonstration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2875–2884.
[42]
Hao Lü and Yang Li. 2013. Gesture studio: authoring multi-touch interactions through demonstration and declaration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 257–266.
[43]
Blair MacIntyre, Maribeth Gandy, Steven Dow, and Jay David Bolter. 2004. DART: a toolkit for rapid design exploration of augmented reality experiences. In Proceedings of the 17th annual ACM symposium on User interface software and technology. 197–206.
[44]
Nicolai Marquardt, Robert Diaz-Marino, Sebastian Boring, and Saul Greenberg. 2011. The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 315–326.
[45]
Nicolai Marquardt and Saul Greenberg. 2012. Informing the design of proxemic interactions. IEEE Pervasive Computing 11, 2 (2012), 14–23.
[46]
Nolwenn Maudet, Germán Leiva, Michel Beaudouin-Lafon, and Wendy Mackay. 2017. Design Breakdowns: Designer-Developer Gaps in Representing and Interpreting Interactive Systems. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 630–641.
[47]
Brad Myers, Sun Young Park, Yoko Nakano, Greg Mueller, and Andrew Ko. 2008. How designers design and program interactive behaviors. In 2008 IEEE Symposium on Visual Languages and Human-Centric Computing. IEEE, 177–184.
[48]
Yasuto Nakanishi. 2012. Virtual prototyping using miniature model and visualization for interactive public displays. In Proceedings of the Designing Interactive Systems Conference. 458–467.
[49]
Tek-Jin Nam. 2005. Sketch-based rapid prototyping platform for hardware-software integrated interactive products. In CHI’05 extended abstracts on Human factors in computing systems. 1689–1692.
[50]
Michael Nebeling and Katy Madier. 2019. 360proto: Making Interactive Virtual Reality & Augmented Reality Prototypes from Paper. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.
[51]
Michael Nebeling, Janet Nebeling, Ao Yu, and Rob Rumble. 2018. Protoar: Rapid physical-digital prototyping of mobile augmented reality applications. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.
[52]
Dan R Olsen Jr. 2007. Evaluating user interface systems research. In Proceedings of the 20th annual ACM symposium on User interface software and technology. 251–258.
[53]
Hyungjun Park, Hee-Cheol Moon, and Jae Yeol Lee. 2009. Tangible augmented prototyping of digital handheld products. Computers in Industry 60, 2 (2009), 114–125.
[54]
Sarah Prange and Florian Alt. 2020. I Wish You Were Smart (er): Investigating Users’ Desires and Needs Towards Home Appliances. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–8.
[55]
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 779–788.
[56]
Gang Ren, Wenbin Li, and Eamonn O’Neill. 2016. Towards the design of effective freehand gestural interaction for interactive tv. Journal of Intelligent & Fuzzy Systems 31, 5 (2016), 2659–2674.
[57]
Nazmus Saquib, Rubaiat Habib Kazi, Li-Yi Wei, and Wilmot Li. 2019. Interactive body-driven graphics for augmented video performance. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
[58]
Hartmut Seichter, Julian Looser, and Mark Billinghurst. 2008. ComposAR: An intuitive tool for authoring AR applications. In 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE, 177–178.
[59]
Teddy Seyed, Alaa Azazi, Edwin Chan, Yuxi Wang, and Frank Maurer. 2015. Sod-toolkit: A toolkit for interactively prototyping and developing multi-sensor, multi-device environments. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces. 171–180.
[60]
Kihoon Son, Hwiwon Chun, Sojin Park, and Kyung Hoon Hyun. 2020. C-Space: An Interactive Prototyping Platform for Collaborative Spatial Design Exploration. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.
[61]
Maximilian Speicher and Michael Nebeling. 2018. GestureWiz: A human-powered gesture design environment for user interface prototypes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–11.
[62]
Ryo Suzuki, Rubaiat Habib Kazi, Li-Yi Wei, Stephen DiVerdi, Wilmot Li, and Daniel Leithinger. 2020. RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 166–181.
[63]
John Underkoffler and Hiroshi Ishii. 1999. Urp: a luminous-tangible workbench for urban planning and design. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 386–393.
[64]
Tianyi Wang, Xun Qian, Fengming He, Xiyun Hu, Yuanzhi Cao, and Karthik Ramani. 2021. GesturAR: An Authoring System for Creating Freehand Interactive Augmented Reality Applications. In The 34th Annual ACM Symposium on User Interface Software and Technology. 552–567.
[65]
Tianyi Wang, Xun Qian, Fengming He, Xiyun Hu, Ke Huo, Yuanzhi Cao, and Karthik Ramani. 2020. CAPturAR: An Augmented Reality Tool for Authoring Human-Involved Context-Aware Applications. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 328–341.
[66]
Zeyu Wang, Cuong Nguyen, Paul Asente, and Julie Dorsey. 2021. DistanciAR: Authoring Site-Specific Augmented Reality Experiences for Remote Environments. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–12.

Cited By

View all
  • (2024)Using Artmaking Generative AIs to Support Augmented Reality Learning Designs With Adobe Aero AppInquiries of Pedagogical Shifts and Critical Mindsets Among Educators10.4018/979-8-3693-1078-6.ch006(132-152)Online publication date: 8-Mar-2024
  • (2024)PilotAR: Streamlining Pilot Studies with OHMDs from Concept to InsightProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785768:3(1-35)Online publication date: 9-Sep-2024
  • (2024)Demonstrating PilotAR: A Tool to Assist Wizard-of-Oz Pilot Studies with OHMDCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677554(230-234)Online publication date: 5-Oct-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
April 2022
10459 pages
ISBN:9781450391573
DOI:10.1145/3491102
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 April 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. AR prototyping
  2. Gestural interaction
  3. Mobile augmented reality
  4. Proxemic interaction

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

CHI '22
Sponsor:
CHI '22: CHI Conference on Human Factors in Computing Systems
April 29 - May 5, 2022
LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)256
  • Downloads (Last 6 weeks)20
Reflects downloads up to 14 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Using Artmaking Generative AIs to Support Augmented Reality Learning Designs With Adobe Aero AppInquiries of Pedagogical Shifts and Critical Mindsets Among Educators10.4018/979-8-3693-1078-6.ch006(132-152)Online publication date: 8-Mar-2024
  • (2024)PilotAR: Streamlining Pilot Studies with OHMDs from Concept to InsightProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785768:3(1-35)Online publication date: 9-Sep-2024
  • (2024)Demonstrating PilotAR: A Tool to Assist Wizard-of-Oz Pilot Studies with OHMDCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677554(230-234)Online publication date: 5-Oct-2024
  • (2024)Jigsaw: Authoring Immersive Storytelling Experiences with Augmented Reality and Internet of ThingsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642744(1-14)Online publication date: 11-May-2024
  • (2024)ProInterAR: A Visual Programming Platform for Creating Immersive AR InteractionsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642527(1-15)Online publication date: 11-May-2024
  • (2024)RoboVisAR: Immersive Authoring of Condition-based AR Robot VisualisationsProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634972(462-471)Online publication date: 11-Mar-2024
  • (2023)Reality and Beyond: Proxemics as a Lens for Designing Handheld Collaborative Augmented RealityProceedings of the ACM on Human-Computer Interaction10.1145/36264637:ISS(21-40)Online publication date: 1-Nov-2023
  • (2022)ProjectAR: Rapid Prototyping of Projection Mapping with Mobile Augmented RealityAdjunct Proceedings of the 2022 Nordic Human-Computer Interaction Conference10.1145/3547522.3547679(1-5)Online publication date: 8-Oct-2022
  • (2022)CleAR Sight: Exploring the Potential of Interacting with Transparent Tablets in Augmented Reality2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR55827.2022.00034(196-205)Online publication date: Oct-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media