Enhancing VR Sketching with a Dynamic Shape Display
Authors: 
Wen Ying, 
Seongkook HeoAuthors Info & Claims
Article No.: 22, Pages 1 - 11
Abstract
Sketching on virtual objects in Virtual Reality (VR) can be challenging due to the lack of a physical surface that constrains the movement and provides haptic feedback for contact and movement. While using a flat physical drawing surface has been proposed, it creates a significant discrepancy between the physical and virtual surfaces when sketching on non-planar virtual objects. We propose using a dynamic shape display that physically mimics the shape of a virtual surface, allowing users to sketch on a virtual surface as if they are sketching on a physical object’s surface. We demonstrate this using VRScroll, a shape-changing device that features seven independently controlled flaps to imitate the shape of a virtual surface automatically. Our user study showed that participants exhibited higher precision when tracing simple shapes with the dynamic shape display and produced clearer sketches. We also provided several design implications for dynamic shape displays aimed at enabling precise sketching in VR.
Supplemental Material
MP4 File
Accompanying video: VRScroll_VRST24_Final_Encode.mp4 Subtitles for the accompanying video: VRScroll_VRST24_Final_Video_Subtitles.vtt 30-second Teaser video: VRScroll_VRST24_Preview_Encode.mp4 Auxiliary (supporting materials): Materials.zip
- Download
- 41.95 MB
MP4 File
Accompanying video: VRScroll_VRST24_Final_Encode.mp4 Subtitles for the accompanying video: VRScroll_VRST24_Final_Video_Subtitles.vtt 30-second Teaser video: VRScroll_VRST24_Preview_Encode.mp4 Auxiliary (supporting materials): Materials.zip
- Download
- 10.20 MB
ZIP File
Accompanying video: VRScroll_VRST24_Final_Encode.mp4 Subtitles for the accompanying video: VRScroll_VRST24_Final_Video_Subtitles.vtt 30-second Teaser video: VRScroll_VRST24_Preview_Encode.mp4 Auxiliary (supporting materials): Materials.zip
- Download
- 4.78 MB
References
[1]
2016. Tilt Brush by Google. https://www.tiltbrush.com/
[2]
2020. Wacom VR Pen. https://developer.wacom.com/en-us/wacomvrpen
[3]
Julian Adenauer, Johann Habakuk Israel, and Rainer Stark. 2012. Virtual reality technologies for creative design. In CIRP Design 2012: Sustainable Product Development. Springer, 125–135.
[4]
Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. 2018. SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3173574.3173759
[5]
Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh, and George Fitzmaurice. 2017. Experimental Evaluation of Sketching on Surfaces in VR. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 5643–5654. https://doi.org/10.1145/3025453.3025474
[6]
Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: As-Natural-as-Possible Sketching System for Creating 3d Curve Models. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (Monterey, CA, USA) (UIST ’08). Association for Computing Machinery, New York, NY, USA, 151–160. https://doi.org/10.1145/1449715.1449740
[7]
Olivier Bau, Ivan Poupyrev, Ali Israr, and Chris Harrison. 2010. TeslaTouch: Electrovibration for Touch Surfaces. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA) (UIST ’10). Association for Computing Machinery, New York, NY, USA, 283–292. https://doi.org/10.1145/1866029.1866074
[8]
Hrvoje Benko, Christian Holz, Mike Sinclair, and Eyal Ofek. 2016. NormalTouch and TextureTouch: High-Fidelity 3D Haptic Shape Rendering on Handheld Virtual Reality Controllers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16). Association for Computing Machinery, New York, NY, USA, 717–728. https://doi.org/10.1145/2984511.2984526
[9]
John Brooke 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.
[10]
David Johan Christensen and Kasper Stoy. 2006. Selecting a meta-module to shape-change the ATRON self-reconfigurable robot. In Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006. IEEE, 2532–2538.
[11]
Lacey Colligan, Henry W.W. Potts, Chelsea T. Finn, and Robert A. Sinkin. 2015. Cognitive workload changes for nurses transitioning from a legacy system with paper documentation to a commercial electronic health record. International Journal of Medical Informatics 84, 7 (2015), 469 – 476. https://doi.org/10.1016/j.ijmedinf.2015.03.003
[12]
Tomás Dorta, Gokce Kinayoglu, and Michael Hoffmann. 2016. Hyve-3D and the 3D Cursor: Architectural co-design with freedom in Virtual Reality. International Journal of Architectural Computing 14, 2 (2016), 87–102.
[13]
Tobias Drey, Jan Gugenheimer, Julian Karlbauer, Maximilian Milo, and Enrico Rukzio. 2020. VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376628
[14]
Lisa A. Elkin, Matthew Kay, James J. Higgins, and Jacob O. Wobbrock. 2021. An Aligned Rank Transform Procedure for Multifactor Contrast Tests. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 754–768. https://doi.org/10.1145/3472749.3474784
[15]
Hesham Elsayed, Mayra Donaji Barrera Machuca, Christian Schaarschmidt, Karola Marky, Florian Müller, Jan Riemann, Andrii Matviienko, Martin Schmitz, Martin Weigel, and Max Mühlhäuser. 2020. VRSketchpen: unconstrained haptic assistance for sketching in virtual 3d environments. In 26th ACM Symposium on Virtual Reality Software and Technology. 1–11.
[16]
Aluna Everitt and Jason Alexander. 2017. PolySurface: a design approach for rapid prototyping of shape-changing displays using semi-solid surfaces. In Proceedings of the 2017 Conference on Designing Interactive Systems. 1283–1294.
[17]
Shuo Feng, Weiping He, Shouxia Wang, and Mark Billinghurst. 2022. Pressure-sketch: a tablet-based design system in immersive VR. Virtual Reality 26, 3 (2022), 1207–1215.
[18]
Sean Follmer, Daniel Leithinger, Alex Olwal, Akimitsu Hogge, and Hiroshi Ishii. 2013. InFORM: Dynamic Physical Affordances and Constraints through Shape and Object Actuation. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST ’13). Association for Computing Machinery, New York, NY, USA, 417–426. https://doi.org/10.1145/2501988.2502032
[19]
Antonio Gomes, Andrea Nesbitt, and Roel Vertegaal. 2013. MorePhone: A Study of Actuated Shape Deformations for Flexible Thin-Film Smartphone Notifications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 583–592. https://doi.org/10.1145/2470654.2470737
[20]
Eric J Gonzalez, Eyal Ofek, Mar Gonzalez-Franco, and Mike Sinclair. 2021. X-Rings: A Hand-Mounted 360° Shape Display for Grasping in Virtual Reality. Association for Computing Machinery, New York, NY, USA, 732–742. https://doi.org/10.1145/3472749.3474782
[21]
Johann Felipe Gonzalez Avila, John C McClelland, Robert J Teather, Pablo Figueroa, and Audrey Girouard. 2021. Adaptic: A Shape Changing Prop with Haptic Retargeting. In Symposium on Spatial User Interaction (Virtual Event, USA) (SUI ’21). Association for Computing Machinery, New York, NY, USA, Article 4, 13 pages. https://doi.org/10.1145/3485279.3485293
[22]
Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 1957–1967. https://doi.org/10.1145/2858036.2858134
[23]
H.G. Hoffman. 1998. Physically touching virtual objects using tactile augmentation enhances the realism of virtual environments. In Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180). 59–63. https://doi.org/10.1109/VRAIS.1998.658423
[24]
Johann Israel, Eva Wiese, Magdalena Mateescu, Christian Zöllner, and Rainer Stark. 2009. Investigating three-dimensional sketching for early conceptual design—Results from expert discussions and user studies. Computers & Graphics 33 (01 2009), 462–473.
[25]
Hans-Christian Jetter, Roman Rädle, Tiare Feuchtner, Christoph Anthes, Judith Friedl, and Clemens Nylandsted Klokmose. 2020. "In VR, Everything is Possible!": Sketching and Simulating Spatially-Aware Interactive Spaces in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3313831.3376652
[26]
Daniel Keefe, Robert Zeleznik, and David Laidlaw. 2007. Drawing on Air: Input Techniques for Controlled 3D Line Illustration. IEEE Transactions on Visualization and Computer Graphics 13, 5 (2007), 1067–1081. https://doi.org/10.1109/TVCG.2007.1060
[27]
Daniel F. Keefe, Daniel Acevedo Feliz, Tomer Moscovich, David H. Laidlaw, and Joseph J. LaViola. 2001. CavePainting: A Fully Immersive 3D Artistic Medium and Interactive Experience. In Proceedings of the 2001 Symposium on Interactive 3D Graphics(I3D ’01). Association for Computing Machinery, New York, NY, USA, 85–93. https://doi.org/10.1145/364338.364370
[28]
Hyunyoung Kim, Celine Coutrix, and Anne Roudaut. 2018. Morphees+: Studying Everyday Reconfigurable Objects for the Design and Taxonomy of Reconfigurable UIs. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3173574.3174193
[29]
Kin Chung Kwan and Hongbo Fu. 2019. Mobi3DSketch: 3D Sketching in Mobile AR. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300406
[30]
Jessica Lo and Audrey Girouard. 2014. Fabricating Bendy: Design and Development of Deformable Prototypes. IEEE Pervasive Computing 13, 3 (2014), 40–46. https://doi.org/10.1109/MPRV.2014.47
[31]
John C. McClelland, Robert J. Teather, and Audrey Girouard. 2017. Haptobend: Shape-Changing Passive Haptic Feedback in Virtual Reality. In Proceedings of the 5th Symposium on Spatial User Interaction (Brighton, United Kingdom) (SUI ’17). Association for Computing Machinery, New York, NY, USA, 82–90. https://doi.org/10.1145/3131277.3132179
[32]
Payod Panda, Charles Ho, and Derek Ham. 2021. Morphaces: Exploring Morphable Surfaces for Tangible Sketching in VR. In Creativity and Cognition (Virtual Event, Italy) (C&C ’21). Association for Computing Machinery, New York, NY, USA, Article 8, 1 pages. https://doi.org/10.1145/3450741.3465387
[33]
Raf Ramakers, Johannes Schöning, and Kris Luyten. 2014. Paddle: Highly Deformable Mobile Devices with Physical Controls. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 2569–2578. https://doi.org/10.1145/2556288.2557340
[34]
Majken K. Rasmussen, Esben W. Pedersen, Marianne G. Petersen, and Kasper Hornbæk. 2012. Shape-Changing Interfaces: A Review of the Design Space and Open Research Questions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for Computing Machinery, New York, NY, USA, 735–744. https://doi.org/10.1145/2207676.2207781
[35]
Christian Rendl, David Kim, Sean Fanello, Patrick Parzer, Christoph Rhemann, Jonathan Taylor, Martin Zirkl, Gregor Scheipl, Thomas Rothländer, Michael Haller, and Shahram Izadi. 2014. FlexSense: A Transparent Self-Sensing Deformable Surface. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 129–138. https://doi.org/10.1145/2642918.2647405
[36]
Vincent Rieuf, Carole Bouchard, Vincent Meyrueis, and Jean-François Omhover. 2017. Emotional activity in early immersive design: Sketches and moodboards in virtual reality. Design Studies 48 (2017), 43–75. https://doi.org/10.1016/j.destud.2016.11.001
[37]
Jackie Ritchie, Joselle Bontilao, Sarah Kennelly, Jack Topliss, Jessica Dunn, Andre Renaud, Tim Huber, Barro W de Gast, and Thammathip Piumsomboon. 2021. COMFlex: An Adaptive Haptic Interface with Shape-Changing and Weight-Shifting Mechanism for Immersive Virtual Reality. In Asian CHI Symposium 2021 (Yokohama, Japan) (Asian CHI Symposium 2021). Association for Computing Machinery, New York, NY, USA, 210–214. https://doi.org/10.1145/3429360.3468214
[38]
John W Romanishin, Kyle Gilpin, Sebastian Claici, and Daniela Rus. 2015. 3D M-Blocks: Self-reconfiguring robots capable of locomotion via pivoting in three dimensions. In 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 1925–1932.
[39]
Alexa F. Siu, Eric J. Gonzalez, Shenli Yuan, Jason B. Ginsberg, and Sean Follmer. 2018. ShapeShift: 2D Spatial Manipulation and Self-Actuation of Tabletop Shape Displays for Tangible and Haptic Interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173865
[40]
Hyunyoung Song, François Guimbretière, Chang Hu, and Hod Lipson. 2006. ModelCraft: Capturing Freehand Annotations and Edits on Physical 3D Models. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (Montreux, Switzerland) (UIST ’06). Association for Computing Machinery, New York, NY, USA, 13–22. https://doi.org/10.1145/1166253.1166258
[41]
Anthony Steed, Eyal Ofek, Mike Sinclair, and Mar Gonzalez-Franco. 2021. A mechatronic shape display based on auxetic materials. Nature communications 12, 1 (2021), 1–10.
[42]
Patrick L. Strandholt, Oana A. Dogaru, Niels C. Nilsson, Rolf Nordahl, and Stefania Serafin. 2020. Knock on Wood: Combining Redirected Touching and Physical Props for Tool-Based Interaction in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376303
[43]
Hemant Bhaskar Surale, Aakar Gupta, Mark Hancock, and Daniel Vogel. 2019. TabletInVR: Exploring the Design Space for Using a Multi-Touch Tablet in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300243
[44]
Ryo Suzuki, Rubaiat Habib Kazi, Li-yi Wei, Stephen DiVerdi, Wilmot Li, and Daniel Leithinger. 2020. RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 166–181. https://doi.org/10.1145/3379337.3415892
[45]
Ryo Suzuki, Eyal Ofek, Mike Sinclair, Daniel Leithinger, and Mar Gonzalez-Franco. 2021. HapticBots: Distributed Encountered-Type Haptics for VR with Multiple Shape-Changing Mobile Robots. Association for Computing Machinery, New York, NY, USA, 1269–1281. https://doi.org/10.1145/3472749.3474821
[46]
Zsolt Szalavári and Michael Gervautz. 1997. The Personal Interaction Panel – a Two-Handed Interface for Augmented Reality. Computer Graphics Forum 16, s3 (1997), C335–C346. https://doi.org/10.1111/1467-8659.16.3conferenceissue.35 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/1467-8659.16.3conferenceissue.35
[47]
Huawei Tu, Xiangshi Ren, and Shumin Zhai. 2012. A Comparative Evaluation of Finger and Pen Stroke Gestures. Association for Computing Machinery, New York, NY, USA, 1287–1296. https://doi.org/10.1145/2207676.2208584
[48]
Philipp Wacker, Adrian Wagner, Simon Voelker, and Jan Borchers. 2018. Physical Guides: An Analysis of 3D Sketching Performance on Physical Objects in Augmented Reality. In Proceedings of the Symposium on Spatial User Interaction (Berlin, Germany) (SUI ’18). Association for Computing Machinery, New York, NY, USA, 25–35. https://doi.org/10.1145/3267782.3267788
[49]
Eric Whitmire, Hrvoje Benko, Christian Holz, Eyal Ofek, and Mike Sinclair. 2018. Haptic Revolver: Touch, Shear, Texture, and Shape Rendering on a Reconfigurable Virtual Reality Controller. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3173660
[50]
Jacob O Wobbrock, Leah Findlater, Darren Gergle, and James J Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI conference on human factors in computing systems. 143–146.
[51]
Jiangtao Wu, Chao Yuan, Zhen Ding, Michael Isakov, Yiqi Mao, Tiejun Wang, Martin L Dunn, and H Jerry Qi. 2016. Multi-shape active composites by 3D printing of digital shape memory polymers. Scientific reports 6, 1 (2016), 1–11.
[52]
Robert Xiao, Julia Schwarz, Nick Throm, Andrew D. Wilson, and Hrvoje Benko. 2018. MRTouch: Adding Touch Input to Head-Mounted Mixed Reality. IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018), 1653–1660. https://doi.org/10.1109/TVCG.2018.2794222
[53]
Wen Ying and Seongkook Heo. 2023. VRScroll: A Shape-Changing Device for Precise Sketching in Virtual Reality. In 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). 815–816. https://doi.org/10.1109/VRW58643.2023.00250
[54]
Shigeo Yoshida, Yuqian Sun, and Hideaki Kuzuoka. 2020. PoCoPo: Handheld Pin-Based Shape Display for Haptic Rendering in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376358
[55]
Chih-Han Yu, Kristina Haller, Donald Ingber, and Radhika Nagpal. 2008. Morpho: A self-deformable modular robot inspired by cellular structure. In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 3571–3578.
[56]
Xue Yu, Stephen DiVerdi, Akshay Sharma, and Yotam Gingold. 2021. ScaffoldSketch: Accurate Industrial Design Drawing in VR. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 372–384. https://doi.org/10.1145/3472749.3474756
Index Terms
- Enhancing VR Sketching with a Dynamic Shape Display
Recommendations
Demonstrating VRScroll: A Shape-Changing Device for Precise Sketching in Virtual Reality
CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing SystemsSketching precisely in Virtual Reality (VR) is challenging due to the lack of a physical surface for users to lean on, feel the contact and the movement of the pen, and constrain the pen movement. Using physical surface interfaces, such as a touch ...
TeslaMirror: Multistimulus Encounter-Type Haptic Display for Shape and Texture Rendering in VR
SIGGRAPH '20: ACM SIGGRAPH 2020 Emerging TechnologiesWe propose a novel concept of a hybrid tactile display with multistimulus feedback, allowing the real-time experience of the position, shape, and texture of the virtual object. TeslaMirror consists of a gross shape display based on 6 DOF robot, fine ...
Experimental Evaluation of Sketching on Surfaces in VR
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsSketching in immersive 3D virtual reality (VR) environments has great potential for a variety of interactive 3D design applications. Precisely sketching the intended strokes in mid-air, however, can be a challenge. In this paper, we present a set of ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2024
633 pages
ISBN:9798400705359
DOI:10.1145/3641825
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 October 2024
Check for updates
Badges
Honorable Mention
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
VRST '24: 30th ACM Symposium on Virtual Reality Software and Technology
October 9 - 11, 2024
Trier, Germany
Acceptance Rates
Overall Acceptance Rate 66 of 254 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 145Total Downloads
- Downloads (Last 12 months)145
- Downloads (Last 6 weeks)41
Reflects downloads up to 31 Dec 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in