An Artists' Perspectives on Natural Interactions for Virtual Reality 3D Sketching

3D sketching as formally defined by Arora et al. [6] is “a type of technology-enabled sketching where: (i) the physical act of mark making is accomplished off-the-page in a 3D, body-centric space, (ii) a computer-based tracking system records the spatial movement of the drawing implement, and (iii) the resulting sketch is often displayed in this same 3D space, e.g., via the use of immersive computer displays, as in virtual and augmented realities (VR and AR)” (Arora et al.  [6], p. 149). This way of sketching is flexible and fast [102], and is intuitive for 3D input [49, 98]. Due to these advantages, several companies have released applications that enable users to sketch and design in 3D such as Tilt Brush [39] (now open-source OpenBrush [35]), Gravity Sketch [92], and Quill [66]. These examples of commercial 3D sketching software have made 3D sketching available in various disciplines, including art, modeling, filmmaking, architecture, visualization design & research, medicine, and cultural heritage [100].

Despite the advantages of 3D sketching, correctly positioning a stroke in 3D space is challenging, as users are affected by high sensorimotor [105] and cognitive [13, 79] demands, depth perception issues in stereo displays [12, 16, 17], and the absence of physical support [7]. Previous work has studied the control and ergonomic aspects of sketching in mid-air [7, 57] and the learnability of 3D sketching [13, 105] to identify the cause(s) of these positioning inaccuracies. Other work has studied the advantages and disadvantages of 3D sketching as a medium for creativity and design by comparing it against pen-and-paper [47]. Finally, previous work has also studied how 3D sketching affects the act of ideation [32, 112]. Here, we aim to understand the expectations of artists for 3D sketching, focusing on their needs for alternate unimodal interactions and multimodal interactions within 3D sketching.

Since the early 1990s, previous research has proposed multiple novel interaction devices and techniques for 3D sketching [10]. The devices include pens [33, 46, 85, 101] and physical surfaces [31, 52] that aim to provide a surface to draw on. For example, Elsayed et al. [33] demonstrated that active haptic feedback reduces errors in VR when a physical surface is not present. The interaction techniques, like virtual surfaces  [8, 11, 61], beautification [11, 34] or novel metaphors to create strokes [50, 52, 86] aim to reduce the sensorimotor and cognitive demands of 3D sketching. For example, Barrera Machuca et al. [11], via Multiplanes, empowered participants by assisting them in sketching with snapping and beautification of strokes, which reduced the participants’ cognitive and sensorimotor demands. Finally, another approach uses visual guides to improve the user’s shape accuracy [14, 41, 97, 113]. One limitation of these approaches is that they mostly focus on unimodal interactions that use either controllers, gestures, or gaze.

We were able to identify previous work focusing on unimodal and limited multimodal interactions for 3D sketching systems (see Table 1). Some previous research outside of VR focuses on the effect of multimodal interactions on creativity [118] or user experience [110]. There has also been a lot of work that uses multimodal interactions for 3D modeling using CAD systems [20, 25, 74, 91, 96]. For example, participants in the study conducted by Wolf et al. [110] reported that using multimodal interactions, instead of unimodal interactions, allowed them to feel a higher state of presence. Another example is VR-CAD, in which Bourdot et al. [20] reported that using natural interactions allowed the participants to intuitively manage CAD objects, minimizing complications that are commonly expected of CAD applications. The advantages of using multimodal interactions for design in VR include: a higher sense of flow, higher intuitive use and lower mental workload, and a higher sense of presence [110]. They also provide similar creativity levels to unimodal interactions [118]. Due to the advantages they provide, it is important to understand how to add or incorporate multimodal interactions in 3D sketching systems.

Multimodal interactions are the combination of multiple input types like gestures, speech, pen, gaze, and touch. The combination of these inputs can have three properties: synchronous versus asynchronous, symmetric versus asymmetric, and dependent versus independent [63]. Synchronous interaction is one where the user can perform multiple interactions at the same time, whereas, in an asynchronous interaction, the actions do not need to happen at the same time. An example of synchronous interaction would be selecting an object while using speech to tell the system to change colors at the same time. With asynchronous interaction, one could select an object and then give a spoken instruction to change the color after the selection was made, but not at the same time. Symmetrical interactions are usually bimanual in nature, and the actions on one hand mirror what the other hand is doing; asymmetrical interactions do not have to mirror what the other hand is doing and thus act independently of each other. An example of symmetrical interaction is when one is painting a mirror image, like painting the wings of a butterfly with both hands. Similarly. asymmetrical interaction is, e.g., when one is painting and one hand has a menu palette and the other has a brush, so both hands are performing different tasks. A dependent interaction is one where an interaction depends on the other to accomplish a task, such as hands working in tandem with one hand controlling the color palette while the other hand controls the brush. In contrast, the interactions involved do not rely on each other in an independent interaction. An independent interaction could be when both hands can act as brushes and each can be used to draw, regardless of one another.

Researchers have continued investigating multimodal interaction since the work from Bolt [19]. Another important work by Hinckley et al. looked at Pen+Touch and described what type of interactions were possible [45]. Multiple studies have also examined multimodal gesture and speech inputs using mid-air gestures [5, 24, 43, 64, 71, 106]. For example, using gesture elicitation, Williams et al. [106] showed that multimodal interactions are essential to interact with augmented reality (AR) HMDs in a natural way. Yet, some have examined only a subset of gestures, such as 2D gestures (e.g., multitouch) [69, 84] or paddling gestures [48]. While multiple studies using gesture + speech interactions have been studied, they have concentrated in 2D environments or 3D environments using desktop displays, with less work in AR/VR [82, 107]. It is possible to find multimodal interaction examples, such as Internet of Things home controls [54], 3D computer-aided design in a 2-dimensional environment [58], and web browsing on televisions [71, 75, 108]. For example, Wittorf et al. [108] found that users preferred certain mid-air gestures when interacting with a wall-display. The larger amount of work has been in multimodal gesture and speech fusion and recognition [19, 24, 53, 81], although some of them have used limited gesture sets [26] or limited speech dictionaries [64]. Overall, the research conducted thus far has tested input feasibility and human adaptability and created more intuitive and discoverable interaction sets [109], yet the type of inputs are limited, without clear transferability to more complex applications.

Participants. For the study, thirteen participants (8 females and 5 males) studying art at the local university were recruited. Their ages ranged between 20 and 28 (M = 22.4, SD = 2.4). Eleven participants had previously used AR/VR before. Seven indicated that they had their vision corrected, two through the use of glasses, one through contact lenses, and the other four did not specify. Two participants had previously experimented with Tilt Brush (now called OpenBrush) in VR. Two participants were double majors (computer science and fine art), while the rest were specifically fine art majors. Our study was limited to artists, because they have experience working in fine arts from their classes and studio practice, giving the study a population closer to that of established artists as compared to naive users. All participants were either enrolled in or recently graduated from a Bachelor of Fine Art (BFA) degree. The BFA program requires foundational class work that includes coursework in drawing, painting, sculpture, and digital media. Additionally, the pre-survey questionnaire allowed for the participant to volunteer information, such as specific applications they had worked with (e.g., ZBrush, Maya, Blender, Cinema 4D, Autodesk 3DS Max), but none of our participants volunteered that information.

Equipment. The 3D sketching program was run on an Alienware Aurora R14 desktop equipped with an AMD Ryzen 9 5900 12-core processor running at 3.0 GHz, with a total of 32 GB of system RAM and an NVIDIA Ge-Force RTX 3080 with 26 GB of onboard memory with the GPU running at 1710 MHz. The desktop ran Microsoft Windows 11 Home (version 10.0.22621, build 22621). The participants used an HTC Vive Pro Eye with two controllers and two lighthouses to access the 3D Sketching system. Finally, a Go-Pro Hero 7 with a 128 GB memory card was used to record the interaction of the participants during the study. For the 3D sketching application, we used a fork of OpenBrush v2.3.0 [35]. OpenBrush was run through Unity 3D, version 2019.4.25f1 (as recommended by the contributors). As the participants drew in the 3D Sketching program, their drawings were recorded by Unity’s Recorder to capture the participant’s perspective from the HMD.

Procedure. Upon arrival, each participant followed a series of tasks, mentioned here and in Figure 4. The participant first signed three forms: a vision attestation form, the consent form, and a pre-survey questionnaire (including their demographics and any prior VR experience). They were informed of what a semi-structured interview is, and that this study uses semi-structured interviews to collect data. The participant then watched a video tutorial that showed the basics of using OpenBrush in VR ¹. After the video, the participant was fitted with the HMD and controllers to repeat the basic operations they had just seen in the OpenBrush video tutorial, allowing them to practice by replicating what they had just watched. When the tutorial was finished, the participant removed the HMD and controllers. Next, the participant watched another video that explained the different types of unimodal and multimodal natural interactions and their categories ². After the second video and before starting the study, the researchers allowed the participants to ask any questions, but none of the participants had questions on the procedure.

For the first part of the study (Phase One), the participant was fitted with the HMD/controllers and was tasked with drawing a 3-dimensional dog using any tool available in the 3D sketching application. The participant had a 2-meter by 2.1-meter rectangular space, free of obstacles, in order to sketch freely in OpenBrush. Phase One ended after 10 minutes, at which time the participant was given the choice to take a 2-minute break or continue directly to Phase Two. In Phase Two, the participant was fitted with the HMD/controllers (if the 2-minute break was taken) and was tasked with drawing a ground, a path, and a tree. Just as in Phase One, they were allowed to use any tools they liked but had a 15-minute time limit. Participants could add additional constructs to the scene as long as they had drawn the ground, the path, and the tree, and the time limit had not been reached. Some of the completed works can be observed on the right-hand side of Figure 2 and Figure 3 with the corresponding participants appearing on the left. After removing the HMD, the participant was asked to complete a System Usability Scale (SUS) [9, 65]. Afterward, a post-study interview (see the supplementary materials for the interview questions) was done, where the participants told the researchers about their experiences. Finally, participants were offered class credit or a $20 Amazon gift card for their time. In total, the entire study lasted around 57 minutes.

Data Collection. Each participant’s movements in the physical space during their sketching session were recorded using a Go-Pro camera (Figure 2 and Figure 3). The camera was fixed with an overview of the sketching area. The recording of the session began after participants watched the video tutorial about the multimodal interaction. After the video tutorial, we asked participants if they had any questions about the video and the task, but none of the participants had questions. Then, participants were asked to follow the “think-out-loud” method [77] while sketching to help understand their thought processes as they drew specific elements. Moreover, the study researcher periodically asked the participants if multimodal interaction techniques would assist with the participant’s current task. This occurred every time the participant switched to something “new” or after asking for assistance about how to navigate the system. If the participants asked for assistance, the researchers provided verbal help to resolve the issue and followed up by asking if an alternative interaction technique could have aided in accomplishing that task or prevented the issue. Sometimes, participants did not have a response to follow-up questions. To allow researchers to examine the participants’ actions while suggesting other unimodal or multimodal interactions, the screen of the PC running OpenBrush was recorded.

Following a completed participant session, the video/audio recordings from the Go-Pro camera were synchronized with the headset recordings from Unity. This allowed a simultaneous analysis of the participants’ real-world motions and what they saw in the virtual world. The audio recordings were also automatically transcribed using the Microsoft Word Web App’s transcription feature [67]. Each of the two authors reviewed half of the transcripts to fix transcription mistakes. When necessary, corrections were made to the transcriptions using the Go-Pro recording.

Data Analysis. Following an approach inspired by Braun and Clarke [22, 23], this study uses researcher reflexivity as a pillar of the thematic analysis. Because of this epistemological and ontological position, researchers were able to avoid measuring inter-coder agreement. The agreement poses the existence of a researcher “bias” and tries to minimize it (as well as performing consensus coding), anchored in the belief that there is an objective way of coding and that this objective method is more desirable. Instead, researchers in this study recognize the situated nature of coding and its inherent partiality and subjectivity [28].

Three researchers conducted a qualitative analysis of the interviews. Two of these researchers (or, more specifically, coders) ran the user study and were familiar with the data. The third coder had previous thematic analysis experience and helped the lead coders through the process. Two coders were male, and one was a female. Two coders had undergraduate degrees in Fine Arts, either in animation and digital art or in film/cinema production. One had formal training in drawing and sketching, and the other had over 14 years of experience in 2D art. The third coder did not have formal training in drawing or sketching.

Of the two researchers who led the user study, one coder was assigned seven interview transcripts, and the other was assigned six. Each transcript was assigned to the coder who originally conducted the interview. This assignment leverages familiarity with the data as key to analysis [18, 21]. The two coders used a template with columns for transcript excerpts, codes, and comments. The coders were further familiarized with the data by re-reading their transcripts and taking notes. Individually and inductively, they coded their transcripts to create a system to encode the data while keeping a list of this encoded data and their descriptions to track their own process. Then, they shared the coded data and discussed the construction of themes. The themes were refined in conversations among two coders who conducted the study and then proposed to the third coder for further discussion. For this final part, the third coder participated in the discussions and helped define the final themes. The lead coders met five times and three additional times with the third coder. Ultimately, the themes were proposed to the rest of the team for further discussion.

The analysis characterizes artists’ expectations about what features 3D sketching applications should have. The questions focus on identifying the tools of commercial 3D sketching systems that help artists in their sketching process (RQ1) or integrating unimodal and multimodal natural interactions into 3D sketching systems (RQ2). Recognizing that the participants are art students at Colorado State University, our results account for this population, which has specific cultural expectations of design tools [94, 103]. Our data indicates a familiarity with complex desktop tools, yet not enough experience with 3D sketching. The research indicates that the participants (i.e., artists) wanted to improve current features and add other input modalities. It also shows that artists expect 3D sketching systems to have more features than other design tools. The following section further develops the paper’s themes to describe the requested features and input modalities artists suggested for 3D sketching systems.

Following the need to run 3D sketching evaluations with stable systems [15], the usability of the 3D sketching system was evaluated by having participants answer the SUS questionnaire. A post-study interview was conducted to get participants’ opinions on the current 3D sketching system. (See the supplementary materials for the interview questions.) Participants rated the system positively, as shown in Figure 8. The overall average score was 77.01 (SD = 12.5), corresponding to letter grade B, showing the system’s usability is above average. When looking at the individual statements, participant statements indicate usability. One participant’s evaluation with an average score was 4.15 (SD = 0.99) stated, “I think that I would like to use this system frequently.” Another evaluation with an average rating of 4.08 (SD = 0.76) declared, “I thought the system was easy to use.” Both respondents scored the system above average. The positive rating of the system, which earned it the letter grade B, is supported further by additional participant interviews. P5, for example, commented, “I love this, this is great!” P1 mentioned, “That is nice, kind of very satisfying.” Similarly, in feedback about the application, P2 said that “it is very nice.”

RQ1 was about the tools of commercial 3D sketching systems help artist in their sketching process. Our participants identified four elements, Menu, Shape Creation, Manipulations, and Animation, of the commercial system study that did not meet their needs or could be improved. Here we describe our findings for each one of them:

Menu. One feature not available in the sketching application is the ability to maximize the display space by removing the menu on the non-dominant hand. This menu could potentially distract the artists and hide important parts of the sketch when the artist is doing a visual search. Moacdieh et al. [70] studied the effects of cluttering the display, which ultimately affects the performance. It has been shown that having a higher field of view (FOV) [83] leads to better performance. Therefore, manually hiding the menu when artists are not using it maximizes the FOV, which would allow for a better visual search, and increase the performance of the artist when sketching in VR. Another requested feature was a preview pane to test different combinations of brushes, brush sizes, and colors. Just like the menu on the non-dominant hand, this preview pane should only be shown at the artist’s request. Removing it when not necessary would maximize the FOV.

Shape Creation. One of the desired features requested by artists was the ability to create basic geometric shapes. Gravity Sketch and ShapesXR (Table 2) are VR applications that already provide this functionality, which is similar to creating shapes on desktop applications (Table 2). Basic 2D shape creation in VR has previously been explored by Barrera Machuca et al. [11] by using beautification. Two artists also wanted to generate shapes with the assistance of AI. Chen et al. [27] have explored using natural language to generate colored 3D shapes, as well as composite 3D shapes, such as tables and chairs. Incorporating such technologies would allow artists to create basic geometric shapes.

Manipulations. The artists also wanted to manipulate the strokes by turning them into straight lines or splitting them. Turning a non-straight stroke into a straight line has previously been demonstrated by the use of beautification [11]. This turns a sequence of dots that look almost straight into a straight line. On the other hand, Jiang et al. [52] showed that splitting a stroke can be done via a cut operation. The sketching application has a feature called Snip that breaks the stroke, however, this function affects the stroke not only at the location where the split occurs, but also the shapes of the newly split strokes. The cut operation only affects the stroke at the site where it is cut, leaving the rest of the shape of the two strokes unaffected. One artist wanted to manipulate objects closer by hand-sculpting them. Currently, Blender (Table 2) supports basic sculpting in VR, but for finer control and details, users still have to launch the desktop counterpart of Blender to finalize those sculptures.

Animation. The sketching application includes some brushes with animations. Several applications, such as ShapesXR (Table 2), also provide limited animation when hovering over an object. Although only one artist suggested creating animations, it would be worthwhile if a future release allows artists to create a basic animation that extends beyond the animated brushes that are already included in the application.

All of these suggested tools share a commonality: they resemble features found in other desktop and VR applications used for 3D modeling. Our results suggest that new 3D sketching applications should include tools that artists are already familiar with from other software, as the artists expect to find similar tools across software.

RQ2 concerned identifying which natural multimodal interactions 3D sketching systems can add to help the artist in their sketching process. Interestingly, we found that our participants mostly focused on tools not related to the process of sketching in a broad sense, but on the interactions that help manipulate the strokes such as object interaction, selection, and manipulation. We also found that when proposing multimodal interactions, our participants only proposed interactions with two different input methods, e.g., gesture and speech or controller and gesture, but no more.

Brush. As an alternative to using the VR controller to control the brush, only two artists mentioned the pen. One would think that since all the participants are artists, that they would elect to go with the pen as the preferred input device. When we analyzed the demographics of two artists who suggested a pen, we noticed that one is in their senior year and the other had recently graduated. The artist who had just graduated used VR extensively and had worked with a team to create a VR game. Because of their familiarity with HMDs and their different uses, this artist chose the pen because they wanted the most appropriate tool for interacting in VR as a sketching medium. The second artist chose the pen because they focused on digital drawing and pixel art as their primary form of artwork. These artists’ daily activities and type of artistic practice affected their choice of using a brush.

Object Interaction. We observed a relationship between artists who wanted to use bimanual interaction versus artists who wanted to use gestures. A possible explanation for the choice of interaction may depend on the number of hours that each of these participants spends in front of a computer as opposed to those artists who spend the majority of their time on a controller. The artist who suggested a bimanual interaction spends considerably more time on a computer, using both hands to manage the keyboard and mouse. In contrast, the artist who predominately uses the controller must make broader gestures due to both hands being tied up with the controller.

Menu. For an alternative way to access menu, we looked at the two most prominent modalities requested by the artists to access the menu: controller and speech. The artist who chose the controller is accustomed to using similar types of input devices, such as game controllers, therefore, it may have been a natural preference influenced by their experience in gaming. The artist who chose speech selected this interaction as a personal preference, possibly related to phone use, wherein users can use voice activation to engage with their phones. What we can gather from these artists is that accessing the menu through other modalities is a personal choice that may be influenced by other familiar technologies in their environments.

Manipulations. Interestingly, there were two artists who wanted to split a stroke by pulling it apart. Both artists had experienced VR before and both were in a similar age group. The first artist works on sculptural art projects using both hands, therefore, using both hands to pull apart the stroke could be a natural translation from the physical world. The second artist primarily works with desktop applications, possibly influencing their choice to select using a bimanual interaction for pulling a stroke apart. We hypothesize that this artist may want to pull apart a stroke as they predominately use their hands to do separate operations when on the keyboard and mouse. For deleting strokes, two participants wanted to use gesture and speech combined. We hypothesize that using both gesture and speech to specifically choose a stroke and use speech to indicate what must be done with it could be a simplified process for this artist.

Selection. For selecting strokes, 7 artists suggested speech. Upon checking pre-survey demographics, we noticed that they all have previous experience with VR systems. We hypothesize that because VR applications often come with audio-visual tutorials, but not an in-depth written manual, speech is a more natural response in the environment—allowing the application to process the verbal command on their behalf.

RQ3 was about examining the artists’ opinion on the usability of commercial 3D sketching systems. Our findings show a high usability score for the sketching application (Figure 8). However, there were four artists who rated the system with a ‘D’ or below average. The pre-survey questionnaire revealed that these artists played computer games for less than 6 hours on a weekly basis. We speculate that for people who are accustomed to playing games, it is relatively easy to switch from one game controller to another, or a VR controller for that matter [3]. Also, the only way that artists were able to directly interact with the sketching application was through the VR controllers. Using this unfamiliar input technology might have been challenging for them, leading to frustration and a lower usability score.

During the interviews, the artists made suggestions for tools that could help them during their sketching process. One tool would be a disappearing menu and a preview pane. These should only appear at the request of the artist, and can be tied to an on-and-off mechanism, e.g., a toggle button on the controller or a spoken command, such as “show me the menu”.

The artists also mentioned the generation of primitive shapes, which would help them save time for sketching. Two artists mentioned the possibility of being assisted by AI. Currently, large language models, such as ChatGPT by OpenAI [78], can be integrated into game engines  [95]. Such similar software can help artists as novel tools.

Developers can implement beautification techniques, such as those displayed in Multiplanes [11] to estimate the shape that the artist is attempting to draw. Methods such as Nestor [42] and GCN [104], which uses a neural network, have also been shown to be successful. For the splitting function, as in HandPainter [52], a stroke splitting can break a mesh into two watertight pieces, and two new objects should be added as nodes to the scene graph.

Allowing artists to implement their animation can be challenging. Developers can create empty animation objects that can be used by the artist to assign modified object properties, and these can be saved in the timeline as they are added. By letting the artists play the animation, they can see how the object changes based on the modified properties.

During our study, the only way to sketch in the application was through the VR controllers. Every artist has a personalized style, and the sketching application should be able to accommodate that style. As it was suggested by the participants, other unimodal interactions should be made available to cater to each artist’s style. Additionally, some participants suggested adding multimodal interactions. Therefore, we recommend adding other input modalities, such as gesture and speech, to help the artists navigate the software in a more natural way. By adding this additional functionality the participant will be able to use the software in a natural way.

Additionally, the hardware used in this study exists to meet general VR user needs. While the current hardware allows 3D sketching to be done, designing controllers that are more specifically oriented toward sketching and other artistic applications could offer additional options for users to interact effectively with the system. Designing hardware options in varying sizes could also accommodate users with different hand sizes and different ranges of mobility. By increasing the hardware options along with the software capabilities, users can direct more conscious effort towards working and creating and less on interacting with the tools.

Having multiple modalities available for interaction provides a versatile toolkit for expressing users’ ideas. Users can start with quick, fluid gestures to lay down the basic structure of their sketches. They could then use speech or controller functions to refine their ideas. This encourages experimentation and exploration as users are not confined to specific tools or techniques. This freeform approach can lead to novel design concepts.