Understanding Version Control as Material Interaction with Quickpose

Design ideation and creative practice are important domains where practitioners engage materials. Previous research in these areas offers critical touchstones which we rely on for our discussion of material interaction. Prior empirical studies offer evidence that broad exploration [21], variation [86], iteration [11], and reflection [13] are valuable parts of a design process. These studies show that these activities help people achieve better design outcomes, but they do not propose explanations of why these activities occur when practitioners engage materials, nor how to reason about these activities in interfaces. For example, in Section 2.1 we argue that iteration, in part, occurs during a material interaction because practitioners’ goals change in response to working with a material. We then propose that supporting iteration within material interaction means supporting the complete reframing, continual refining, and the splitting and fracturing of goals. This prior empirical work thus lays the foundation for our design principles and measures.

In addition to these empirical studies, researchers also recommend design principles for supporting creativity with interfaces. While Quickpose aligns with these guidelines, such as exploration [67], history-keeping [80], and margin-keeping [26], our goal in this work is to build practice-oriented recommendations and principles which help designers reason about why these recommendations are useful and how we might extend them.

Tools for investigating alternatives, from interactive systems to programming constructs, support our theme of Local Knowledge of Materials. This category of research investigates supporting users in managing and exploring variations through parallel authoring for image manipulation [85] and interface design [34]. Other approaches to variation have been explored in programming constructs for multiverse analysis [49, 72]. These projects do not track versions, but allow users to explore local variations effectively. In a similar vein, novel tools allowing users to explore similar examples in web design [68] led to better design outcomes [46]. These parallel authoring and example search interfaces strongly resonate with our principle of local knowledge building through variation and comparison, which we discuss in Section 2.2.

Tools to support the organization and communication of the design process have found success through contextual annotation and flexible arrangement, which support our theme of Annotation for Holistic Interpretation. These projects have explored how annotations and virtual canvas editors help communicate the design process [15, 60] and help manage ideas and support reflection [39, 50, 51, 77]. These works offer support for our principle in Section 2.3 that knowledge practices center on contextualization and interpretation of versions. Quickpose builds on this work as a version control canvas that primarily supports a practitioner in engaging the material at hand, including the history of versions and annotations, rather than communicating or reflecting afterwards on a design process.

Interfaces for creative coding have prioritized rapid exploration and iteration, and have focused on connecting many small programs together to form directed acyclic graphs (DAGs) which lend themselves to variation [7, 22, 78, 98]. While these implicitly support variation of individual parts of the program, they do not version the program itself. This prevents users from explicitly combining changes across components and using their program history beyond what they manually duplicate. Similarly, p5.fab [84] supports rapid experimentation by allowing users to directly control machine execution, but does not help scaffold this experimentation over time. While all of these tools seem to support material interaction with programs (and, with p5.fab, manufacturing devices), they do not manage versions of their programs. In contrast, Quickpose replicates the entire program in each version, allowing users to easily explore and manage entirely different programs which originated from the same starting point. Quickpose is designed to support more involved exploration and discovery in line with our discussion in Section 2.1.

Previous research has investigated how programmers take up code as a material, often without the explicit support of their programming tools. First, they highlight how programmers already explore, experiment, reason with, and build tacit knowledge about programs and their executions [5, 6]. Second, they emphasize how existing programming environments poorly support programmers in this process [5, 6, 73]. Yoon et al. [94, 96] study programmers’ backtracking—returning to an earlier state of the program—behaviors to show how current version control tools fail to support these behaviors. In this paper we explore how version control tools can better support programmers in material interaction: reasoning and reflecting on the entire process of programming across versions.

Besides supporting experimentation, iteration, and history navigation, version control tools have also focused on annotation of version histories for reflection [88], learning [10, 28, 30], and sharing [52, 62]. These projects highlight the importance of annotation for bringing context to versions and aiding interpretation (Section 2.3).

De Rosso et al. [18] discuss their redesign of Git, a widespread version control system. They were able to address common issues with the system by identifying conceptual errors from users and redesigning Git’s abstractions to address those errors. While this line of research is valuable, our investigation in this paper is not meant to deliver design guidance for a system like Git. Git is primarily intended for managing changes in the context of software engineering [18]: communicating and combining changes, tagging versions for institutional purposes (marking releases of software, demarcating experimental or abandoned features, etc), and maintaining a single, common history to restore in case of failure. While these are important aims of software engineering, this paper explores how version control can support an entirely different set of priorities, which we discuss in Section 2.

Quickpose represents versions as canvas elements with thumbnails of those versions’ render outputs. Clicking on a version in the Quickpose interface updates the Processing IDE and output window to that version, while shift-clicking a version creates a new version (a fork), and updates the IDE to that version (Fig 3). To ensure no history is lost, states with child nodes are locked for editing, but are always able to fork.

Because the version history graph is represented as interactive canvas elements, versions can be grouped, anchored and parented via arrows, styled, and scaled just as users interact with any other canvas element. This also allows versions to be easily integrated into existing patterns of working with canvas editors to organize versions alongside notes, images, and other graphics.

While forking and navigation between versions are manual operations, Quickpose also saves “checkpoints”: edits to a version which are saved internally when a user executes a save or when they navigate away from a version having made edits. This ensures that edits are not lost or accidentally overwritten even if a user has not dedicated a new version to hold those changes yet.

Because versions in Quickpose behave like other canvas elements, style elements like color can be used to not only informally demarcate certain versions, but can also be used as a handle for technical features, such as the “export by color” menu which exports all the thumbnails and code iterations of a single color into a folder in the user’s local project folder.

In order to study how practitioners use Quickpose, regular backups of the canvas are made in addition to logs of user activity. These logs and archives are stored in the user’s project folder locally. These logs gave us a detailed picture of user interaction with the tool, which we analyze in our user study.

While Processing was a convenient place to implement Quickpose given its user base of artists and its focus on programs which generate visual outputs, the Quickpose interface can be used in other domains. As a front-end system for managing versions, it could handle versions in a variety of other places, such as an image editor, programming interface, or design tool. We describe the API structure (“Core Abstractions” in Fig 6) below. These are the only abstractions needed to implement Quickpose for a new programming environment. For example, an image editor could interface with the core of Quickpose as long as it was able to change between versions, make copies of versions, and serve thumbnail images. Due to the small and well-defined nature of these abstractions, which make few assumptions about programming environment or language, we argue Quickpose can be straightforwardly adapted to a variety of contexts.

We recorded each instance of a user forking a version. For each project, we then calculated the depth (the number of nodes from the new forked node to the root node of the version tree) and the distance (the number of nodes on the shortest path from the new forked node to the previous forked node). We show the results for three projects in Figure 7. In this plot we see multiple backtracks, represented as sharp slopes up, or in other words, a decrease in depth from the previous fork. These jumps indicate users went back significantly in their history to start again in a new direction.

Simultaneous development on multiple branches, in contrast to backtracking, is identified by a high distance from the previous fork but a small change in depth from the previous fork. For example, a participant may have two long branches, b1 and b2, fork from the end of branch b1, then fork from the end of b2, between these forks is a small difference in depth (the two forks are similar distances away from the root node), but a high change in distance(the two forks have a large traversal distance between them). Figure 8 shows all instances of forking in our dataset, arranged according to both distance and change in depth; the area marked “Parallel Development” represents the high distance, low difference-in-depth area.

From user canvases, we recorded instances of multiple “final” versions, marked either by a text label indicating finality, or, as in Figure 9, a filename indicating an export. In the Figure 9 canvas, the participant was generating video files from desired versions and annotating the version with the filename of the generated video. We therefore interpreted two kinds of "marking final": (1) either referencing an export of the artifact, like the example above, or (2) indicating that a version was to be kept for future development. For example, P2 generated multiple variations of the same sketch but with different aesthetic qualities for different situations. P2 described in an interview with researchers how these versions were intended to form a "palette" for future development. In the four projects where these annotations were present, we recorded 53 annotations of this kind, although one project contained 36 of these. Many of these annotations (all but 9) were not on leaf nodes and many were on multiple parts of a branch. Marking many successive iterations or intermediate versions each as “final” or “to export” indicates an evolving goal, where marking versions across disparate branches as “final” indicates bifurcation of goals. We did not record any instances of participants using the “Export by Color” feature. We will note, however, that this feature only exported still images, and all of the participants in our study worked on video animation projects.

In addition to forking behavior, we also recorded when participants navigated between versions by clicking on them on the Quickpose canvas to update the editor and the render. In Figure 10, we show two Quickpose projects as directed graphs with both the forks (shown in grey) and navigations (shown in blue) between nodes (shown in green). These visualizations show how participants navigated broadly across their version history even in cases in which their forking history shows a more linear behavior; at each version, this navigation history shows evidence of how participants reflected on and compared versions across their history.

In order to investigate broad (low depth and high degree) versus deep (high depth and low degree) exploration, we chart the nodes of four projects by degree (how many edges a node has) and depth (its distance from the root node) in Fig 12. We theorized that many high depth, low degree nodes would indicate deep exploration, while low depth, high degree nodes would indicate broad exploration. Our results visualized here show that while participants were able to explore deeply, there were relatively few nodes that had more than three edges (i.e., forked more than twice). (See discussion in Section 7, including how future work may support broad exploration.)

We saw participants use a single annotation to describe multiple versions in four of the projects. Participants either referenced multiple versions explicitly, by “grouping” them together in the Quickpose interface (Fig 11, left), or implicitly, through positioning an annotation near multiple versions on the canvas and describing the content of the versions (for example, annotating “these versions rotate instead of translate”). However, we found that these annotations that described multiple versions appeared only once per project. (See discussion in Section 7 for possible reasons why.)

In six of the projects, participants used interpretive language to annotate a version. This included comparisons, such as "Reminds me of crops growing" from P3 as shown in Figure 11, or included emotional descriptors like "magical" or "interesting." In the data collected, participants used 13 annotations in this way out of a total of 178 canvas annotations.

We found 25 contextualizing annotations in eight of the projects. These annotations added information about the version, why it was created, or why it was or was not interesting to the participant. For example, participants contextualized versions with annotations like "Failed rotation experiment" and "The previous effect but animating forward then reversing in a loop".

All of our participants mentioned how Quickpose supported exploration of multiple goals by backtracking in order to explore new possibilities. P3 highlighted how Quickpose helped support goal formation through supporting exploration of many different directions: P3: “Quickpose is really good for when you don’t really know where you’re going.” P2 discussed how Quickpose supported iteratively backtracking as their goals refined and changed, allowing them to collect all their explorations in a single place for reflection and re-use: P2: “I would start a branch and explore it all the way down...you keep some of your changes and incorporate that into a different direction, and other times you...start fresh and...do something totally different...in which case you go back to the start and branch off.” Our participants all indicated that saving and storing many versions quickly allowed them to more easily explore, especially if their goals were not well defined. P3 highlighted how the low barrier to make a new version allowed saving versions which might only become meaningful later: P3: “If you don’t know where you’re going, then you’ve got to record everything.” P2 additionally highlighted how quickly saving many versions enabled a sustained focus on exploration, rather than thinking about version control as a separate step in the programming process: P2: “What’s really great about [Quickpose] is that it frees you up...I can always go back to the root and go in a completely different direction after working for three hours in one way and it’s all still contained and you don’t break that flow.”

P3 and P2 expressed that Quickpose supported context building, idea generation, and comparison. P2 draws attention to how seeing all the versions at a glace supported more exploration: P2: “There’s more immediate inspiration...Having [all the versions] there gives you more ideas...I think that’s why [the project] branched out so quickly.” All participants also mentioned that Quickpose lowered the barrier for exploring variations. P1 described how Quickpose supported them iterating locally on a single “base file:” P1: “You can have a base file from where you can try different things...tweaking it a little or tweaking it a lot.” P1 and P2 indicated that exploring local variations aided building local knowledge about programs. P2 highlighted how easily forking aided their ability to form tacit knowledge about their code, discussing how seeing local variations together on the Quickpose canvas helped them understand the parameter space of their program: P2:“...just playing with numbers, I wouldn’t even really understand [the code]—I really do have to save this, because I don’t really understand what it’s doing.” In all of these excerpts, participants found that seeing all variations at a glance and creating and navigating versions quickly helped how they form ideas and reason about their programs, features which we designed Quickpose to enable specifically.

P3 mentioned that Quickpose supported annotating and organizing in an idiosyncratic way: P3: “That’s what Quickpose is for, whatever kind of organization you want.” P2 referred to the Quickpose canvas as a “mindmap,” allowing them to map out their ideas and explorations through successive versions. P3 requested that web pages be integrated into the Quickpose canvas itself to store reference material within the version history, analogous to “leaving a comment in the source code.” P3 cited how they would like to embed articles and resources that were not just about a single version, but were aiding the exploration more holistically.

Through the interviews, participants expressed their perception of their programming interactions while working with Quickpose. These excerpts present evidence that participants’ perceptions of their experiences with Quickpose aligned with our theories of material interaction. While testimony from participants is not sufficient on its own, it further supports that our measures of material interaction were in fact measuring salient aspects of the user experience.

In Section 6, we discussed how practitioners reason about their work with Quickpose, which includes not only understanding and reflecting on past versions, but also future directions, intentions, and goals. We found evidence that once version control (Quickpose) became the platform for reasoning about the materials at hand (and taken up as a material itself), participants also indicated future directions, latent possibilities, and unarticulated goals. For example, P1 annotated versions with what possibilities or variations had yet to be explored. Analogous to feedback (understanding what a computer has just done) and feedforward (understanding what a computer is about to do), we propose a similar counterpart to backtracking (navigating to an earlier version of the code) as forwardtracking: outlining future directions, intentions, and unexplored space in a user’s “mindmap.” Our discussion of continuous goal reformation highlighted how working with materials is a continuous process of reflecting on prior versions, engaging actively with the material at hand, and using both to project into the future. With our distilled themes of material interaction in hand, we argue that version control should then support "forwardtracking" as a practice of setting and refining goals, which includes outlining unexplored or latent possibilities in materials, as an important part of supporting material interaction.

Using the results of our study, we present an updated set of measures for material interaction for future research (Table 3).

Measuring continual goal reformation. We found that significant backtracks could be characterized by a sharp decrease in depth from a previous fork, simultaneous development could be indicated by forks with a small change in depth but a high distance between them, and that versions marked as exports or to be used later were valuable ways to capture how goals were iteratively refined. While we did not record any instances of our “Export by Color” feature, we saw through canvas annotations, later supported by interviews, that participants had exported animations as videos, which our tool didn’t support. We also observed that participants left “final” versions within Quickpose canvases to draw on within future projects. Therefore, instead of looking for “exported” versions, we instead propose measuring versions which are either linked to files outside of the Quickpose project, versions marked for future use, or versions which have been drawn into other projects.

Measuring contextual exploration. We found frequent navigation between disparate parts of the version history indicative that participants were reflecting on and comparing across the version history. Additionally, we saw that high depth and low degree nodes could indicate deep exploration while low depth, high degree nodes could indicate broad exploration. Although we did not measure many low depth, high degree nodes, we argue that this is not because the measure is faulty, but because Quickpose did not adequately support that interaction. All of our participants requested greater support for procedurally making versions by varying one or a few variables—such as Quickpose generating a design gallery across the variables—indicating they felt unsupported in broad variation.

Measuring holistic, linked annotation. While contextualizing versions were our most common category of annotations, they were also the most vague categorization, and frequently overlapped with interpretive annotations. We therefore refined our measures for contextualizing and interpretive annotations, combining them to annotations which support interpretation and understanding of why a version is meaningful.

These measures were developed within the context of Quickpose (and version control systems more broadly), and therefore we cannot predict how they will generalize to other tools and domains. However, the diverse fields, methods, and practices from which we drew our discussion of material interaction might indicate that they have the potential to be helpful elsewhere. As we discuss below, this can only be answered by putting our themes, principles, and measures into practice in other domains.

While this paper does not propose a comprehensive theory of material interaction, in this section we outline how our themes might already be helpful for making sense of previous recommendations for design tools in HCI and help reconcile them with new findings. For example, in his seminal creativity support tools paper [80], Shneiderman recommends keeping a detailed history to record which actions or alternatives have already been tried by the user. This record is intended to support iteration and exploration of the material at hand. However, this recommendation also raises questions: Why is keeping a record of previous actions beneficial? How should interactions with this history be measured? Sterman et al. find evidence of practitioners preferring a less detailed, lower fidelity history in some cases [81]—how can this evidence be reconciled with Shneiderman’s design recommendation? In this example, we see our themes and design principles of material interaction working to bolster the original recommendation, offering meaningful dimensions to study, and also offering an explanation of Sterman et al.’s finding. First, our themes can help explain why keeping records might be beneficial. For example, record-keeping aids in building local knowledge through enabling users to reflect on how their changes have impacted their artifact, or it helps users backtrack if their goals change in response to the material and they need to pursue another direction. Second, our measures suggest that we could study history-keeping by tracking both how users backtrack to earlier versions but also how they use earlier versions to inform current work by navigating across the version history. Finally, these themes of material interaction offer an explanation of why some practitioners prefer lower-fidelity versioning methods [81]. For example, the authors cite a performer who prefers keeping only the audio to their performances so that they can have a record of previous work but also feel able to gradually modify or spontaneously develop the performance. With the themes of material interaction in hand, one reason for this practice could be that versions which do not contain enough information to fully recreate the artifact require exploration and variation on every backtrack. Thus, low-fidelity capture is a process constraint that forces a practitioner to reengage with their material at every step. Where before this practice might have seemed counter-intuitive in light of Shneiderman’s recommendation, when viewed with a lens of material interaction, the goal of maintaining a low-fidelity capture to spurn exploration is aligned with Shneiderman’s call of engendering innovation through history keeping. Additionally, our themes of material interaction generate new research directions beyond the scope of Quickpose. For example, it suggests exploring a more expansive definition of version control systems, which go beyond tools for backing up, collaborating, and managing code to also include tools for supporting reflection, comparison, and goal formation. While we propose material interaction as a lens through which to study and design version control systems, material interaction within other areas of HCI research remain to be explored. We hope these themes, design principles, and measures will scaffold generative work with material interaction in other domains, which might require them to be reworked and appended.