Inkeraction: An Interaction Modality Powered by Ink Recognition and Synthesis

As shown in Figure 2, Inkeraction represents a handwritten document as a hierarchical graph with three levels of containment (shown with elbow connectors) and various marking relationships (shown with curved connectors). This representation captures the inherent structure of nested handwritten notes, as explored in prior work [31, 34]. Unlike previous recognition work that treated nested structures as monolithic entities, hindering interactions with intra-structure components (e.g., Ye et al. [78] recognized a list as a standalone object), Inkeraction preserves these structures, enabling fine-grained interaction with each lower-level object. For example, the list in Figure 2a is represented as a container comprising individual list items, each with a bullet and word, allowing us to interact with any of these objects. Theoretically, the hierarchical containment relationships can have an unlimited number of levels, but we found that three levels were sufficient to represent the objects within our scope.

In addition to the containment relationships, we add marking relationships like connecting and pointing to for inter-structure interactions.

To describe the recognized objects, the hierarchical containment and varied marking relationships, Inkeraction employs a dictionary of labels. The labeling system is similar to prior work [31, 34], but modified with the opinions from our expert participants, developers, and designers. See Figure 3 for common labels in the dictionary.

To obtain the hierarchical relationship graphs, our recognition system employs a segmentation model and a classification model working in tandem. The segmentation model iteratively groups strokes representing meaningful objects at different levels within a handwritten document, ultimately creating a stacked hierarchy of Graph Neural Networks (GNNs), see Figure 4. At each level, a GNN receives a set of input objects and clusters them, generating output objects for the next level. The classification model then analyzes the embeddings generated by the GNNs at different levels, assigning labels to these objects, such as “word” for lower-level objects and “list” for higher-level ones. Finally, Inkeraction performs text recognition and character segmentation specifically for text objects.

Segmentation. At the initial level of the GNN hierarchy, each node represents a single stroke or a non-stroke object (e.g., an image). Each stroke undergoes normalization, smoothing, and processing through a stroke embedding model to extract an initial fixed-size representation. The stroke embedding model uses a stack of Transformer layers [72] followed by a fully connected layer. Non-stroke objects have their embeddings retrieved from a constant dictionary based on their type. At each level, initial edge embeddings are calculated based on heuristic features between nodes, such as relative distance and size, similar to the approach described in [78].

The node embeddings are then iteratively updated. Each iteration begins by identifying a fixed number of neighbors for each node using the k-nearest neighbors algorithm, based on the Euclidean distance between nodes. Subsequently, each node’s embedding is updated through a multi-head attention mechanism over the edge embeddings and the embeddings of its neighbours followed by a residual update. Let h_i denote the node embedding for node i, and e_ij denote the edge embedding for the edge connecting node i to its neighbor node j. Each node embedding h_i is updated with the output of a multi-head attention mechanism, where h_i serves as the query, and the set of e_ij|h_j act as the keys and values. The attention layer is followed by a reduction sum and a fully connected layer, which calculates the updated node embedding by combining the mean of the attention outputs with the original node embedding (see Figure 4b). We repeat the update iteration a fixed number of times.

After updating the node embeddings, merge logits are calculated for each neighbor of each node in the GNN. The merge logits between source node i and target node j are also computed over a multi-head attention mechanism, where the queries, keys, and values are h_i, e_ij, and h_j, respectively. The attention outputs are then reduced to a two-dimensional logit tensor representing the probability of merging or not merging. If the “merge” logit value is larger than the “no merge” logit value, we conclude that node i wants to merge with node j.

Cliques of n nodes that mutually desire merging are fed to a merger model, which utilizes a multi-head attention layer followed by a reduction sum layer. The attention layer computes an embedding from each node by attending over other node embeddings and the corresponding edge embeddings. The merged node embedding is computed as an average over all the attention heads and outputs (see Figure 4c). After updating and merging node embeddings, the edge embeddings are recomputed. If any nodes have been merged, their associated edge embeddings are recalculated based on the heuristic features.

At each level, we iteratively repeat the node embedding update and merge process until no more cliques of nodes desire merging or a maximum number of iterations is reached. We then repeat the process with another set of model weights to create a higher-level GNN from the current node embeddings. For example, if we previously merged strokes into words, we would now merge words into lines of text. This iterative segmentation process enables us to hierarchically cluster strokes and non-stroke inputs into three GNNs: L1, L2, and L3.

Classification. The classification algorithms process the segmentation results to categorize objects and relationships. Each node in the GNNs is assigned a label (e.g., “word” at L1, “textline” at L2, and “list” at L3). The classification model is a single fully-connected layer that maps a node embedding to the logits for the given number of classes.

The relationships between objects are computed in the same way as the merge logits. Instead of producing the merge logits, the relationship model produces logits for each type of relationship (e.g. “no relationship”, “underlines”, and “points_to”) between two objects.

Text Handling. For text objects, Inkeraction runs text recognition using an LSTM-based model [9]. The recognized text is then segmented into characters using a Transformer-based model [33], which allows us to calculate text geometrics such as character sizes and word baselines.

Stroke cloning and curve synthesis are achieved through geometric transformations. A stroke, represented by s, is a sequence of N coordinates. We can apply a 3 × 3 geometric transformation matrix T to s for translation, rotation, and scaling: T · s.

To clone a stroke c from s, we copy s and transform it to the desired position: c = T · Copy(s).

Curve synthesis involves identifying a template and calculating a T to modify the shape and position of the template. For instance, when a user moves an arrow composed of multiple strokes, the original arrow serves as the template. We identify the longest stroke in the arrow as its tail and define its two endpoints, A and B, as reference points. Based on user’s action, we calculate the after-interaction coordinates for these points: A′ and B′. The transformation matrix T, represented by a scaling factor (r), a rotational factor (θ), and two translation factors (dx and dy), is then determined by solving the equations A′ = T · A and B′ = T · B. This calculated T is applied to all strokes of the template to generate the synthesized curve.

There are different use cases for synthesizing handwriting text. We can extend existing handwriting fragments (e.g., finishing an incomplete sentence), generate entirely new handwritten content (e.g., producing a standalone phrase), and modify existing handwritten text (e.g., correcting a character in a word). Inkeraction extends and generates handwriting building upon the work of Graves [24], and modifies existing handwriting using the algorithms suggested by Maksai et al.[47].

Graves [24] proposed a Recurrent Neural Network (RNN) model along with a “priming” technique to synthesize styled handwriting. This technique leverages the model’s inherent ability to attend to its previously generated strokes. By design, when synthesizing a stroke sequence x for a piece of text t, the model analyzes the already generated strokes x_p and their corresponding text t_p to produce the remaining strokes x_r for the remaining text t_r, where t = t_p + t_r. This mechanism ensures a smooth transition and consistent style. By changing x_p and t_p (i.e., priming model with x_p and t_p), the model can generate strokes for text t_r in different styles. Even strokes from users not in the training data can be used to prime the model, mimicking their handwriting style. However, the generated strokes may still exhibit some bias towards the training data. This bias can be reduced by incorporating diverse writing samples, encouraging the model to generalize ink synthesis.

Inkeraction uses this priming technique for handwriting extension and generation. When extending handwriting, the model is primed on the existing strokes (i.e., x_p), conditioned on the current label (i.e., t_p) and the text to be extended (i.e., t_r). Similarly, for generating new content, the model is primed on template strokes, conditioned on the label of the template strokes and the text to be generated. User-produced strokes (e.g., from previous writing) can serve as templates, allowing personalized stroke synthesis without RNN retraining. Alternatively, built-in template strokes in a standard style can be used to generate strokes in a generic font. After generating strokes, we apply geometric transformations to ensure their proper positioning within the context. We further improved the stopping criteria compared to [24]. The original model used a heuristic to stop synthesis when the sampling window exceeded the last character, which could fail in certain cases. In our work, instead of using this heuristic, we predict an end-of-sequence binary value. In other words, the original model predicts (d_x, d_y, PenUp/PenDown) for each step but our implementation predicts (d_x, d_y, PenUp/PenDown, isEndOfSequence), where d_x and d_y represent the pen coordinates for the step.

To modify existing handwriting, we use the model from Inkorrect [47]. This model extends the work of Graves [24] with a dedicated style capture component, which makes modified strokes more similar to the original handwriting.

This feature checks for misspelled words in handwriting and displays them with a red underline, as shown in Figure 8a . The user can tap on the red underlines to bring up a popup menu for confirmation or rejection (Figure 8b). If confirmed, a correction will be synthesized to match the user’s handwriting. If the synthesized correction overlaps with existing strokes, we can use the relationship graph to shift strokes, as described in Section 5.1. Other rendering versions are shown in Figure 8c . Spell Check suggestions are rendered as soon as Inkeraction recognizes the handwriting, and the rendered underlines will stay there until the user confirms or rejects them.

This feature predicts the content the user is going to write and renders the prediction in purple strokes that match the user’s handwriting, see Figure 9a . The user can tap on the prediction to accept it (Figure 9b), or they can ignore it and continue writing (Figure 9c). Predictions are displayed as soon as Inkeraction recognizes the handwriting.

Word processors offer many ways to format typed content, which can be applied to ink content as well. For example, users can tap on a bullet to change a bulleted list to a numbered list (Figure 10 left), or use gestures to adjust list spacing (Figure 10 middle). Given the vast range of formatting possibilities, further research is warranted to determine the specific options necessary for handwriting. For example, underlining may be redundant as users can easily do it themselves. It is also important to consider how these formatting options can be activated within the constraints of limited ink interaction space.

Note-taking tasks frequently involve transcribing content from various sources, which can be simplified using the Transcription feature. Figure 11 shows this feature enables copying of digital content from websites. Moreover, Transcription can be enhanced with microphones and cameras. For instance, it can be used in conjunction with speech input technologies to take real-time lecture notes. Additionally, optical character recognition (OCR) enables the digitization of notes written on whiteboards.

Sometimes we need to revise notes for aesthetic purposes. With the Beautification feature, users can select text and apply beautification effects. There are many ways to beautify handwriting, and Inkeraction offers two algorithms for this purpose, as shown in Figure 12. The unsynthesized algorithm aligns text items and corrects margins using text baselines, yielding a more natural-looking result (Figure 12b). The synthesized algorithm rewrites text using a generic font, producing a more standardized output (Figure 12c).

Figure 1f shows the Brainstorm feature, which can help the user to expand an existing list. Similarly, Inkeraction can be combined with LLMs to summarize, bulletize, shorten, elaborate, and rephrase written text.

Generative models can empower inking interfaces by integrating API calls [54]. This transforms inking surfaces into hubs of ideation and computation, where users can express their thoughts and commands and ask the surfaces to comprehend and execute them. Examples include scheduling calendar events, setting alarms for deadlines, or creating task sequences through lists or arrows as shown in Figure 13.

While the Assisted Manipulation feature simplifies note arrangement, generative models offer even greater assistance. Figure 14 demonstrates how Inkeraction can automatically organize a cluttered page with a LLM. Inkeraction provides essential segmentation and text recognition data, which the LLM leverages to identify and represent relationships among the segmented content in a structured format. Inkeraction then visualizes these structures through connectors and handwritten text. Researchers have developed algorithms for relation extraction and structure generation. For example, TextSketch [67] employed traditional natural language processing to guide users in creating diagrams and visual notes. Studies have shown that LLMs outperform traditional models in relation extraction [73]. Due to the free-form nature of user-generated content, such as sketches and diagrams, developing algorithms for effective organization and structuring remains an ongoing research challenge. Inkeraction’s recognition capabilities offer valuable input for external algorithms to comprehend relationships, while its synthesis abilities allow machines to generate structures on behalf of users. As LLMs evolve, we believe their ability to extract various types of relations could also advance, especially with the help of Inkeraction’s relationship graph.

This feature supports the user cases in Figure 6 and Figure 7. It enables users to reorder, drag, delete, mark off, and move items within a list and a connected graph, both of which can be created by the users themselves. To utilize Assisted Manipulation, the user first selects an item by tapping it with their finger or circling it with the stylus. Once an item is selected, the user can drag the item to achieve the desired outcome.

This feature utilizes a built-in dictionary of misspelling-correction pairs to suggest appropriate corrections for recognized words. The dictionary includes the four underlined misspellings in the following paragraph: “The Twelvth Night is a marvellous comedy. The play begins with a disasterous journey to sea in a yaucht.” These four words are predominantly used in British English but spelled differently in American English. If a correction is too lengthy, the feature will shift neighboring words, as shown in Figure 8c .

This feature automatically suggests the subsequent word in a phrase based on the recognition of the initial word. It employs a compact dictionary that stores first-second word pairings. For testing purposes, we implemented the following phrases from a grocery list: “Salad dressing; Peanut butter; Cottage cheese; Cream cheese; Baking powder; Potato chips.” Refer to Figure 9 for an illustration of the interaction.

This feature converts selected web text into handwriting. We used the Android WebView to display a web page within the application. The user interface resembles the one in Figure 11.

This feature changes handwriting using either the unsynthesized or the synthesized algorithms in Figure 12.

We designed this feature to provide suggestions for a packing list. To use this feature, select a list and two predefined items, “swimsuit” and “coat,” will be automatically added to the list.

We recruited participants who regularly took handwritting notes. Twelve participants (P1 - P12) with an average age of 30.17 (SD=3.97) joined the study. Each participant signed a consent form before taking part in the study, and was given a notebook as a token of appreciation after the study. All participants (8 males, 4 females) had a college degree or above. Half of the participants took handwritting notes daily while the other half took weekly. Eight participants owned tablets with a stylus pen.

After answering demographic questions, the participants were shown the prototype and each experienced the six features in a balanced Latin square order. For each feature, a facilitator demonstrated how to use it first, and then the participant tried it out on their own. For the Beautification feature, the participants were able to try two different algorithms.

After each feature, each participant was asked to rate the feature with three 7-point scale Single Ease Questions (SEQs) :

The participants were also asked to clarify their ratings.

After trying all of the six features, participants ranked them based on their overall experience. Then, the facilitator explained how the features worked and asked participants about their thoughts on privacy concerns. Finally, we collected feedback on new ink interaction features that participants would like to see in future note-taking apps. Each study session took about 45 minutes. d

We recorded the screen and audio during the study. Then we analyzed the data from the SEQs and rankings, along with the participants’ transcribed comments. Finally, we identified common themes in the data.

Spell Check, Beautification, and Auto Completion features were the easiest to learn. Participants were already familiar with similar spell checking and word completing features from other word processors, as P3 said, “It’s just like Google Docs.” Beautification was easy to learn because it could be triggered by a simple button. Assisted Manipulation, however, was considered as the most difficult to learn. P6 and P8 attributed this to the feature discoverability while P1 and P9 felt there were many different ways one could manipulate strokes. P6 explained, “if you don’t tell me [the different stroke manipulation operations], I won’t use it.” P1 suggested “more time to demonstrate” will make Assisted Manipulation more learnable.

Not all highly learnable features were considered usable. Spell Check and Beautification were rated the easiest to use, while Auto Completion was rated the most difficult to use. Nine participants complained about the performance and latency of the Auto Completion feature. The Auto Completion feature takes time to recognize written strokes before it can generate a suggestion, but users may write much faster. P1 commented, “It’s too slow, if I think of something, I will keep writing and it won’t catch up.” Moreover, the implemented version required the user to use a tap finger gesture to confirm the suggestion, which further slowed down the writing process. P5 suggested the feature should “automatically complete every word” without the confirmation. All participants were able to experience the Assisted Manipulation feature with their own list and connected graph. However, some participants were frustrated by the gestures they need to learn, which resulted in a moderate usability rating.

Despite its low learnability and moderate usability ratings, Assisted Manipulation was considered as the most useful feature. Participants were excited about the flexibility and assistance provided by the feature. P6 said, “handwriting is fast and [this makes] modifying easy, so you will benefit from both physical and digital writing tools.” Participants noted that the manipulation could be especially useful for learning (P1, P5, P10) and planning (P8, P11, P12). Spell Check was also highly rated. Five participants (P1, P6, P8, P9, and P12) mentioned that spell errors were common in handwriting, and P9 further emphasized that, “I can’t tolerate any imperfections in my notes. [Such as] not looking good, typos. So I usually type. If you have this feature, I will start writing notes.”

Auto Completion and Brainstorm were ranked as the least useful features. Auto Completion suffered from its performance issues, as explained previously. Given the current performance, four participants (P3, P9, P11, P12) thought the feature could still be useful for complex words or sentences. As for the Brainstorm feature, participants felt the current prototype was too limited since only two items were generated. Some participants thought even a more capable AI would not be able to get what they wanted. P7 said, “when people are writing, they are not writing basic stuff and the topics may not be common knowledge”, which was echoed by P1, P2, and P10.

Figure 18 shows the overall ranking results.

Assisted Manipulation was the most favored one, with a mean rank of 2.42 (SD = 1.39), Ten people ranked it in their top 3 choices. Spell Check was the runner-up, which is supported by its high learnability, usability, and usefulness scores. Brainstorm was the least favored one, with a mean rank of 4.58 (SD = 1.16). The Brainstorm feature was criticized for its usefulness. The Auto Completion feature was considered the second least favored one due to its performance issues.

Beautification and Transcription were middle-ranked. When asked about their preference for different beautification algorithms, four participants (P1, P6, P10, P12) preferred the unsynthesized algorithm. They explained that they wanted to keep their writing styles. Five participants (P3, P4, P5, P7, P8) liked the synthesized algorithm better. They said that they wanted to make their writing more recognizable for sharing purposes. Three participants (P2, P9, P11) had no preference and suggested keeping both algorithms for different use cases. Seven participants suggested that the beautification feature should include the option to change writing styles and have typed fonts as well. Due to implementation limitations, it was difficult to select sentences in the Transcription feature, and participants wanted to use it to transcribe more than just words. P1 commented, “it’s just like copy and paste…[but it] would be better if you can select a sentence.” This was also shared by P4, P8, and P9.

Seven of the participants had no privacy concerns regarding the recognition and synthesis models. However, the other five participants felt that there should be enhanced security measures in place. P1, P2, P5, and P6 expressed that no stroke data should be uploaded to servers. They suggested that the models should run locally on devices instead. P6 also mentioned that he kept important personal information like passwords in notes, so he would not feel comfortable storing this information in the cloud. P11 said she would prefer to use these features with an authentication method that did not tie to her identity.

In addition to the feature suggestions mentioned above, the participants also provided advice on how to improve the Brainstorm feature and incorporate multimedia. Two participants wanted the Brainstorm feature to be more personal and more capable. P2 wanted the generative model to get to know her better and provide more specific information. For example, instead of adding “sunscreen” to the packing list, the AI should tell her which sunscreen to bring. P4 hoped the generative model could know his writing preferences and provide contextual suggestions, such as the date of today. P7, P9, and P11 shared common suggestions on building features for multimedia support, such as annotating multimedia, linking multimedia, and finding connections between multimedia and strokes.

To compare Inkeraction prototype with traditional inking software, we built a baseline version inside the prototype. The baseline disabled recognition and synthesis capabilities but had basic features such as pen, eraser, undo/redo buttons. In the Inkeraction prototype, users could select recognized objects by tapping on their strokes. This recognition-based selection was not available in the baseline. To enable selection in the baseline, we added a rectangular marquee selection tool, which selects all the strokes within the drawn rectangle, see Figure 19a .

Ten participants (P13 - P22) with an average age of 29.6 (SD=3.86) joined the second study. All participants (5 females, 4 males, 1 non-binary) signed consent forms and received appreciation notebooks, and they had a master degree or above. Four took daily handwritten notes, while the rest did so weekly.

After completing demographic questions, participants were briefed about Inkeraction and the study’s purpose. To compare Inkeraction’s features described in Section 6.2 with the baseline, we designed five tasks, excluding the Brainstorm feature due to its lack of a baseline equivalent. Figure 19 shows these tasks and demonstrates their completion in both baseline and Inkeraction. Table 2 outlines the typical user actions needed for task completion. Before each task, a facilitator provided instructions. For Correcting, Rearranging, and Aligning tasks, the facilitator helped the participants set up the content needed for each trial. Then each participant completed each task twice, once using the Inkeraction prototype and once with the baseline. They had ample opportunity to familiarize themselves with the tasks and tools before the trials began. To ensure counterbalancing, half of the participants commenced with an Inkeraction trial, while the others started with a baseline trial. The tool order was reversed after each task. Each participant completed a total of ten trials. For the Correcting and Writing tasks, relevant content was displayed on a screen during the trials.

Afterward, the participant compared Inkeraction features with the baseline via a 7-point Likert scale (See Table 4). Then, they saw unimplemented features via video, sharing their thoughts on Inkeraction’s potential as a thinking and inking tool. The entire study took about 45 minutes.

To assess the efficiency and effectiveness of inking tools, we measured the following metrics:

During the study, we captured screen and audio recordings. Analysis of these recordings and log data enabled us to calculate the metrics. We summarized the results from the Likert scale and transcribed participant comments, and aggregated the survey data for further analysis.

Table 2 indicates that Inkeraction users completed tasks with statistically significantly fewer actions, demonstrating increased efficiency. Notably, in the Aligning task, Inkeraction reduced the actions needed by a factor of 18.5 compared to the baseline.

We also showed the typical user actions we observed. Not all participants followed these typical actions, as they could make mistakes or used other tools. Examining the types of performed actions, we found that Inkeraction reduced the number of actions by minimizing writing actions, simplifying selection, and automating alignment. For example, in the Transcribing, Correcting, and Writing tasks, handwriting a word cost participants an average of 6.08 (SD = 1.27) strokes and additional erasing actions, while Inkeraction allowed the participants to compose a word through one or two actions.

Inkeraction’s recognition capabilities also aided in tasks requiring selection (e.g., Rearranging, Aligning). For example, in the Rearranging task, the marquee selection tool cost more actions to select a desired target compared to the recognition-powered tap-to-select technique ¹. It took a participant 2.29 (SD = 1.22) strokes on average to marquee select a desired target, while the tap-to-select technique only cost 1.28 (SD = 0.26) taps. The tap-to-select technique had larger selection area and thus required less efforts to use [46], which may explain the improvement.

Additionally, Inkeraction reduced the number of selection and dragging by automatically aligning items. For example, in the Rearranging task, a participant typically dragged over three items (Mean = 3.56, SD = 1.33) to complete the baseline trial, compared to only two drags (Mean = 2.00, SD = 0.00) in the Inkeraction trial. As an extreme example, P18 dragged 7 times in the Rearranging baseline trial, which took them 95.05 seconds. The automatic alignment made ink manipulating more efficient (less steps) and more consistent (smaller deviation) across different users.

Figure 21 shows the average time participants spent on each trial. Using Inkeraction, participants finished all the tasks statistically significantly faster than using the baseline. Comparing the data from all the trials (N = 100), there is a moderate positive linear correlation between the number of user actions and the completion time (r = 0.77). More actions would introduce more time, but the types of actions and the time gap between actions can also vary the time.

We also found that the Auto Completion feature, criticized by its lagging performance in Study I, still helped the participants with their Writing tasks. It statistically significantly reduced the number of actions and the completion time. This task has six two-word phrases. Auto Completion managed to suggest 5.30 (SD = 1.83) words on average, of which 4.80 (SD = 1.62) words were confirmed by the participants.

Table 3 presents the survey results from 27 human raters. Votes were converted to scores ranging from -27 to 27, with positive scores indicating preference for Inkeraction and higher scores are better. Out of 50 pairs of writing samples, \(76\%\) received positive scores. Inkeraction achieved a mean score of 8.7 (SD = 10.5) across all participants and tasks.

Inkeraction outperformed the baseline in all tasks, with the highest score in the Correcting task. This is likely due to Inkeraction’s Spell Check feature, which replaced entire misspelled words and ensured alignment, while some participants made character-level edits (e.g., erasing a “l” in “marvellous”) that resulted in unbalanced samples. The Rearranging task received the lowest score, although still favoring Inkeraction. The similarity between the samples in this task, where only two words were swapped, may account for this. Notably, the samples from P15, P18, and P19 received higher vote gaps in the Rearranging task. Among these samples, two baseline samples had misaligned words or bullets while the other baseline sample missed a stroke (due to an inaccurate marquee selection).

Examining mean scores by participant, we found that 9 out of 10 participants’ samples received positive mean scores, indicating improved writing quality with Inkeraction. An exception was P22, who spent significantly more time on baseline trials. For example, in the Aligning task, they spent 45 seconds on the baseline trial, while the mean completion time was 27.11 seconds. We think this indicated that P22 was spending extra efforts in these trials, which led to higher writing quality outperforming the writing quality afforded by Inkeraction. Further investigation is warranted to confirm this hypothesis.

Table 4 presents the Likert scale results, indicating that participants generally perceived Inkeraction as superior to the baseline. Inkeraction enhanced efficiency (low scores on S1), simplified interactions (high scores on S2 and S4), reduced effort (low scores on S3 and S5), and promoted focus (high scores on S6). Three participants remained neutral regarding S3, they mentioned that certain interactions remained equally challenging. For example, P17 noted that while Inkeraction reduced steps, it did not alter the intrinsic difficulty of dragging. Seven participants concurred that Inkeraction aided cognition, citing the reduced actions (P13) and simplified manipulations (P14, P17) as factors facilitating focus on thinking. The other three participants felt there was not enough evidence to support this statement. For example, P19 explained their negative score by saying “the tasks are too simple so there is little evidence.”

After watching other untested features, the participants were very excited about the potential use cases of Inkeraction, as expressed by P17, “I need these, right now!” The participants also provided suggestions. P22 thought that these features could be optimized for students, who need to take notes from slides and constantly organize notes. P20 provided ideas to improve the Auto Completion feature. Instead of using a finger tap to confirm a suggested word, they thought a user could indicate their confirmation by superimposing on the suggested word.

The recognition of strokes may not always match what the user expects. This can be due to the imperfect recognition algorithms, or the dynamic and versatile nature of ink. For example, a user may start writing a text block with multiple lines of text, but then decide to treat it as a list instead.

To address this issue, ink should be transparent as it approaches the user’s desired outcome, so the user can act accordingly [45]. In our prototype, we implemented a preview technique, which allows users to see the recognized structures and possible outcomes before they make a change. When the user drags an item, we use a force-directed layout algorithm to show the recognized structures (see Figure 22). The layout algorithm considers the containment relationships between nodes. The children of a node have a stronger connecting force than the forces between the node and its neighbors. Once the user hovers over a location for an extended time, we cancel the force-directed layout and render the possible outcome for that location. Newly synthesized items will be rendered in gray (see Figure 23b). This preview technique helps users to better align their expectations with the recognition results. It also makes the inkeraction more transparent, as users can see what is happening behind the scenes.

Limited user input makes it hard to understand the desired outcome when there are many possible outcomes. For example, when dragging an item into a list, the user may want to insert the item as a new list item, or merge the dragging item with an existing list item.

To address the challenge of understanding user intent, we can use micro-interactions to derive the user’s desired outcome. In our prototype, we implemented Activation Areas (AAs) to interpret the user’s intent during a drag operation. Each structure has its own AA, which is based on its bounding box. We add padding to each bounding box. Figure 24 shows an example of AAs in a scene with a list and a text block.

One design dilemma we faced was that the more help we offered, the more likely users would feel hindered. Users may want different levels of assistance depending on tasks and context.

To avoid hindering users, we propose two techniques: Unassisted Mode and AI Undo. In Unassisted Mode, users can perform actions without help, such as leaving an item overlapping with other strokes. To do this, users can use a two-finger drag gesture to move strokes without assistance. We also considered a postfix method called AI Undo. When Inkeraction performs some tasks on behalf of the user, an undo option will show up and allow the user to cancel the Inkeraction’s effects. The undo option will fade away in seconds to minimize interruptions. AI Undo can also be used for mode switching. See Figure 25 for examples.

However, these techniques can be tedious for users to use, especially if there are several steps involved. To address this issue, we propose personalizing a decision model for individual users. The decision model would take the interaction context as input and output the most likely helping steps. This would allow us to provide users with the level of assistance that they want.

Our user study found that the speed of handwriting recognition can impact the user experience. While it would be ideal for handwriting to be recognized and displayed as quickly as typing, the machine learning algorithms used for handwriting recognition are more complex than the straightforward input and output devices of keyboards and monitors.

To accommodate for latency, we suggest avoiding competition with the user or require them to take immediate action. For example, the current Auto Completion feature only provides suggestions for one following word after the user has typed a word. To avoid competition, we can provide longer suggestions. To avoid immediate actions, we can ask the user to react to suggestions after they have finished their writing task.