KR102771373B1 - Managing real-time handwriting recognition - Google Patents

Abstract

Methods, systems, and computer-readable media relating to techniques for providing handwriting input functionality on a user device are provided. A handwriting recognition module is trained to recognize tens of thousands of characters using a single handwriting recognition model and to have a repertoire including a large number of non-overlapping scripts. The handwriting input module provides real-time, stroke-order and stroke-direction independent handwriting recognition for multi-character handwriting input. In particular, real-time, stroke-order and stroke-direction independent handwriting recognition is provided for multi-character or sentence-level Chinese handwriting recognition. User interfaces for providing handwriting input functionality are also disclosed.

Description

{MANAGING REAL-TIME HANDWRITING RECOGNITION}

The present invention relates to providing a handwriting input capability on a computing device, and more particularly, to providing a real-time, multi-script, stroke-order independent handwriting recognition and input capability on a computing device.

Handwriting input methods are an important alternative input method for computing devices equipped with touch-sensitive surfaces (e.g., touch-sensitive display screens or touchpads). Many users, especially those in some Asian or Arab countries, are accustomed to writing in cursive and may feel more comfortable writing by hand as opposed to typing on a keyboard.

For certain logographic writing systems, such as Hanzi and Kanji (also known as Hanja), although alternative syllabic input methods (e.g., Pinyin or Kana) may be available for entering characters of the corresponding logographic writing systems, such syllabic input methods are not suitable when the user does not know how the logographic characters are spelled phonetically and uses imprecise phonetic spellings of the logographic characters. Therefore, the availability of handwriting input on a computing device becomes important to users who are not able to pronounce the words of the relevant logographic writing system well enough or at all.

Although handwriting input has gained popularity in some parts of the world, improvements are still needed. In particular, human handwriting is very diverse (e.g., in terms of stroke order, size, font, etc.), and high-quality handwriting recognition software is complex and requires extensive training. Thus, providing efficient real-time handwriting recognition on mobile devices with limited memory and resources has been a challenge.

Moreover, in today's multicultural world, users from different countries speak different languages, and often need to type in more than one script (e.g., type a message mentioning a movie title in English into Chinese). However, manually switching the recognition system to the desired script or language during the typing process is cumbersome and inefficient. Furthermore, extending the recognition capability of a device to handle multiple scripts simultaneously significantly increases the demands on computer resources and the complexity of the recognition system, thus severely limiting the utility of conventional multi-script handwriting recognition techniques.

Additionally, conventional handwriting techniques rely heavily on language- or script-specific idiosyncrasies to achieve recognition accuracy. Such idiosyncrasies are not readily portable across other languages or scripts. Thus, adding handwriting input capabilities for new languages or scripts is a daunting task that vendors of devices and software do not undertake lightly. As a result, users of many languages have lost an important alternative input method for their electronic devices.

Conventional user interfaces for providing handwriting input include an area for accepting handwriting input from a user and an area for displaying handwriting recognition results. For portable devices with small form factors, significant improvements in user interfaces are still needed to generally improve efficiency, accuracy, and user experience.

This specification describes a technique for providing multi-script handwriting recognition using a universal recognizer. The universal recognizer is trained using a large multi-script corpus of writing samples for characters in different languages and scripts. The training of the universal recognizer is language-independent, script-independent, stroke-order-independent, and stroke-direction-independent. Thus, the same recognizer can recognize mixed-language, mixed-script handwriting input without requiring manual switching between input languages during use. Additionally, the universal recognizer is lightweight enough to be deployed as a standalone module on mobile devices to enable handwriting input in different languages and scripts used in different regions of the world.

Furthermore, since the universal recognizer is trained on stroke-order-independent and stroke-direction-independent spatially-derived features and does not require any temporal or sequence information at the stroke-level, the universal recognizer offers numerous additional features and advantages over conventional time-based recognition methods (e.g., those based on Hidden Markov Methods (HMMs)). For example, a user is allowed to input strokes of one or more characters, phrases, and sentences in any order and still obtain the same recognition results. Thus, out-of-order multi-character input and out-of-order modifications (e.g., additions or rewriting) of previously entered characters are now possible.

Moreover, the universal recognizer is used for real-time handwriting recognition, where temporal information for each stroke is available and optionally used to disambiguate or segment the handwriting input before character recognition is performed by the universal recognizer. The real-time, stroke-order-independent recognition described herein differs from, and may provide better performance than, conventional offline recognition methods (e.g., optical character recognition (OCR)). Furthermore, the universal recognizer described herein can reduce the overall complexity of the recognition system by handling high variability in individual writing habits (e.g., variability in speed, tempo, stroke order, stroke direction, stroke continuity, etc.) without explicitly embedding distinguishing features of different variations in the recognition system (e.g., variations in speed, tempo, stroke order, stroke direction, stroke continuity, etc.).

As described herein, in some embodiments, temporally-derived stroke distribution information is optionally reintroduced into the universal recognizer to disambiguate similar-looking recognition outputs for the same input image and to improve recognition accuracy. Since temporally-derived features and spatially-derived features are acquired through separate training processes and are only combined in the handwriting recognition model after the separate training is completed, reintroduction of temporally-derived stroke distribution information does not compromise the stroke-order and stroke-direction independence of the universal recognizer. Furthermore, the temporally-derived stroke distribution information is carefully designed to capture distinguishing temporal characteristics of similar-looking characters without relying on explicit knowledge of differences in stroke orders of similar-looking characters.

User interfaces for providing handwriting input functionality are also described herein.

In some embodiments, a method of providing multi-script handwriting recognition comprises the steps of training a multi-script handwriting recognition model based on spatially-guided features of a multi-script training corpus, the multi-script training corpus including individual handwriting samples corresponding to characters in at least three non-overlapping scripts; and providing real-time handwriting recognition of a user's handwriting input using the multi-script handwriting recognition model trained on the spatially-guided features of the multi-script training corpus.

In some embodiments, a method of providing multi-script handwriting recognition comprises: receiving a multi-script handwriting recognition model, wherein the multi-script recognition model has been trained on spatially-derived features of a multi-script training corpus, the multi-script training corpus including individual handwriting samples corresponding to characters of at least three non-overlapping scripts; receiving handwriting input from a user, the handwriting input including one or more handwritten strokes provided on a touch-sensitive surface coupled to the user device; and in response to receiving the handwriting input, providing to the user in real time one or more handwriting recognition results based on the multi-script handwriting recognition model trained on spatially-derived features of the multi-script training corpus.

In some embodiments, a method of providing real-time handwriting recognition includes the steps of: receiving a plurality of handwritten strokes from a user, the plurality of handwritten strokes corresponding to handwritten characters; generating an input image based on the plurality of handwritten strokes; providing the input image to a handwriting recognition model to perform real-time recognition of the handwritten character, the handwriting recognition model providing stroke order-independent handwriting recognition; and, in real-time upon receiving the plurality of handwritten strokes, displaying an identical first output character regardless of the individual order in which the plurality of handwritten strokes were received from the user.

In some embodiments, the method further comprises the steps of: receiving a second plurality of handwritten strokes from a user, the second plurality of handwritten strokes corresponding to a second handwritten character; generating a second input image based on the second plurality of handwritten strokes; providing the second input image to a handwriting recognition model to perform real-time recognition of the second handwritten character; and displaying, in real-time upon receiving the second plurality of handwritten strokes, a second output character corresponding to the second plurality of handwritten strokes, wherein the first output character and the second output character are displayed simultaneously in a spatial sequence regardless of the individual order in which the first plurality of handwriting inputs and the second plurality of handwriting inputs were provided by the user.

In some embodiments, the second plurality of handwritten strokes spatially follow the first plurality of handwritten strokes along a default writing direction of a handwriting input interface of the user device, and the second output character follows the first output character in a spatial sequence along the default writing direction, the method comprising: receiving a third handwritten stroke from the user to revise the handwritten character, the third handwritten stroke being received temporally after the first plurality of handwritten strokes and the second plurality of handwritten strokes; in response to receiving the third handwritten stroke, assigning the handwritten stroke to a same recognition unit as the first plurality of handwritten strokes based on a relative proximity of the third handwritten stroke to the first plurality of handwritten strokes; generating a revised input image based on the first plurality of handwritten strokes and the third handwritten stroke; providing the revised input image to a handwriting recognition model to perform real-time recognition of the revised handwritten character; and in response to receiving a third handwriting input, further comprising the step of displaying a third output character corresponding to the revised input image, the third output character replacing the first output character and displayed simultaneously with the second output character in a spatial sequence along the default writing direction.

In some embodiments, the method further comprises the steps of: receiving a deletion input from a user while a third output character and a second output character are simultaneously displayed as recognition results in a candidate display area of a handwriting input interface; and, in response to the deletion input, deleting the second output character from the recognition result while maintaining the third output character in the recognition result.

In some embodiments, the step of rendering in real time a first plurality of handwritten strokes, a second plurality of handwritten strokes, and a third handwritten stroke in a handwriting input area of a handwriting input interface as each of the handwriting strokes is provided by a user; and in response to receiving a delete input, the step of deleting the individual rendering of the second plurality of handwritten strokes from the handwriting input area while maintaining the individual renderings of the first plurality of handwritten strokes and the third handwritten stroke in the handwriting input area.

In some embodiments, a method for providing real-time handwriting recognition comprises: receiving a handwriting input from a user, the handwriting input comprising one or more handwriting strokes provided in a handwriting input area of a handwriting input interface; identifying a plurality of output characters for the handwriting input based on a handwriting recognition model; classifying the plurality of output characters into two or more categories based on predetermined classification criteria; displaying, in an initial view of a candidate display area of the handwriting input interface, individual output characters within a first category of the two or more categories, the initial view of the candidate display area simultaneously having an affordance for invoking an extended view of the candidate display area; receiving a user input selecting an affordance for invoking the extended view; and, in response to the user input, displaying, in the extended view of the candidate display area, individual output characters within the first category of the two or more categories and individual output characters within at least a second category that were not previously displayed in the initial view of the candidate display area.

In some embodiments, a method of providing real-time handwriting recognition includes: receiving a handwriting input from a user, wherein the handwriting input comprises a plurality of handwritten strokes provided in a handwriting input area of a handwriting input interface; recognizing a plurality of output characters from the handwriting input based on a handwriting recognition model, wherein the output characters comprise at least a first emoji character and at least a first character from a script of a natural human language; and displaying a recognition result comprising the first emoji character and the first character from the script of the natural human language in a candidate display area of the handwriting input interface.

In some embodiments, a method of providing handwriting recognition comprises: receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided on a touch-sensitive surface coupled to a device; rendering the plurality of handwritten strokes in real time in a handwriting input area of a handwriting input interface; receiving one of a pinch gesture input and an expand gesture input for the plurality of handwritten strokes; upon receiving the pinch gesture input, generating a first recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as a single recognition unit; upon receiving the expand gesture input, generating a second recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as two separate recognition units separated by the expand gesture input; and upon generating separate recognition results of the first recognition result and the second recognition result, displaying the generated recognition results in a candidate display area of the handwriting input interface.

In some embodiments, a method of providing handwriting recognition includes: receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided in a handwriting input area of a handwriting input interface; identifying a plurality of recognition units from the plurality of handwritten strokes, each recognition unit comprising a respective subset of the plurality of handwritten strokes; generating a multi-character recognition result comprising individual characters recognized from the plurality of recognition units; displaying the multi-character recognition result in a candidate display area of the handwriting input interface; receiving a delete input from the user while the multi-character recognition result is displayed in the candidate display area; and in response to receiving the delete input, removing an end character from the multi-character recognition result displayed in the candidate display area.

In some embodiments, a method of providing real-time handwriting recognition includes: determining an orientation of a device; providing a handwriting input interface on the device in a horizontal input mode according to the device in a first orientation, wherein an individual line of handwriting input entered in the horizontal input mode is segmented into one or more individual recognition units along a horizontal writing direction; and providing a handwriting input interface on the device in a vertical input mode according to the device in a second orientation, wherein an individual line of handwriting input entered in the vertical input mode is segmented into one or more individual recognition units along a vertical writing direction.

In some embodiments, a method of providing real-time handwriting recognition comprises: receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided on a touch-sensitive surface coupled to a device; rendering the plurality of handwritten strokes in a handwriting input area of a handwriting input interface; segmenting the plurality of handwritten strokes into two or more recognition units, each recognition unit comprising a distinct subset of the plurality of handwritten strokes; receiving an edit request from the user; in response to the edit request, visually distinguishing the two or more recognition units in the handwriting input area; and providing means for individually deleting each of the two or more recognition units from the handwriting input area.

In some embodiments, a method for providing real-time handwriting recognition comprises: receiving a first handwriting input from a user, wherein the first handwriting input comprises a plurality of handwriting strokes, the plurality of handwriting strokes forming a plurality of recognition units distributed along respective writing directions associated with a handwriting input area of a handwriting input interface; rendering each of the plurality of handwriting strokes in the handwriting input area as the handwriting strokes are provided by the user; initiating an individual fading process for each of the plurality of recognition units after the recognition units are fully rendered, wherein during the individual fading process, rendering of the recognition units in the first handwriting input becomes increasingly more faded; receiving a second handwriting input from the user over an area of the handwriting input area occupied by faded recognition units of the plurality of recognition units; and in response to receiving the second handwriting input: rendering the second handwriting input in the handwriting input area; and clearing all of the faded recognition units from the handwriting input area.

In some embodiments, a method for providing handwriting recognition comprises the steps of separately training a set of spatially-derived features and a set of temporally-derived features of a handwriting recognition model, wherein the set of spatially-derived features is trained on a corpus of training images, each of the images being an image of a handwriting sample for an individual character of an output character set, and the set of temporally-derived features is trained on a corpus of stroke-distribution profiles, each stroke-distribution profile numerically characterizing the spatial distribution of a plurality of strokes in the handwriting sample for an individual character of the output character set; combining the set of spatially-derived features and the set of temporally-derived features in the handwriting recognition model; and providing real-time handwriting recognition of a user's handwriting input using the handwriting recognition model.

Details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will be apparent from the specification, drawings, and claims.

FIG. 1 is a block diagram illustrating a portable multifunction device having a touch-sensitive display according to some embodiments.
FIG. 2 illustrates a portable multifunction device having a touch-sensitive display according to some embodiments.
FIG. 3 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface according to some embodiments.
FIG. 4 illustrates an exemplary user interface for a multifunction device having a touch-sensitive surface that is separate from a display according to some embodiments.
FIG. 5 is a block diagram of an operating environment of a handwriting input system according to some embodiments.
FIG. 6 is a block diagram of a multi-script handwriting recognition model according to some embodiments.
FIG. 7 is a flowchart of an exemplary process for training a multi-script handwriting recognition model according to some embodiments.
FIGS. 8A and 8B illustrate exemplary user interfaces illustrating real-time multi-script handwriting recognition and input on a portable multifunction device according to some embodiments.
FIGS. 9A and 9B are flowcharts of exemplary processes for providing real-time multi-script handwriting recognition and input on a portable multi-function device.
FIGS. 10A through 10C are flowcharts of exemplary processes for providing real-time stroke-order independent handwriting recognition and input on a portable multifunction device according to some embodiments.
FIGS. 11A through 11K illustrate exemplary user interfaces for selectively displaying recognition results of one category in a normal view of a candidate display area and selectively displaying recognition results of other categories in an extended view of the candidate display area, according to some embodiments.
FIGS. 12A and 12B are flowcharts of an exemplary process for selectively displaying recognition results of one category in a normal field of view of a candidate display area and selectively displaying recognition results of other categories in an extended field of view of the candidate display area, according to some embodiments.
FIGS. 13A through 13E illustrate exemplary user interfaces for entering emoji characters via handwriting input according to some embodiments.
FIG. 14 is a flowchart of an exemplary process for entering emoji characters via handwriting input according to some embodiments.
FIGS. 15A through 15K illustrate exemplary user interfaces for notifying a handwriting input module of how to divide the currently accumulated handwriting input into one or more recognition units using a pinch or expand gesture, according to some embodiments.
FIGS. 16A and 16B are flowcharts of an exemplary process for notifying a handwriting input module of how to divide the currently accumulated handwriting input into one or more recognition units using a pinch or expand gesture, according to some embodiments.
FIGS. 17A through 17H illustrate exemplary user interfaces for providing character-by-character deletion of a user's handwriting input according to some embodiments.
FIGS. 18A and 18B are flowcharts of an exemplary process for providing character-by-character deletion of a user's handwriting input according to some embodiments.
FIGS. 19A through 19F illustrate exemplary user interfaces for switching between vertical writing mode and horizontal writing mode according to some embodiments.
FIGS. 20A through 20C are flowcharts of exemplary processes for switching between vertical writing mode and horizontal writing mode according to some embodiments.
FIGS. 21A through 21H illustrate user interfaces for providing means for displaying and optionally deleting individual recognition units identified in a user's handwriting input, according to some embodiments.
FIGS. 22A and 22B are flowcharts of an exemplary process for providing means for displaying and optionally deleting individual recognition units identified in a user's handwriting input, according to some embodiments.
FIGS. 23A through 23L illustrate exemplary user interfaces for using a new handwriting input provided over an existing handwriting input in a handwriting input area as an implicit confirmation input for entering a recognition result displayed for the existing handwriting input, according to some embodiments.
FIGS. 24A and 24B are flowcharts of an exemplary process for using a new handwriting input provided over an existing handwriting input in a handwriting input area as an implicit confirmation input for entering a recognition result displayed for the existing handwriting input, according to some embodiments.
FIGS. 25A and 25B are flowcharts of an exemplary process for integrating time-derived stroke distribution information into a handwriting recognition model based on space-derived features without compromising stroke order and stroke direction independence of the handwriting recognition model, according to some embodiments.
FIG. 26 is a block diagram illustrating separate training and subsequent integration of spatially-derived features and temporally-derived features of an exemplary handwriting recognition system according to some embodiments.
Figure 27 is a block diagram illustrating an exemplary method for producing a stroke distribution profile of a character.
Identical drawing symbols throughout the drawings represent corresponding parts.

Many electronic devices have graphical user interfaces with soft keyboards for entering text. On some electronic devices, a user may also install or enable a handwriting input interface that allows the user to enter text by writing on a touch-sensitive surface or touch-sensitive display screen coupled to the device. Conventional handwriting recognition input methods and user interfaces have several problems and drawbacks. For example,

Typically, conventional handwriting input capabilities are enabled on a language-by-language or script-by-script basis. Each additional input language requires installation of a separate handwriting recognition model that occupies separate storage space and memory. Combining handwriting recognition models for different languages rarely provides synergy effects, and mixed-language or mixed-script handwriting recognition has traditionally taken a very long time due to complex disambiguation processes.

In addition, conventional handwriting recognition systems rely too heavily on language-specific or script-specific features for character recognition. The recognition of mixed-language handwriting input has poor accuracy. Furthermore, the available combinations of recognized languages are very limited. Most systems require the user to manually specify the desired language-specific handwriting recognizer before providing handwriting input in each non-default language or script.

Many existing real-time handwriting recognition models require temporal or sequence information at a stroke-by-stroke level, which leads to inaccurate recognition results when dealing with the high variability in the way characters can be written (e.g., high variability in the shape, length, tempo, segmentation, order and direction of strokes due to fonts and individual habits). Some systems also require users to adhere to strict spatial and temporal criteria when providing handwriting input (e.g., due to inherent assumptions about the size, sequence and time frame of each character input). Deviations from these criteria have led to inaccurate recognition results that are difficult to correct.

Currently, most real-time handwriting input interfaces allow users to input only a few characters at a time. Long phrases or sentences are inputted into short segments and entered separately. This formal input not only imposes a cognitive burden on the user to maintain the flow of the composition, but also makes it difficult for the user to edit or revise the previously entered characters or phrases.

The embodiments described below address these and related issues.

Below, FIGS. 1 through 4 provide descriptions of exemplary devices. FIGS. 5, 6, 26, and 27 illustrate exemplary handwriting recognition and input systems. FIGS. 8A and 8B, 11A through 11K, 13A through 13E, 15A through 15K, 17A through 17H, 19A through 19F, 21A through 21H, and 23A through 12L illustrate exemplary user interfaces for handwriting recognition and input. FIGS. 7, 9a and 9b, 10a-10c, 12a and 12b, 14, 16a and 16b, 18a and 18b, 20a-20c, 22a and 22b, 24a and 24b, and 25 are flowcharts illustrating methods for enabling handwriting recognition and input on user devices, including the steps of training handwriting recognition models, providing real-time handwriting recognition results, providing means for inputting and correcting handwriting input, and providing means for inputting the recognition results as text input. The user interfaces in FIGS. 8A and 8B, 11A through 11K, 13A through 13E, 15A through 15K, 17A through 17H, 19A through 19F, 21A through 21H, and 23A through 12L are used to illustrate the processes in FIGS. 7, 9A and 9B, 10A through 10C, 12A and 12B, 14, 16A and 16B, 18A and 18B, 20A through 20C, 22A and 22B, 24A and 24B, and 25.

Example devices

Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

It will also be appreciated that while the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the present invention, a first contact could be referred to as a second contact, and similarly, a second contact could be referred to as a first contact. Although both the first contact and the second contact are contacts, they are not the same contacts.

The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the description of the invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and/or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "include," "including," "comprise," and/or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As used herein, the term "if" can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if determined" or "if [a stated state or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [a stated state or event]" or "in response to detecting [a stated state or event]," depending on the context.

Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communications device, such as a mobile telephone, that also includes other functionality, such as PDA and/or music player functionality. Exemplary embodiments of portable multifunction devices include, without limitation, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, Calif. Other portable electronic devices, such as laptops or tablet computers having touch-sensitive surfaces (e.g., touch screen displays and/or touch pads), may also be used. It should also be understood that in some embodiments, the device is a desktop computer having a touch-sensitive surface (e.g., a touch screen display and/or touch pad), rather than a portable communications device.

In the discussion below, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device may include one or more other physical user-interface devices, such as a physical keyboard, mouse, and/or joystick.

A device typically supports a variety of applications, such as one or more of a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness assistance application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.

The various applications that may be executed on the device may utilize at least one common physical user-interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as corresponding information displayed on the device, may be adjusted and/or varied from one application to the next and/or within an individual application. In this manner, the common physical architecture of the device (such as the touch-sensitive surface) may support the various applications with user interfaces that are intuitive and clear to the user.

Now, we turn to embodiments of portable devices having touch-sensitive displays. FIG. 1 is a block diagram illustrating a portable multifunction device (100) having touch-sensitive displays (112) according to some embodiments. The touch-sensitive display (112) is sometimes conveniently referred to as a “touch screen” and may also be known or referred to as a touch-sensitive display system. The device (100) may include memory (102) (which may include one or more computer-readable storage media), a memory controller (122), one or more processing units (CPUs) (120), a peripheral interface (118), RF circuitry (108), audio circuitry (110), a speaker (111), a microphone (113), an input/output (I/O) subsystem (106), other input or control devices (116), and an external port (124). The device (100) may include one or more light sensors (164). These components may communicate via one or more communication buses or signal lines (103).

It should be understood that the device (100) is only one example of a portable multi-function device, and that the device (100) may have more or fewer components than those illustrated, may combine two or more components, or may have different configurations or arrangements of the components. The various components illustrated in FIG. 1 may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and/or application specific integrated circuits.

The memory (102) may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory (102) by other components of the device (100), such as the CPU (120) and the peripheral interface (118), may be controlled by the memory controller (122).

A peripheral interface (118) may be used to couple input and output peripherals of the device to the CPU (120) and memory (102). One or more processors (120) drive or execute various software programs and/or sets of instructions stored within the memory (102) to perform various functions for the device (100) and process data.

In some embodiments, the peripheral interface (118), CPU (120) and memory controller (122) may be implemented on a single chip, such as chip (104). In some other embodiments, they may be implemented on separate chips.

The RF (radio frequency) circuit (108) receives and transmits RF signals, also called electromagnetic signals. The RF circuit (108) converts electrical signals to and from electromagnetic signals, and communicates with communication networks and other communication devices via the electromagnetic signals.

The audio circuit (110), the speaker (111), and the microphone (113) provide an audio interface between a user and the device (100). The audio circuit (110) receives audio data from the peripheral interface (118), converts the audio data into an electrical signal, and transmits the electrical signal to the speaker (111). The speaker (111) converts the electrical signal into audible sound waves. The audio circuit (110) also receives the electrical signal converted from the sound waves by the microphone (113). The audio circuit (110) converts the electrical signal into audio data, and transmits the audio data to the peripheral interface (118) for processing. The audio data may be retrieved from and/or transmitted to the memory (102) and/or the RF circuit (108) by the peripheral interface (118). In some embodiments, the audio circuit (110) also includes a headset jack (e.g., 212 of FIG. 2 ).

The I/O subsystem (106) couples input/output peripherals on the device (100), such as a touch screen (112) and other input control devices (116), to the peripheral interface (118). The I/O subsystem (106) may include a display controller (156), and one or more input controllers (160) for other input or control devices. The one or more input controllers (160) receive/transmit electrical signals from/to other input or control devices (116). The other input control devices (116) may include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller(s) (160) may be coupled to any (or none) of a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., 208 of FIG. 2) may include up/down buttons for controlling the volume of the speaker (111) and/or microphone (113). One or more buttons may include a push button (e.g., 206 of FIG. 2).

The touch-sensitive display (112) provides an input interface and an output interface between the device and a user. The display controller (156) receives and/or transmits electrical signals from/to the touch screen (112). The touch screen (112) displays visual output to the user. The visual output may include graphics, text, icons, videos, and any combination thereof (collectively referred to as “graphics”). In some embodiments, some or all of the visual output may correspond to user-interface objects.

The touch screen (112) has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic and/or tactile contact. The touch screen (112) and the display controller (156) (along with any associated modules and/or sets of instructions in the memory (102)) detect contact (and any movement or interruption of the contact) on the touch screen (112) and translate the detected contact into interaction with user-interface objects displayed on the touch screen (112) (e.g., one or more soft keys, icons, web pages, or images). In an exemplary embodiment, the point of contact between the touch screen (112) and the user corresponds to the user's finger.

The touch screen (112) may utilize LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, although other display technologies may be utilized in other embodiments. The touch screen (112) and display controller (156) may detect contact and any movement or cessation thereof using any of a number of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements, to determine one or more points of contact with the touch screen (112). In an exemplary embodiment, projected mutual capacitance sensing technology is utilized, such as found in the iPhone®, iPod touch®, and iPad® from Apple Inc. of Cupertino, Calif.

The touch screen (112) may have a video resolution greater than 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. A user may contact the touch screen (112) using any suitable object or appendage, such as a stylus, a finger, etc. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures, which may be less precise than stylus-based input due to the larger contact area of a finger on the touch screen. In some embodiments, the device converts rough finger-based inputs into precise pointer/cursor positions or commands for the user to perform desired actions. Handwriting input may be provided on the touch screen (112) via the positioning and movement of finger-based or stylus-based contacts. In some embodiments, the touch screen (112) renders finger-based or stylus-based input as immediate visual feedback of the current handwriting input, providing the visual effect of actual writing on a writing surface (e.g., a sheet of paper) using a writing instrument (e.g., a pen).

In some embodiments, in addition to the touch screen, the device (100) may include a touchpad (not shown) for activating or deactivating certain functions. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike the touch screen, does not display visual output. The touchpad may be a touch-sensitive surface separate from the touch screen (112) or an extension of a touch-sensitive surface formed by the touch screen.

The device (100) also includes a power system (162) for supplying power to the various components. The power system (162) may include a power management system, one or more power sources (e.g., a battery, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with generating, managing, and distributing power within portable devices.

The device (100) may also include one or more light sensors (164). FIG. 1 illustrates a light sensor coupled to a light sensor controller (158) within the I/O subsystem (106). The light sensor (164) may include charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. The light sensor (164) receives light from the surrounding environment projected through one or more lenses and converts the light into data representing an image. The light sensor (164), in conjunction with an imaging module (143) (also referred to as a camera module), may capture still images or video.

The device (100) may also include one or more proximity sensors (166). FIG. 1 illustrates a proximity sensor (166) coupled to a peripheral interface (118). Alternatively, the proximity sensor (166) may be coupled to an input controller (160) within the I/O subsystem (106). In some embodiments, the proximity sensor turns off and disables the touch screen (112) when the multifunction device is positioned near a user's ear (e.g., when the user is on a phone call).

The device (100) may also include one or more accelerometers (168). FIG. 1 illustrates an accelerometer (168) coupled to a peripheral interface (118). Alternatively, the accelerometer (168) may be coupled to an input controller (160) within the I/O subsystem (106). In some embodiments, information is displayed on the touch screen display in either a portrait view or a landscape view based on analysis of data received from the one or more accelerometers. In addition to the accelerometer(s) (168), the device (100) optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information regarding the position and orientation (e.g., portrait or landscape) of the device (100).

In some embodiments, the software components stored in the memory (102) include an operating system (126), a communications module (or set of instructions) (128), a contact/motion module (or set of instructions) (130), a graphics module (or set of instructions) (132), a text input module (or set of instructions) (134), a global positioning system (GPS) module (or set of instructions) (135), and applications (or set of instructions) (136). Additionally, in some embodiments, the memory (102) stores a handwriting input module (157), as illustrated in FIGS. 1 and 3 . The handwriting input module (157) includes a handwriting recognition model and provides handwriting recognition and input capabilities to a user of the device (100) (or device (300)). Further details of the handwriting input module (157) are provided in connection with FIGS. 5-27 and their accompanying descriptions.

An operating system (126) (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and/or drivers to control and manage general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitate communication between various hardware and software components.

The communications module (128) facilitates communication with other devices via one or more external ports (124), and also includes various software components for processing data received by the RF circuitry (108) and/or the external port (124). The external port (124) (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) is configured to couple directly to other devices or indirectly via a network (e.g., the Internet, wireless LAN, etc.).

The contact/motion module (130) can detect contact with the touch screen (112) (along with the display controller (156)) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact/motion module (130) includes various software components to perform various operations associated with detecting contact, such as determining whether contact has occurred (e.g., detecting a finger-down event), determining whether there has been movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger-dragging events), and determining whether contact has ceased (e.g., detecting a finger-up event or a break in contact). The contact/motion module (130) receives contact data from the touch-sensitive surface. Determining the movement of a contact point indicated by a series of contact data may include determining the speed (magnitude), velocity (magnitude and direction), and/or acceleration (change in magnitude and/or direction) of the contact point. These actions may be applied to single contacts (e.g., single finger contacts) or to multiple simultaneous contacts (e.g., “multitouch”/multiple finger contacts). In some embodiments, the contact/motion module (130) and the display controller (156) detect contact on the touchpad.

The contact/motion module (130) can detect a gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns. Accordingly, a gesture can be detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger-down event, followed by detecting a finger-up (lifting) event at the same location (or substantially the same location) as the finger-down event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger-down event, followed by detecting one or more finger-drag events, followed by detecting a finger-up (lifting) event.

The contact/motion module (130) is optionally utilized by the handwriting input module (157) to register input of handwriting strokes within a handwriting input area of a handwriting input interface displayed on a touch-sensitive display screen (112) (or within an area of a touch pad (355) corresponding to a handwriting input area displayed on the display (340) of FIG. 3). In some embodiments, an initial finger-down event, a final finger-up event, and positions, motion paths, and intensities associated with the contact during any time therebetween are recorded as handwriting strokes. Based on such information, the handwriting strokes can be rendered on the display as feedback for the user input. Additionally, one or more input images can be generated based on the handwriting strokes registered by the contact/motion module (130).

The graphics module (132) includes various known software components for rendering and displaying graphics on the touch screen (112) or other display, including components for changing the intensity of the graphics being displayed. As used herein, the term "graphics" includes any object that can be displayed to a user, including without limitation text, web pages, icons (e.g., user-interface objects including soft keys), digital images, videos, animations, and the like.

In some embodiments, the graphics module (132) stores data representing the graphics to be used. Each graphic may be assigned a corresponding code. The graphics module (132) receives one or more codes specifying the graphics to be displayed from an application or the like, along with coordinate data and other graphic characteristic data, if necessary, and then generates screen image data and outputs it to the display controller (156).

A text input module (134), which may be a component of the graphics module (132), provides soft keyboards for entering text in various applications (e.g., Contacts (137), Email (140), IM (141), Browser (147), and any other application requiring text input). In some embodiments, the handwriting input module (157) is selectively invoked through the user interface of the text input module (134), such as via a keyboard selection affordance. In some embodiments, the same or a similar keyboard selection affordance is also provided in the handwriting input interface to invoke the text input module (134).

The GPS module (135) determines the location of the device and provides this information for use in various applications (e.g., to the phone (138) for use in location-based dialing, to the camera (143) as photo/video metadata, and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map/navigation widgets).

The applications (136) may include the following modules (or sets of instructions) or a subset or superset thereof: a contacts module (137) (sometimes called an address book or contact list); a phone module (138); a video conferencing module (139); an email client module (140); an instant messaging (IM) module (141); a workout assistance module (142); a camera module (143) for still and/or video images; an image management module (144); a browser module (147); a calendar module (148); widget modules (149) that may include one or more of a weather widget (149-1), a stocks widget (149-2), a calculator widget (149-3), an alarm clock widget (149-4), a dictionary widget (149-5), and other widgets acquired by the user, and user-created widgets (149-6); a widget creator module (150) for creating user-created widgets (149-6); A search module (151); a video and music player module (152) which may consist of a video player module and a music player module; a memo module (153); a map module (154); and/or an online video module (155).

Examples of other applications (136) that may be stored within the memory (102) include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, speech recognition and speech transcription.

In conjunction with the touch screen (112), the display controller (156), the contact module (130), the graphics module (132), the handwriting input module (157), and the text input module (134), the contact module (137) may also be used to manage an address book or list of contacts (e.g., stored in the application internal state (192) of the contact module (137) within the memory (102) or the memory (370), including: adding name(s) to the address book; deleting name(s) from the address book; associating phone number(s), email address(es), physical address(es), or other information with a name; associating an image with a name; categorizing and sorting names; initiating and/or facilitating communication by phone (138), video conferencing (139), email (140), or IM (141) by providing phone numbers or email addresses; etc.

In conjunction with the RF circuitry (108), audio circuitry (110), speaker (111), microphone (113), touch screen (112), display controller (156), contact module (130), graphics module (132), handwriting input module (157), and text input module (134), the phone module (138) can be used to input a sequence of characters corresponding to a phone number, access one or more phone numbers in an address book (137), modify an input phone number, dial each phone number, engage in a conversation, and disconnect or hang up when the conversation is complete. As described above, the wireless communications can utilize any of a plurality of communication standards, protocols, and technologies.

Along with the RF circuitry (108), the audio circuitry (110), the speaker (111), the microphone (113), the touchscreen (112), the display controller (156), the light sensor (164), the light sensor controller (158), the contact module (130), the graphics module (132), the handwriting input module (157), the text input module (134), the contact list (137), and the phone module (138), the video conferencing module (139) includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

In conjunction with the RF circuitry (108), touch screen (112), display controller (156), contact module (130), graphics module (132), handwriting input module (157), and text input module (134), the email client module (140) includes executable instructions to generate, send, receive, and manage emails in response to user commands. In conjunction with the image management module (144), the email client module (140) greatly facilitates generating and sending emails having still or video images captured by the camera module (143).

In conjunction with the RF circuitry (108), the touch screen (112), the display controller (156), the contact module (130), the graphics module (132), the handwriting input module (157), and the text input module (134), the instant messaging module (141) includes executable instructions to input a sequence of characters corresponding to an instant message, to modify previously input characters, to send individual instant messages (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for phone-based instant messages, or XMPP, SIMPLE, or IMPS for Internet-based instant messages), to receive instant messages, and to view received instant messages. In some embodiments, the sent and/or received instant messages may include graphics, photographs, audio files, video files, and/or other attachments such as supported by MMS and/or Enhanced Messaging Service (EMS). As used herein, “instant messaging” refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE or IMPS).

In conjunction with the RF circuitry (108), the touch screen (112), the display controller (156), the contact module (130), the graphics module (132), the handwriting input module (157), the text input module (134), the GPS module (135), the map module (154), and the music player module (146), the workout assistance module (142) includes executable instructions to generate workouts (e.g., having time, distance, and/or calorie burn goals); communicate with workout sensors (sports devices); receive workout sensor data; calibrate sensors used to monitor workouts; select and play music for workouts; and display, store, and transmit workout data.

The camera module (143), together with the touch screen (112), the display controller (156), the light sensor(s) (164), the light sensor controller (158), the contact module (130), the graphics module (132) and the image management module (144), includes executable instructions to capture still images or video (including a video stream) and store them in the memory (102), modify characteristics of the still image or video, or delete the still image or video from the memory (102).

In conjunction with the touch screen (112), display controller (156), contact module (130), graphics module (132), handwriting input module (157), text input module (134), and camera module (143), the image management module (144) includes executable instructions to arrange, modify (e.g., edit), or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and store still and/or video images.

In conjunction with the RF circuitry (108), the touch screen (112), the display system controller (156), the contact module (130), the graphics module (132), the handwriting input module (157), and the text input module (134), the browser module (147) includes executable instructions for browsing the Internet in accordance with user commands, including retrieving, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

In conjunction with the RF circuitry (108), the touch screen (112), the display system controller (156), the contact module (130), the graphics module (132), the handwriting input module (157), the text input module (134), the email client module (140), and the browser module (147), the calendar module (148) includes executable instructions that cause calendars and data associated with the calendars (e.g., calendar entries, to-do lists, etc.) to be created, displayed, modified, and stored in response to user commands.

Along with the RF circuitry (108), the touch screen (112), the display system controller (156), the contact module (130), the graphics module (132), the handwriting input module (157), the text input module (134), and the browser module (147), the widget modules (149) are mini-applications that can be downloaded and used by a user (e.g., a weather widget (149-1), a stock widget (149-2), a calculator widget (149-3), an alarm clock widget (149-4), and a dictionary widget (149-5)), or created by a user (e.g., a user-created widget (149-6)). In some embodiments, the widgets include Hypertext Markup Language (HTML) files, Cascading Style Sheets (CSS) files, and JavaScript files. In some embodiments, a widget includes an Extensible Markup Language (XML) file and a JavaScript file (e.g., Yahoo! widgets).

In conjunction with the RF circuit (108), touch screen (112), display system controller (156), contact module (130), graphics module (132), handwriting input module (157), text input module (134), and browser module (147), the widget creator module (150) can be used by users to create widgets (e.g., change user-specific portions of a web page into widgets).

In conjunction with the touch screen (112), the display system controller (156), the contact module (130), the graphics module (132), the handwriting input module (157), and the text input module (134), the search module (151) includes executable instructions that cause the memory (102) to search for text, music, sound, images, video, and/or other files that match one or more search criteria (e.g., one or more user-specific search terms) in accordance with user commands.

In conjunction with the touch screen (112), the display system controller (156), the contact module (130), the graphics module (132), the audio circuitry (110), the speaker (111), the RF circuitry (108), and the browser module (147), the video and music player module (152) includes executable instructions that enable a user to download and play recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions that cause videos to be displayed, played, or otherwise played (e.g., on the touch screen (112) or on an externally connected display via the external port (124). In some embodiments, the device (100) may include the functionality of an MP3 player, such as an iPod (a trademark of Apple Inc.).

In addition to the touch screen (112), display controller (156), contact module (130), graphics module (132), handwriting input module (157) and text input module (134), the memo module (153) includes executable commands to create and manage memos, to-do lists, etc. according to user commands.

In conjunction with the RF circuitry (108), touch screen (112), display system controller (156), contact module (130), graphics module (132), handwriting input module (157), text input module (134), GPS module (135), and browser module (147), the map module (154) can be used to receive, display, modify, and store maps and data associated with the maps (e.g., driving directions; data regarding stores and other points of interest at or near a particular location; and other location-based data) in accordance with user commands.

In conjunction with the touch screen (112), the display system controller (156), the contact module (130), the graphics module (132), the audio circuitry (110), the speaker (111), the RF circuitry (108), the handwriting input module (157), the text input module (134), the email client module (140), and the browser module (147), the online video module (155) includes instructions that enable a user to access, browse, receive (e.g., by streaming and/or downloading), play (e.g., on the touch screen or on an externally connected display via the external port (124), send emails with links to particular online videos, and otherwise manage online videos in one or more file formats, such as H.264. In some embodiments, an instant messaging module (141) is used rather than the email client module (140) to send links to particular online videos.

Each of the modules and applications identified above corresponds to a set of executable instructions for performing one or more of the functions described above and the methods described herein (e.g., the computer-implemented methods and other information processing methods described herein). These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these modules may be combined or otherwise rearranged in various embodiments. In some embodiments, the memory (102) may store a subset of the modules and data structures identified above. Additionally, the memory (102) may store additional modules and data structures not described above.

In some embodiments, the device (100) is a device in which operation of a predefined set of functions on the device is performed exclusively via a touch screen and/or a touchpad. By using the touch screen and/or the touchpad as the primary input control device for operation of the device (100), the number of physical input control devices (such as push buttons, dials, etc.) on the device (100) can be reduced.

FIG. 2 illustrates a portable multifunction device (100) having a touch screen (112) according to some embodiments. The touch screen may display one or more graphics within a user interface (UI) (200). In this embodiment, as well as other embodiments described below, a user may select one or more of the graphics by making a gesture over the graphics, for example, using one or more fingers (202) (not drawn to scale in the drawing) or one or more styli (203) (not drawn to scale in the drawing). In some embodiments, the selection of the one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture may include one or more taps of a finger in contact with the device (100), one or more swipes (left to right, right to left, upward, and/or downward), and/or a roll (right to left, left to right, upward, and/or downward). In some embodiments, inadvertent contact with a graphic may not select that graphic. For example, a swipe gesture that sweeps over an application icon may not select the corresponding application when the corresponding gesture for selection is a tap.

The device (100) may also include one or more physical buttons, such as a “home” or menu button (204). As previously described, the menu button (204) may be used to navigate to any application (136) in the set of applications that may be run on the device (100). Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen (112).

In one embodiment, the device (100) includes a touch screen (112), a menu button (204), a push button (206) to power the device on/off and to lock the device, volume control button(s) (208), a subscriber identity module (SIM) card slot (210), a headset jack (212), and a docking/charging external port (124). The push button (206) may be used to toggle the device on/off by pressing the button and holding it pressed for a predefined time interval; to lock the device by pressing the button and releasing the button before the predefined time interval has elapsed; and/or to unlock the device or initiate an unlocking process. In an alternative embodiment, the device (100) may also accept verbal input via the microphone (113) to activate or deactivate certain features.

FIG. 3 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface according to some embodiments. The device (300) need not be portable. In some embodiments, the device (300) is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, a telephony device, or a control device (e.g., a household or industrial controller). The device (300) typically includes one or more processing units (CPUs) (310), one or more network or other communication interfaces (360), memory (370), and one or more communication buses (320) for interconnecting these components. The communication buses (320) may include circuitry (sometimes referred to as a chipset) for interconnecting the system components and controlling communication therebetween. The device (300) includes an input/output (I/O) interface (330) that typically includes a display (340), which is a touch screen display. The I/O interface (330) may also include a keyboard and/or mouse (or other pointing device) (350) and a touchpad (355). The memory (370) may include high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; and may also include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. The memory (370) may optionally include one or more storage devices remotely located from the CPU(s) (310). In some embodiments, the memory (370) stores programs, modules, and data structures similar to the programs, modules, and data structures stored in the memory (102) of the portable multifunction device (100) (FIG. 1), or a subset thereof. Additionally, the memory (370) may store additional programs, modules, and data structures that are not present in the memory (102) of the portable multifunction device (100). For example, the memory (370) of the device (300) may store a drawing module (380), a presentation module (382), a word processing module (384), a website creation module (386), a disk authoring module (388), and/or a spreadsheet module (390), whereas the memory (102) of the portable multifunction device (100) (FIG. 1) may not store these modules.

Each of the identified elements in FIG. 3 may be stored in one or more of the previously mentioned memory devices. Each of the identified modules corresponds to a set of instructions for performing the function described above. The identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these modules may be combined or otherwise rearranged in various embodiments. In some embodiments, the memory (370) may store a subset of the identified modules and data structures. Additionally, the memory (370) may store additional modules and data structures not described above.

FIG. 4 illustrates an exemplary user interface on a device (e.g., device (300), FIG. 3) having a touch-sensitive surface (451) (e.g., a tablet or touchpad (355), FIG. 3) separate from a display (450) (e.g., a touch screen display (112)). While many of the examples that follow will be given with reference to inputs on the touch screen display (112) (which is a combination of the touch-sensitive surface and the display), in some embodiments, the device detects inputs on a touch-sensitive surface that is separate from the display, as illustrated in FIG. 4. In some embodiments, the touch-sensitive surface (e.g., 451 in FIG. 4) has a principal axis (e.g., 452 in FIG. 4) that corresponds to a principal axis (e.g., 453 in FIG. 4) on the display (e.g., 450). According to these embodiments, the device detects contacts (e.g., 460 and 462 in FIG. 4) with the touch-sensitive surface (451) at locations corresponding to individual locations on the display (e.g., 460 corresponds to 468 and 462 corresponds to 470 in FIG. 4 ). In this manner, user inputs (e.g., contacts (460, 462) and movements thereof) detected by the device on the touch-sensitive surface (e.g., 451 in FIG. 4) are used by the device to manipulate a user interface on the display of the multifunction device (e.g., 450 in FIG. 4) when the touch-sensitive surface is separate from the display. It should be appreciated that similar methods may be utilized for other user interfaces described herein.

Now, we turn to embodiments of handwriting input methods and user interfaces (“UI”) that may be implemented on a multifunction device (e.g., device (100)).

FIG. 5 is a block diagram illustrating an exemplary handwriting input module (157) that interacts with an I/O interface module (500) (e.g., the I/O interface (330) of FIG. 3 or the I/O subsystem (106) of FIG. 1 ) to provide handwriting input capabilities on the device according to some embodiments. As illustrated in FIG. 5 , the handwriting input module (157) includes an input processing module (502), a handwriting recognition module (504), and a result generation module (506). In some embodiments, the input processing module (502) includes a segmentation module (508) and a normalization module (510). In some embodiments, the result generation module (506) includes a radical clustering module (512) and one or more language models (514).

In some embodiments, the input processing module (502) receives handwriting inputs from a user by communicating with an I/O interface module (500) (e.g., the I/O interface (330) of FIG. 3 or the I/O subsystem (106) of FIG. 1). The handwriting is entered via any suitable means, such as the touch-sensitive display system (112) of FIG. 1 and/or the touchpad (355) of FIG. 3. The handwriting inputs include data representing each stroke provided by the user within a predetermined handwriting input area within the handwriting input UI. In some embodiments, the data representing each stroke of the handwriting input includes data such as start and end locations, intensity profiles, and motion paths of sustained contact (e.g., contact between the user's finger or stylus and the touch-sensitive surface of the device) within the handwriting input area. In some embodiments, the I/O interface module (500) transmits sequences of handwriting strokes (516) with associated temporal and spatial information to the input processing module (502) in real time. Concurrently, the I/O interface module also provides real-time rendering (518) of handwritten strokes within the handwriting input area of the handwriting input user interface as visual feedback for the user's input.

In some embodiments, as data representing each handwritten stroke is received by the input processing module (502), temporal and sequence information associated with the plurality of consecutive strokes is also recorded. For example, the data optionally includes a stack showing the shape of the strokes, the size of the strokes, the spatial saturation of the individual strokes with their individual stroke sequence numbers, and the relative spatial positions along the direction of writing of the overall handwriting input. In some embodiments, the input processing module (502) provides instructions back to the I/O interface modules (500) to cause the rendering of the received strokes on a display (518) of the device (e.g., the display (340) of FIG. 3 or the touch-sensitive display (112) of FIG. 1 ). In some embodiments, the rendering of the received strokes is animated to provide a visual effect that mimics the actual progression of writing on a writing surface (e.g., a sheet of paper) using a writing instrument (e.g., a pen). In some embodiments, the user is allowed to optionally specify the pen-tip style, color, texture, etc. of the rendered strokes.

In some embodiments, the input processing module (502) processes the currently accumulated strokes in the handwriting input area and assigns the strokes to one or more recognition units. In some embodiments, each recognition unit corresponds to a character to be recognized by the handwriting recognition model (504). In some embodiments, each recognition unit corresponds to an output character or radical to be recognized by the handwriting recognition model (504). A radical is a repeating element found in many compound sign characters. A compound sign character may include two or more radicals arranged according to a common layout (e.g., left-right layout, top-bottom layout, etc.). In one example, a single Chinese character " " is composed of two radicals, the left radical "口" and the right radical "斤".

In some embodiments, the input processing module (502) relies on the segmentation module to assign or divide the currently accumulated handwritten strokes into one or more recognition units. For example, the handwritten character " "When segmenting strokes for , the segmentation module (508) optionally assigns strokes clustered on the left side of the handwriting input (i.e., for the left radical "口") to one recognition unit, and strokes clustered on the right side of the handwriting input (i.e., for the right radical "斤") to another recognition unit. Alternatively, the segmentation module (508) may also segment all strokes (i.e., for the character " ") can be assigned to a single recognition unit.

In some embodiments, the segmentation module (508) segments the currently accumulated handwriting input (e.g., one or more handwriting strokes) into groups of recognition units in various different ways to generate a segmentation grid (520). For example, assume that a total of nine strokes have been accumulated so far in the handwriting input area. According to a first segmentation chain of the segmentation grid (520), strokes 1, Stroke 2, and Stroke 3 are grouped into a first recognition unit (522), and strokes 4, Stroke 5, and Stroke 6 are grouped into a second recognition unit (526). According to a second segmentation chain of the segmentation grid (520), all of strokes 1 to 9 are grouped into one recognition unit (526).

In some embodiments, each segmentation chain is given a segmentation score that measures the likelihood that a particular segmentation chain is an accurate segmentation of the current handwriting input. In some embodiments, factors optionally used to compute the segmentation score of each segmentation chain include: absolute and/or relative size of the stroke, relative and/or absolute span of the stroke in various directions (e.g., x, y, z directions), variations in saturation level of the stroke and/or average thereof, absolute and/or relative distances to adjacent strokes, absolute and/or relative positions of the strokes, the order or sequence in which the strokes are entered, the duration of each stroke, variations in speed (or tempo) at which each stroke is entered and/or average thereof, intensity profile of each stroke along the length of the stroke, etc. In some embodiments, one or more functions or transformations are optionally applied to one or more of these factors to generate segmentation scores for different segmentation chains in the segmentation grid (520).

In some embodiments, after the segmentation module (508) segments the current handwriting input (516) received from the user, the segmentation module (508) passes the segmentation grid (520) to the normalization module (510). In some embodiments, the normalization module (510) generates an input image (e.g., input images (528)) for each recognition unit specified in the segmentation grid (520) (e.g., recognition units (522, 524, 526)). In some embodiments, the normalization module performs any necessary or desired normalization (e.g., stretching, cropping, down-sampling, or up-sampling) on the input images so that the input images can be provided as input to the handwriting recognition model (504). In some embodiments, each input image (528) includes strokes assigned to one individual recognition unit and corresponds to one character or radical to be recognized by the handwriting recognition module (504).

In some embodiments, the input images generated by the input processing module (502) do not contain any temporal information associated with individual strokes, and only spatial information (e.g., information represented by the density and location of pixels in the input image) is preserved in the input image. A handwriting recognition model trained solely on the spatial information of the training handwriting samples is capable of handwriting recognition based only on spatial information. As a result, the handwriting recognition model is independent of stroke order and stroke direction without exhaustively enumerating all possible permutations of stroke orders and stroke directions for all characters in its vocabulary (i.e., all output classes) during training. In fact, in some embodiments, the handwriting recognition module (502) does not distinguish between pixels belonging to one stroke in the input image and pixels belonging to another stroke.

As will be described in more detail later (e.g., with respect to FIGS. 25a to 27), in some embodiments, some time-derived stroke distribution information is reintroduced into the entirely spatial handwriting recognition model to improve recognition accuracy without compromising the stroke order and stroke direction independence of the recognition model.

In some embodiments, an input image generated by the input processing module (502) for a recognition unit does not overlap with an input image of any other recognition unit in the same segmentation chain. In some embodiments, input images generated for different recognition units may overlap slightly. In some embodiments, slight overlap between input images is allowed for recognizing handwriting input that is written in cursive and/or includes run-on characters (e.g., a single stroke connecting two adjacent characters).

In some embodiments, some normalization is performed prior to segmentation. In some embodiments, the functions of the segmentation module (508) and the normalization module (510) may be performed by the same module or by two or more different modules.

In some embodiments, as the input image (528) of each recognition unit is provided as input to the handwriting recognition model (504), the handwriting recognition model (504) generates output consisting of different possibilities for the recognition unit to be an individual output character from the repertoire or vocabulary of the handwriting recognition model (504) (i.e., the list of all characters and radicals recognizable by the handwriting recognition module (504). As described in more detail below, the handwriting recognition model (504) is trained to recognize a large number of characters from multiple scripts (e.g., at least three non-overlapping scripts encoded by the Unicode standard).

Examples of non-overlapping scripts include artificial scripts such as Latin script, Chinese characters, Arabic characters, Persian, Cyrillic characters, and emoji characters. In some embodiments, the handwriting recognition model (504) generates one or more output characters for each input image (i.e., for each recognition unit) and assigns an individual recognition score to each output character based on a confidence level associated with the character recognition.

In some embodiments, the handwriting recognition model (504) generates a candidate grid (530) according to the segmentation grid (520), wherein each arc in a segmentation chain (e.g., corresponding to an individual recognition unit (522, 524, 526)) within the segmentation grid (520) is inflated into one or more candidate arcs (e.g., arcs (532, 534, 536, 538, 540) each corresponding to an individual output character) within the candidate grid (530). Each candidate chain within the candidate grid (530) is scored according to the individual segmentation scores of the segmentation chain underlying the candidate chain, and the recognition scores associated with the output characters in the character chain.

In some embodiments, after the handwriting recognition model (504) generates output characters from the input images (528) of the recognition units, the candidate grid (530) is passed to the result generation module (506) to generate one or more recognition results for the currently accumulated handwriting input (516).

In some embodiments, the result generation module (506) utilizes the radical clustering module (512) to combine one or more radicals in the candidate chain into a compound character. In some embodiments, the result generation module (506) uses one or more language models (514) to determine whether a chain of characters in the candidate lattice (530) is a likely sequence in a particular language represented by the language models. In some embodiments, the result generation module (506) generates a revised candidate lattice (542) by combining two or more arcs in the candidate lattice (530) or by removing certain arcs.

In some embodiments, the result generation module (506) generates an integrated recognition score for each character sequence (e.g., character sequences (544, 546)) that remains in the revised candidate grid (542) based on the recognition scores of the output characters in the character sequence as modified (e.g., augmented or reduced) by the ancillary clustering module (512) and the language models (514). In some embodiments, the result generation module (506) ranks the different character sequences that remain in the revised candidate grid (542) based on their integrated recognition scores.

In some embodiments, the result generation module (506) transmits the top-ranked character sequences as ranked recognition results (548) to the I/O interface module (500) for display to the user. In some embodiments, the I/O interface module (500) outputs the received recognition results (548) (e.g., " " and " ") is displayed in the candidate display area of the handwriting input interface. In some embodiments, the I/O interface module displays a plurality of recognition results (e.g., " " and " ") and allows the user to select a recognition result and input it as text input to the relevant application. In some embodiments, the I/O interface module responds to user confirmation of a recognition result or other inputs by displaying the top recognition result (e.g., " ") is automatically entered. Effective automatic entry of top results can improve the efficiency of the input interface and provide a better user experience.

In some embodiments, the result generation module (506) uses other factors to modify the integrated recognition scores of the candidate chains. For example, in some embodiments, the result generation module (506) optionally maintains a log of the most frequently used characters for a particular user or a number of users. When particular candidate characters or character sequences are found among the list of most frequently used characters or character sequences, the result generation module (506) optionally increases the integrated recognition scores of the particular candidate characters or character sequences.

In some embodiments, the handwriting input module (157) provides real-time updates to the recognition results displayed to the user. For example, in some embodiments, for each additional stroke input by the user, the input processing module (502) optionally re-segments the currently accumulated handwriting input and revises the input images and segmentation grid provided to the handwriting recognition model (504). In turn, the handwriting recognition model (504) optionally revises the candidate grid provided to the result generation module (506). As a result, the result generation module (506) optionally updates the recognition results presented to the user. As used herein, real-time handwriting recognition refers to handwriting recognition in which handwriting recognition results are presented to the user immediately or within a short period of time (e.g., within tens of milliseconds to several seconds). Real-time handwriting recognition differs from offline recognition (e.g., as in offline optical character recognition (OCR) applications) in that recognition is initiated immediately and performed substantially simultaneously with the receipt of handwriting input, rather than all at once after the current user session from a recorded image that is stored for later retrieval. Furthermore, offline character recognition is performed without any temporal information about individual strokes and stroke sequences, so that segmentation is performed without the benefit of such information. Further disambiguation between candidate characters that appear similar is also performed without the benefit of such temporal information.

In some embodiments, the handwriting recognition model (504) is implemented as a convolutional neural network (CNN). FIG. 6 illustrates an exemplary convolutional neural network (602) trained on a multi-script training corpus (604) containing writing samples for characters in multiple non-overlapping scripts.

As illustrated in FIG. 6, the convolutional neural network (602) includes an input plane (606) and an output plane (608). Between the input plane (606) and the output plane (608), there are a plurality of convolutional layers (610) (e.g., a first convolutional layer (610a), zero or more intermediate convolutional layers (not illustrated), and a last convolutional layer (610n)). Each convolutional layer (610) is followed by a separate sub-sampling layer (612) (e.g., a first sub-sampling layer (612a), zero or more intermediate sub-sampling layers (not illustrated), and a last sub-sampling layer (612n)). A hidden layer (614) is present after the convolutional layers and sub-sampling layers and just before the output plane (608). The hidden layer (614) is the last layer before the output plane (608). In some embodiments, a kernel layer (616) (e.g., including a first kernel layer (616a), zero or more intermediate kernel layers (not shown), and a last kernel layer (612n)) is inserted before each convolutional layer (610) to improve the throughput efficiency.

As illustrated in FIG. 6, the input plane (606) receives an input image (614) of a handwriting recognition unit (e.g., a handwritten character or radical), and the output plane (608) outputs a set of probabilities indicating the likelihood that the recognition unit belongs to a particular output class (e.g., a particular character from a set of output characters that the neural network is configured to recognize). Collectively, the output classes of the neural network (or the set of output characters of the neural network) are also referred to as the repertoire or vocabulary of the handwriting recognition model. The convolutional neural network described herein can be trained to have a repertoire of tens of thousands of characters.

As the input image (614) is processed through different layers of the neural network, different spatial features embedded in the input image (614) are extracted by the convolutional layers (610). Each convolutional layer (610), also referred to as a set of feature maps, acts as filters to select specific features from the input image (614) to distinguish between images corresponding to different characters. The sub-sampling layers (612) ensure that the features are captured from the input image (614) at increasingly larger scales. In some embodiments, the sub-sampling layers (612) are implemented using a max-pooling technique. The max-pooling layers create positional invariance over larger local regions and down-sample the output image of the preceding convolutional layer by a factor of Kx and Ky along each direction, where Kx and Ky are the sizes of the max-pooling rectangle. Max-pooling leads to faster convergence by selecting good invariant features, which improves generalization performance. In some embodiments, sub-sampling is achieved using other methods.

In some embodiments, after the last set of convolutional layers (610n) and sub-sampling layers (612n) and before the output plane (608), there is a fully-connected layer, i.e. a hidden layer (614). The fully-connected hidden layer (614) is a multi-layer perceptron that fully connects the nodes in the last sub-sampling layer (612n) and the nodes in the output plane (608). The hidden layer (614) takes in the output images received from that layer before and while the logistic regression reaches one of the output characters in the output layer (608).

During training of the convolutional neural network (602), the features and the individual weights associated with the features in the convolutional layers (610) as well as the weights associated with the parameters in the hidden layer (614) are tuned such that classification errors are minimized for handwriting samples using known output classes in the training corpus (604). Once the convolutional neural network (602) is trained and an optimal set of parameters and associated weights for the different layers in the network are established, the convolutional neural network (602) is used to recognize new handwriting samples (618) that are not part of the training corpus (604), for example, input images generated based on real-time handwriting input received from a user.

As described herein, the convolutional neural network of the handwriting input interface is trained using a multi-script training corpus to enable multi-script or mixed-script handwriting recognition. In some embodiments, the convolutional neural network is trained to recognize a large repertoire of 30,000 to 60,000 characters (e.g., all characters encoded by the Unicode standard). Most state-of-the-art handwriting recognition systems are based on stroke-order dependent Hidden Markov Models (HMMs). In addition, most existing handwriting recognition models are language-specific and contain small repertoires ranging from a few dozen characters (e.g., characters such as the English alphabet, the Greek alphabet, all ten numbers, etc.) to a few thousand characters (e.g., the set of most common Chinese characters). Thus, the universal recognizer described herein can handle dozens of times more characters than most existing systems.

Some conventional handwriting systems may include several individually trained handwriting recognition models, each tailored to a particular language or small set of characters. A writing sample is propagated through the different recognition models until a distinction is made. For example, a handwriting sample may be fed to a series of successive language-specific or script-specific character recognition models, and if the handwriting sample cannot be reliably distinguished by the first recognition model, it is fed to the next recognition model, which attempts to distinguish the handwriting sample within its own repertoire. This approach to distinction is time consuming, and the memory requirements grow rapidly with each additional recognition model that needs to be utilized.

Other recent models ask the user to specify a preferred language and then use the selected handwriting recognition model to distinguish the current input. These implementations are not only cumbersome to use and consume significant memory, but they cannot be used to recognize mixed-language input. It is impractical to ask the user to switch language preferences while entering mixed-language or mixed-script input.

The multi-script or universal recognizer described herein addresses at least some of the above problems with conventional recognition systems. FIG. 7 is a flowchart of an exemplary process (700) for training a handwriting recognition module (e.g., a convolutional neural network) using a large multi-script training corpus such that the handwriting recognition module can subsequently be used to provide real-time multilingual and multi-script handwriting recognition of a user's handwriting input.

In some embodiments, after training of the handwriting recognition model is performed on the server device, the trained handwriting recognition model is provided to the user device. The handwriting recognition model optionally performs real-time handwriting recognition locally on the user device without requiring additional support from the server. In some embodiments, both training and recognition are provided on the same device. For example, the server device may receive a user's handwriting input from the user device, perform handwriting recognition, and transmit the recognition results to the user device in real time.

In an exemplary process (700), in a device having one or more processors and memory, the device trains a multi-script handwriting recognition model based on spatially-derived features (e.g., stroke-order independent features) of a multi-script training corpus (702). In some embodiments, the spatially-derived features of the multi-script training corpus are stroke-order independent and stroke-direction independent (704). In some embodiments, the training of the multi-script handwriting recognition model is independent of temporal information associated with individual strokes in the handwriting samples (706). Specifically, the images of the handwriting samples are normalized to a predetermined size, and the images do not contain any information about the order in which the individual strokes are input to form the image. In addition, the images also do not contain any information about the direction in which the individual strokes are input to form the image. In effect, during training, features are extracted from the handwriting images without regard to how the images are temporally formed by the individual strokes. Thus, during recognition, no temporal information associated with the individual strokes is needed. As a result, recognition provides robust and consistent recognition results despite delayed, out-of-order strokes and arbitrary stroke directions in handwriting input.

In some embodiments, the multi-script training corpus includes handwriting samples corresponding to characters in at least three non-overlapping scripts. As illustrated in FIG. 6, the multi-script training corpus includes handwriting samples collected from many users. Each handwriting sample corresponds to one character in a separate script represented by the handwriting recognition model. In order to properly train the handwriting recognition model, the training corpus includes a large number of writing samples for each character in the scripts represented by the handwriting recognition model.

In some embodiments, the at least three non-overlapping scripts include Chinese characters, emoji characters, and a Latin script (708). In some embodiments, the multi-script handwriting recognition model has at least 30,000 output classes representing 30,000 characters spanning the at least three non-overlapping scripts (710).

In some embodiments, the multi-script training corpus includes individual writing samples for each character in all of the Chinese characters encoded in the Unicode Standard (e.g., a significant portion or all of all CJK (Chinese-Japanese-Korean) unified ideographs). The Unicode Standard defines a total of about 74,000 CJK unified ideographs. The basic block of CJK unified ideographs (4E00-9FFF) includes 20,941 basic ideographs that are used in Chinese as well as Japanese, Korean, and Vietnamese. In some embodiments, the multi-script training corpus includes writing samples for all characters in the basic block of CJK unified ideographs. In some embodiments, the multi-script training corpus additionally includes writing samples for CJK radicals that can be used to structurally construct one or more compound ideographs. In some embodiments, the multi-script training corpus additionally includes writing samples for less frequently used Chinese characters, such as characters encoded in one or more of the CJK Unified Ideographic Extensions.

In some embodiments, the multi-script training corpus additionally includes individual writing samples for each character in all characters in a Latin script encoded by the Unicode standard. The characters in the basic Latin script include not only the Latin lowercase and uppercase letters, but also various basic symbols and numbers commonly used on a standard Latin keyboard. In some embodiments, the multi-script training corpus additionally includes characters in an extended Latin script (e.g., various accent forms of the basic Latin letters).

In some embodiments, the multi-script training corpus includes writing samples corresponding to each character of an artificial script that is not associated with any natural human language. For example, in some embodiments, a set of emoji characters is optionally defined in the emoji script, and writing samples corresponding to each of the emoji characters are included in the multi-script training corpus. For example, a hand-drawn heart-shaped symbol is a handwriting sample for the emoji character "♥" in the training corpus. Similarly, a hand-drawn smiley face (e.g., two dots on an inverted arc) is a handwriting sample for the emoji character " " is a handwriting sample for . Other emoji characters include categories of icons representing different emotions (e.g., happy, sad, angry, embarrassed, shocked, laughing, crying, frustrated, etc.), different objects and characters (e.g., cat, dog, rabbit, heart, fruit, eye, lips, gift, flower, candle, moon, star, etc.), and different actions (e.g., handshake, kiss, run, dance, jump, sleep, eat, meet, love, like, vote, etc.). In some embodiments, the strokes in the handwriting sample corresponding to an emoji character are simplified and/or stylized lines of the actual lines forming the corresponding emoji character. In some embodiments, each device or application may utilize a different design for the same emoji character. For example, a smiley emoji character presented to a female user may be different than a smiley emoji character presented to a male user, even if the handwriting inputs received from two users are substantially identical.

In some embodiments, the multi-script training corpus also includes writing samples for characters in other scripts, such as Greek script (e.g., including Greek letters and symbols), Cyrillic script, Hebrew script, and one or more other scripts encoded according to the Unicode standard. In some embodiments, the at least three non-overlapping scripts included in the multi-script training corpus include characters in Chinese characters, emoji characters, and Latin script. The Chinese characters, emoji characters, and characters in Latin script are, of course, non-overlapping scripts. Many other scripts may overlap with each other for at least some characters. For example, some characters in Latin script (e.g., A, Z) may also be found in many other scripts (e.g., Greek and Cyrillic). In some embodiments, the multi-script training corpus includes Chinese characters, Arabic script, and Latin script. In some embodiments, the multi-script training corpus includes other combinations of overlapping and/or non-overlapping scripts. In some embodiments, the multi-script training corpus contains writing samples for all characters encoded by the Unicode standard.

As illustrated in FIG. 7, in some embodiments, to train a multi-script handwriting recognition model, the device provides handwriting samples of a multi-script training corpus to a single convolutional neural network having a single input plane and a single output plane (712). The device uses the convolutional neural network to determine spatially-derived features (e.g., stroke-order independent features) of the handwriting samples and individual weights for the spatially-derived features to distinguish characters of at least three non-overlapping scripts represented in the multi-script training corpus (714). The multi-script handwriting recognition model differs from conventional multi-script handwriting recognition models in that a single handwriting recognition model having a single input plane and a single output plane is trained using all samples in the multi-script training corpus. Rather than relying on separate sub-networks each handling a small subset of the training corpus (e.g., sub-networks each trained to recognize characters in a particular script or characters used in a particular language), a single convolutional neural network is trained to distinguish all characters appearing in the multi-script training corpus. Furthermore, a single convolutional neural network is trained to distinguish a large number of characters spanning multiple non-overlapping scripts, rather than characters from several overlapping scripts, such as Latin and Greek scripts (e.g., having overlapping letters A, B, E, Z, etc.).

In some embodiments, the device provides real-time handwriting recognition of the user's handwriting input using a multi-script handwriting recognition model trained on spatially-guided features of a multi-script training corpus (716). In some embodiments, providing real-time handwriting recognition of the user's handwriting input comprises continuously revising the recognition output of the user's handwriting input as the user continues to provide additions or revisions to the handwriting input. In some embodiments, providing real-time handwriting recognition of the user's handwriting input further comprises providing the multi-script handwriting recognition model to the user device (718), wherein the user device receives the handwriting input from the user and locally performs handwriting recognition of the handwriting input based on the multi-script handwriting recognition model.

In some embodiments, the device provides a multi-script handwriting recognition model to a plurality of devices where the individual input languages of the plurality of devices do not overlap, and the multi-script handwriting recognition model is used on each of the plurality of devices to recognize handwriting in a different language associated with each user device. For example, if the multi-script handwriting recognition model is trained to recognize characters in many different scripts and languages, the same handwriting recognition model can be used globally to provide handwriting input for any of those input languages. A first device for a user who wishes to input only in English and Hebrew may provide handwriting input capabilities using the same handwriting recognition model as a second device for another user who wishes to input only in Chinese and emoji characters. Instead of requiring a user of a first device to install separate English handwriting input keyboards (e.g., implemented with an English-specific handwriting recognition model) and separate Hebrew handwriting input keyboards (e.g., implemented with a Hebrew-specific handwriting recognition model), the same universal multi-script handwriting recognition model can be installed once on the first device and used to provide handwriting input capabilities for both English and Hebrew, as well as mixed input of the two languages. Furthermore, instead of requiring a second user to install a Chinese handwriting input keyboard (e.g., implemented with a Chinese-specific handwriting recognition model) and a separate emoji handwriting input keyboard (e.g., implemented with an emoji handwriting recognition model), the same universal multi-script handwriting recognition model can be installed once on the second device and used to provide handwriting input capabilities for both Chinese and emoji, as well as mixed input of the two scripts. Handling large repertoires across multiple scripts (e.g., a significant portion or all of the characters encoded in nearly 100 different scripts) using a single multi-script handwriting model improves the usability of the recognizer without significant burden on users and device vendors.

Training a multi-script handwriting recognition model using a large multi-script training corpus is different from conventional HMM-based handwriting recognition systems in that it does not rely on temporal information associated with individual strokes of characters. Furthermore, the resource and memory requirements for a multi-script recognition system do not increase linearly with the increase in the number of symbols and languages covered by the multi-script recognition system. For example, in a conventional handwriting system, increasing the number of languages would mean adding another independently trained model, and the memory requirement would at least double to accommodate the increased capability of the handwriting recognition system. In contrast, when a multi-script model is trained with a multi-script training corpus, increasing the language coverage requires retraining the handwriting recognition model with additional handwriting samples and increasing the size of the output plane, but the increase is very modest. Assuming that the multi-script training corpus contains handwriting samples corresponding to n different languages and the multi-script handwriting recognition model occupies a memory of size m , then when increasing the language coverage to N languages (N>n), the device retrains the multi-script handwriting recognition model based on spatially-guided features of a second multi-script training corpus, wherein the second multi-script training corpus contains second handwriting samples corresponding to N different languages. The variation of M / m is kept substantially constant within the range of 1 to 2, wherein the variation of N / n is from 1 to 100. Once the multi-script handwriting recognition model is retrained, the device can provide real-time handwriting recognition of a user's handwriting input using the retrained multi-script handwriting recognition model.

FIGS. 8A and 8B illustrate exemplary user interfaces for providing real-time multi-script handwriting recognition and input on a portable user device (e.g., device (100)). In FIGS. 8A and 8B , a handwriting input interface (802) is displayed on a touch-sensitive display screen (e.g., touch screen (112)) of the user device. The handwriting input interface (802) includes a handwriting input area (804), a candidate display area (806), and a text input area (808). In some embodiments, the handwriting input interface (802) further includes a plurality of control elements, each of which can be invoked to cause the handwriting input interface to perform a predetermined function. As illustrated in FIG. 8A , a delete button, a space button, a carriage return or Enter button, and a keyboard toggle button are included in the handwriting input interface. Other control elements are possible and may optionally be provided in the handwriting input interface to suit different applications utilizing the handwriting input interface (802). The layout of the different components of the handwriting input interface (802) is merely exemplary and may vary across different devices and across different applications.

In some embodiments, the handwriting input area (804) is a touch-sensitive area for receiving handwriting input from a user. A sustained contact and its associated motion path within the handwriting input area (804) on the touch screen is registered as a handwriting stroke. In some embodiments, a handwriting stroke registered by the device is visually rendered within the handwriting input area (804) at the same locations traced by the sustained contact. As illustrated in FIG. 8A , a user may enter some handwritten characters (e.g., " "), some handwritten English characters (e.g., "Happy"), and hand-drawn emoji characters (e.g., a smiley face). The handwritten characters are distributed in a number of lines (e.g., two lines) in the handwriting input area (804).

In some embodiments, the candidate display area (806) displays one or more recognition results (e.g., 810 and 812) for the currently accumulated handwriting input in the handwriting input area (804). Typically, the top-ranked recognition result (e.g., 810) is displayed first in the candidate display area. As illustrated in FIG. 8A , since the handwriting recognition model described herein can recognize characters of multiple non-overlapping scripts, including Chinese characters, Latin scripts, and emoji characters, the recognition result (e.g., 810) provided by the recognition model accurately includes the Chinese characters, English characters, and emoji characters represented by the handwriting input. The user is not required to pause while entering input to select or switch recognition languages.

In some embodiments, the text input area (808) is an area that displays text input provided by an individual application that employs a handwriting input interface. As illustrated in FIG. 8A , the text input area (808) is used by a memo application, and the text currently displayed within the text input area (808) (e.g., "America .") is a text input already provided by the memo application. In some embodiments, the cursor (813) indicates the current text input position in the text input area (808).

In some embodiments, a user may select a particular recognition result displayed in the candidate display area (806), for example, by an explicit selection input (e.g., a tap gesture on one of the displayed recognition results) or an implicit confirmation input (e.g., a double-tap gesture in the handwriting input area or a tap gesture on an “Enter” button). As illustrated in FIG. 8B , the user has explicitly selected the top-ranked recognition result (810) using a tap gesture (as indicated by the contact (814) over the recognition result (810) in FIG. 8A ). In response to the selection input, the text of the recognition result (810) is inserted into the text input area (808) at the insertion point indicated by the cursor (813). As illustrated in FIG. 8B , once the text of the selected recognition result (810) is entered into the text input area (808), both the handwriting input area (804) and the candidate display area (806) are cleared. The handwriting input area (804) is now ready to accept a new handwriting input, and the candidate display area (806) can now be used to display recognition results for the new handwriting input. In some embodiments, the implicit confirmation input allows the top recognition result to be entered into the text input area (808) without requiring the user to stop and select the top recognition result. A well-designed implicit confirmation input improves the speed of text entry and reduces the cognitive burden placed on the user during text composition.

In some embodiments (not shown in FIGS. 8A and 8B ), the top-ranked recognition result of the current handwriting input is optionally provisionally displayed in the text input area (808). The provisional text input shown in the text input area (808) is visually distinguished from other text inputs in the text input area, for example, by a provisional input box surrounding the provisional text input. The text shown in the provisional input box has not yet been committed or provided to an associated application (e.g., a memo application) and is automatically updated when the top-ranked recognition result changes, for example, in response to a user revision of the current handwriting input, by the handwriting input module.

FIGS. 9A and 9B are flowcharts of an exemplary process (900) for providing multi-script handwriting recognition on a user device. In some embodiments, as illustrated in FIG. 900, the user device receives a multi-script handwriting recognition model (902), wherein the multi-script recognition model has been trained on spatially-derived features (e.g., stroke-order and stroke-direction independent features) of a multi-script training corpus, wherein the multi-script training corpus includes handwriting samples corresponding to characters in at least three non-overlapping scripts. In some embodiments, the multi-script handwriting recognition model is a single convolutional neural network having a single input plane and a single output plane (906), and includes spatially-derived features and separate weights for the spatially-derived features to distinguish characters in the at least three non-overlapping scripts represented in the multi-script training corpus. In some embodiments, the multi-script handwriting recognition model is configured to recognize characters based on individual input images of one or more recognition units identified in the handwriting input (908), wherein individual spatially-derived features used for recognition are independent of individual stroke order, stroke direction, and continuity of strokes in the handwriting input.

In some embodiments, the user device receives handwriting input from the user (908), the handwriting input comprising one or more handwriting strokes provided on a touch-sensitive surface coupled to the user device. For example, the handwriting input comprises discrete data regarding the location and movement of contact between a finger or stylus and the touch-sensitive surface coupled to the user device. In response to receiving the handwriting input, the user device provides to the user in real time (910) one or more handwriting recognition results based on a multi-script handwriting recognition model trained on spatially-derived features of a multi-script training corpus (912).

In some embodiments, when providing real-time handwriting recognition results to the user, the user device segments the user's handwriting input into one or more recognition units (914), each recognition unit including one or more of the handwriting strokes provided by the user. In some embodiments, the user device segments the user's handwriting input based on the shape, position, and size of the individual strokes made by contact between the user's finger or stylus and the touch-sensitive surface of the user device. In some embodiments, the segmentation of the handwriting input further takes into account the relative order and relative positions of the individual strokes made by contact between the user's finger or stylus and the touch-sensitive surface of the user device. In some embodiments, the user's handwriting input is in cursive, and each successive stroke in the handwriting input may correspond to multiple strokes in a character recognized in print form. In some embodiments, the user's handwriting input may include successive strokes spanning multiple recognized characters in print form. In some embodiments, the segmentation of the handwriting input generates one or more input images, each corresponding to a separate recognition unit. In some embodiments, some of the input images optionally include some overlapping pixels. In some embodiments, the input images do not include any overlapping pixels. In some embodiments, the user device generates a segmentation grid, and each segment chain of the segmentation grid represents a separate way to segment the current handwriting input. In some embodiments, each arc in the segmentation chain corresponds to a separate group of strokes in the current handwriting input.

As illustrated in drawing 900, the user device provides a separate image of each of the one or more recognition units as input to the multi-script recognition model (914). For at least one recognition unit of the one or more recognition units, the user device obtains from the multi-script handwriting recognition model at least a first output character from a first script and at least a second output from a second script different from the first script (916). For example, a same input image may cause the multi-script recognition model to output two or more similar-looking output characters from different scripts as recognition results for the same input image. For example, handwriting inputs for the letter "a" in Latin script and the letter "α" in Greek script are often similar. Additionally, handwriting inputs for the letter "J" in Latin script and the letter "丁" in Chinese script are often similar. Similarly, the emoji character " "The handwriting input for may be similar to the handwriting input for the CJK radical "西". In some embodiments, since the visual appearance of the handwriting input may be difficult for a human reader to decipher, the multi-script handwriting recognition model often generates multiple candidate recognition results that likely correspond to the user's handwriting input. In some embodiments, the first script is a CJK basic character block and the second script is a Latin script as encoded by the Unicode standard. In some embodiments, the first script is a CJK basic character block and the second script is a set of emoji characters. In some embodiments, the first script is a Latin script and the second script is emoji characters.

In some embodiments, the user device displays both the first output character and the second output character in the candidate display area of the handwriting input interface of the user device (918). In some embodiments, the user device selectively displays one of the first output character and the second output character based on which of the first script and the second script is an individual script used in a soft keyboard currently installed on the user device (920). For example, assuming that the handwriting recognition model has identified the Chinese character "入" and the Greek character "λ" as output characters for the current handwriting input, the user device determines whether the user has installed a Chinese soft keyboard (e.g., a keyboard using the Pinyin input method) or a Greek input keyboard on the user device. If the user device determines that only the Chinese soft keyboard is installed, the user device selectively displays only the Chinese character "入" as the recognition result to the user and does not display the Greek character "λ".

In some embodiments, the user device provides real-time handwriting recognition and input. In some embodiments, prior to the user making an explicit or implicit selection of a recognition result to be displayed to the user, the user device sequentially revises (922) one or more recognition results for the user's handwriting input in response to continued additions or revisions to the handwriting input by the user. In some embodiments, the user, in response to each revision of the one or more recognition results, displays (924) the individually revised one or more recognition results to the user in a candidate display area of the handwriting input user interface.

In some embodiments, the multi-script handwriting recognition model is trained to recognize all characters in at least three non-overlapping scripts, including Chinese characters, emoji characters, and Latin script encoded according to the Unicode standard (926). In some embodiments, the at least three non-overlapping scripts include Chinese characters, Arabic script, and Latin script. In some embodiments, the multi-script handwriting recognition model has at least 30,000 output classes representing at least 30 characters spanning the at least three non-overlapping scripts (928).

In some embodiments, the user device allows the user to input multi-script handwriting input, such as a phrase that includes characters from more than one script. For example, the user may continuously write and receive handwriting recognition results that include characters from more than one script, without stopping mid-writing to manually switch recognition languages. For example, the user may input the multi-script sentence "Hello means You can write "in Chinese.", but Chinese characters " "You can type the English words "in Chinese." without having to switch the input language from English to Chinese before typing them, or without having to switch the input language from Chinese back to English when typing the English words "in Chinese."

As described herein, a multi-script handwriting recognition model is used to provide real-time handwriting recognition of a user's input. In some embodiments, the real-time handwriting recognition is utilized to provide real-time multi-script handwriting input functionality on a user's device. FIGS. 10A-10C are flowcharts of an exemplary process (1000) for providing real-time handwriting recognition and input on a user's device. Specifically, the real-time handwriting recognition is stroke-order independent at the character level, phrase level, and sentence level.

In some embodiments, character-level stroke-order independent handwriting recognition requires that a handwriting recognition model provides the same recognition result for a particular handwritten character, regardless of the sequence in which the individual strokes of the particular character are provided by the user. For example, individual strokes of Chinese characters are typically written in a particular order. Although native Chinese speakers are often trained in school to write each character in a particular order, many users later adopt personalized styles and stroke sequences that deviate from the traditional stroke order. In addition, cursive writing styles are highly individualized, and multiple strokes in the printed form of a Chinese character are often merged into a single stylized stroke that is meandering and sometimes even connected to the next character. A stroke-order independent recognition model is trained based on images of writing samples that do not have time information associated with the individual strokes. Thus, recognition is independent of stroke-order information. For example, for the Chinese character "十", the handwriting recognition model will provide the same recognition result "十" regardless of whether the user wrote the horizontal stroke first or the vertical stroke first.

As illustrated in FIG. 10A, in process (1000), the user device receives a plurality of handwritten strokes from the user (1002), the plurality of handwritten strokes corresponding to handwritten characters. For example, a handwritten input for the character “十” typically includes a substantially horizontal handwritten stroke intersecting with a substantially vertical handwritten stroke.

In some embodiments, the user device generates an input image based on a plurality of handwritten strokes (1004). In some embodiments, the user device provides the input image to a handwriting recognition model to perform real-time handwriting recognition of the handwritten character (1006), wherein the handwriting recognition model provides stroke-order independent handwriting recognition. Thereafter, the user device, in real-time upon receiving the plurality of handwritten strokes, displays an identical first output character (e.g., the character "十" in printed form) regardless of the individual order in which the plurality of handwritten strokes (e.g., horizontal strokes and vertical strokes) were received from the user (1008).

Some conventional handwriting recognition systems tolerate small stroke order variations in a small number of characters, but do so by specifically including such variations during the training of the handwriting recognition system. Such conventional handwriting recognition systems are not scalable to accommodate arbitrary stroke order variations in a large number of complex characters, such as Chinese characters, since even moderately complex characters will already have a large number of variations in stroke order. Furthermore, simply by including more permutations of allowable stroke order for specific characters, conventional recognition systems would still be unable to handle handwriting inputs in which multiple strokes are combined into a single stroke (e.g., as in extremely cursive writing) or in which a single stroke is broken into multiple sub-strokes (e.g., as in characters captured by very coarse sampling of input strokes). Thus, a multi-script handwriting system trained on spatially-derived features, as described herein, offers advantages over conventional recognition systems.

In some embodiments, stroke-order independent handwriting recognition is performed regardless of temporal information associated with individual strokes within each handwritten character. In some embodiments, stroke-order independent handwriting recognition is performed in conjunction with stroke-distribution information that considers the spatial distribution of individual strokes before the individual strokes are merged into a flat input image. Further details on how temporal-derived stroke-distribution information is used to augment the stroke-order independent handwriting recognition described above are provided later herein (e.g., with respect to FIGS. 25A-27 ). The techniques described with respect to FIGS. 25A-27 do not compromise the stroke-order independence of the handwriting recognition system.

In some embodiments, the handwriting recognition model provides stroke direction independent handwriting recognition (1010). In some embodiments, stroke direction independent recognition requires that, in response to the user device receiving multiple handwriting inputs, each of the multiple handwriting strokes displays the same first output character regardless of the individual stroke direction provided by the user. For example, if a user writes the Chinese character "十" in a handwriting input area of the user device, the handwriting recognition model will output the same recognition result regardless of whether the user draws the horizontal stroke from left to right or from right to left. Similarly, the handwriting recognition model will output the same recognition result regardless of whether the user draws the vertical stroke in a downward direction or an upward direction. In another example, many Chinese characters are structurally composed of two or more radicals. Some Chinese characters each include a left radical and a right radical, and people traditionally write the left radical first and then the right radical. In some embodiments, as long as the resulting handwriting input indicates the left radical to the left of the right radical when the user completes the handwritten character, the handwriting recognition model will provide the same recognition result regardless of whether the user wrote the right radical first or the left radical first. Similarly, some Chinese characters each include an upper radical and a lower radical, and people customarily write the upper radical first and then the lower radical. In some embodiments, as long as the resulting handwriting input indicates the upper radical above the lower radical, the handwriting recognition model will provide the same recognition result regardless of whether the user wrote the upper radical first or the lower radical first. In other words, the handwriting recognition model does not rely on the directions in which the user provides individual strokes of the handwritten character to determine the identity of the handwritten character.

In some embodiments, the handwriting recognition model provides handwriting recognition based on an image of the recognition unit, regardless of the number of sub-strokes provided by the user. In other words, in some embodiments, the handwriting recognition model provides stroke-independent handwriting recognition (1014). In some embodiments, the user device, in response to receiving a plurality of handwritten strokes, displays the same first output character regardless of how many handwritten strokes are used to form a continuous stroke in the input image. For example, if a user inputs a Chinese character "+" in a handwriting input area, the handwriting recognition model outputs the same recognition result regardless of whether the user provided four strokes to form the shape of the character "十" (e.g., two short horizontal strokes and two short vertical strokes to form a cross-shaped character), or two strokes (e.g., an L-shaped stroke and a 7-shaped stroke, or a horizontal stroke and a vertical stroke), or any other number of strokes (e.g., hundreds of extremely short strokes or dots).

In some embodiments, the handwriting recognition model can recognize the same character regardless of the order, direction, and number of strokes in which each single character is written, as well as recognize multiple characters regardless of the chronological order in which the strokes of the multiple characters are provided by the user.

In some embodiments, the user device receives a first plurality of handwritten strokes as well as a second plurality of handwritten strokes from the user (1016), wherein the second plurality of handwritten strokes correspond to a second handwritten character. In some embodiments, the user device generates a second input image based on the second plurality of handwritten strokes (1018). In some embodiments, the user device provides the second input image to a handwriting recognition model to perform real-time recognition of the second handwritten character (1020). In some embodiments, the user device displays, in real-time upon receiving the second plurality of handwritten strokes, a second output character corresponding to the second plurality of handwritten strokes (1022). In some embodiments, the first output character and the second output character are displayed simultaneously in a spatial sequence, regardless of the individual order in which the first plurality of handwritten strokes and the second plurality of handwritten strokes were provided by the user. For example, if a user has written two Chinese characters (e.g., "十" and "八") in a handwriting input area of a user device, as long as the current accumulated handwriting input in the handwriting input area represents strokes for the character "十" to the left of the strokes for the character "八", the user device will display the recognition result "十八" regardless of whether the user wrote the strokes of the character "十" first or the strokes of the character "八" first. In fact, even if the user wrote some of the strokes for the character "八" (e.g., the left-slanted stroke) before some of the strokes for the character "十" (e.g., the vertical stroke), as long as the resulting image of the handwriting input in the handwriting input area represents all of the strokes for the character "十" to the left of all of the strokes for the character "八", the user device will display the recognition result "十八" in the spatial sequence of the two handwritten characters.

In other words, as illustrated in FIG. 10B , in some embodiments, the spatial sequence of the first output character and the second output character corresponds to a spatial distribution of the first plurality of handwritten strokes and the second plurality of strokes along a default writing direction of the handwriting input interface of the user device (e.g., from left to right) (1024). In some embodiments, the second plurality of handwritten strokes are received temporally after the first plurality of handwritten strokes, and the second output character precedes the first output character in the spatial sequence along the default writing direction of the handwriting input interface of the user device (e.g., from left to right) (1026).

In some embodiments, the handwriting recognition model provides stroke-order independent recognition at a sentence-to-sentence level. For example, even if a handwritten character "十" is in a first handwritten sentence and a handwritten character "八" is in a second handwritten sentence, and the two handwritten characters are separated by one or more other handwritten characters and/or words in the handwriting input area, the handwriting recognition model will still provide a recognition result representing the two characters as a spatial sequence "十…八". The recognition result and the spatial sequence of the two recognized characters remain the same regardless of the temporal order in which the strokes of the two characters are provided by the user, except that when the user completes the handwriting input, the recognition units for the two characters are spatially arranged as the sequence "十…八". In some embodiments, a first handwritten character (e.g., "十") is provided by the user as part of a first handwritten sentence (e.g., "十 is a number."), a second handwritten character (e.g., "八") is provided by the user as part of a second handwritten sentence (e.g., "八 is another number."), and the first handwritten sentence and the second handwritten sentence are displayed simultaneously in a handwriting input area of the user device. In some embodiments, when the user confirms that the recognition result (e.g., "十 is a number. 八. is another number.") is an accurate recognition result, the two sentences will be input into a text input area of the user device, and the handwriting input area will be cleared to allow the user to input another handwriting input.

In some embodiments, since the handwriting recognition model is stroke-order independent not only at the character level but also at the phrase level and sentence level, the user can make corrections to a previously incomplete character after subsequent characters have been entered. For example, if the user forgets to enter a particular stroke for a character before moving to enter one or more subsequent characters in the handwriting input area, the user can still later enter the missing stroke at the correct location for the particular character to receive an accurate recognition result.

In conventional stroke-order-independent recognition systems (e.g., HMM-based recognition systems), once a character is entered, it is committed and the user can no longer change it. If the user wants to make any changes, the user must delete the character and all subsequent characters and start over. In some conventional recognition systems, the user is required to complete the handwritten character within a short, predetermined time window, and any strokes entered outside the predetermined time window will not be included in the same recognition unit as other strokes entered during the time window. Such conventional systems are difficult to use and cause great frustration to the user. A stroke-order-independent system does not suffer from such drawbacks, and the user can complete the character in any order and in any time frame that the user deems appropriate. The user can also modify the previously entered character (e.g., add one or more strokes) after entering one or more characters in the handwriting input interface subsequently. In some embodiments, the user may also delete a previously written character individually (e.g., using the methods described later with respect to FIGS. 21A-22B ) and re-write it in the same location in the handwriting input interface.

As illustrated in FIGS. 10b and 10c , the second plurality of handwritten strokes spatially follow the first plurality of handwritten strokes along a default writing direction of a handwriting input interface of the user device, and the second output character follows the first output character in a spatial sequence along the default writing direction in a candidate display area of the handwriting input interface (1028). The user device receives a third handwritten stroke from the user to revise the first handwritten character (i.e., the handwritten character formed by the first plurality of handwritten strokes) (1030), wherein the third handwritten stroke is received temporally after the first plurality of handwritten strokes and the second plurality of handwritten strokes. For example, the user may write two characters (e.g., " ") was entered. The first multiple strokes form the handwritten character "八". The user actually entered the character " " I tried to write " but I missed one stroke. The second multiple strokes are the handwritten characters " " is formed. The user " " Instead " ", the user can simply write one more vertical stroke beneath the strokes for the character "八", and the user device will assign the vertical stroke to the first recognition unit (e.g., the recognition unit for "八"). The user device will output a new output character (e.g., "八") for the first recognition unit, where the new output character will replace the previous output character (e.g., "八") in the recognition result. As illustrated in FIG. 10C , in response to receiving the third handwritten stroke, the user device assigns the third handwritten stroke to the same recognition unit as the first plurality of handwritten strokes based on the relative proximity of the third handwritten stroke to the first plurality of handwritten strokes (1032). In some embodiments, the user device generates a revised input image based on the first plurality of handwritten strokes and the third handwritten stroke (1034). The user device provides the revised input image to the handwriting recognition model to generate a real-time representation of the revised handwritten character. Perform recognition (1036). In some embodiments, in response to receiving the third handwriting input, the user device displays (1040) a third output character corresponding to the revised input image, wherein the third output character replaces the first output character and is displayed concurrently with the second output character in a spatial sequence along the default writing direction.

In some embodiments, the handwriting recognition module recognizes handwriting input written in a default writing direction from left to right. For example, a user may write characters from left to right and in one or more rows. In response to the handwriting input, the handwriting input module presents recognition results including the characters in a spatial sequence from left to right and, optionally, in one or more rows. When the user selects a recognition result, the selected recognition result is input into a text input area of the user device. In some embodiments, the default writing direction is from top to bottom. In some embodiments, the default writing direction is from right to left. In some embodiments, the user optionally changes the default writing direction to an alternative writing direction after a recognition result is selected and the handwriting input area is cleared.

In some embodiments, the handwriting input module allows a user to enter multi-character handwriting input in a handwriting input area, and allows deletion of strokes from the handwriting input one recognition unit at a time, rather than all recognition units at a time. In some embodiments, the handwriting input module allows deletion of strokes from the handwriting input one at a time. In some embodiments, the deletion of recognition units proceeds one by one in a direction opposite to the default writing direction, regardless of the order in which the recognition units or strokes were entered to generate the current handwriting input. In some embodiments, the deletion of strokes proceeds one by one in the reverse order in which the strokes were entered within each recognition unit, and when all strokes in a recognition unit have been deleted, the deletion of strokes proceeds to the next recognition unit in a direction opposite to the default writing direction.

In some embodiments, while the third output character and the second output character are simultaneously displayed as candidate recognition results in the candidate display area of the handwriting input interface, the user device receives a delete input from the user. In response to the delete input, the user device deletes the second output character from the recognition result while maintaining the third output character in the recognition result displayed in the candidate display area.

In some embodiments, as illustrated in FIG. 10C , the user device renders in real time (1042) the first plurality of handwritten strokes, the second plurality of handwritten strokes, and the third handwritten stroke as each of the handwritten strokes is provided by the user. In some embodiments, the user device, in response to receiving a delete input from the user, deletes (1044) the individual rendering of the second plurality of handwritten inputs (e.g., corresponding to the second handwritten character) from the handwriting input area while retaining the individual renderings of the first plurality of handwritten strokes and the third handwritten stroke (e.g., corresponding to the revised first handwritten character) in the handwriting input area. For example, the user may input the character sequence " "After providing a missing vertical stroke in , if the user types a delete input, the character " "Strokes in the recognition unit for are removed from the handwriting input area, and the character " "Recognition results in the candidate display area of the user device" " is removed from. After deletion, the character " "The strokes remain in the handwriting input area, while the recognition result is the character " " represents only.

In some embodiments, the handwritten characters are multi-stroke Chinese characters. In some embodiments, the first plurality of handwritten inputs are provided in cursive script. In some embodiments, the first plurality of handwritten inputs are provided in cursive script and the handwritten characters are multi-stroke Chinese characters. In some embodiments, the handwritten characters are written in Arabic in cursive script. In some embodiments, the handwritten characters are written in other scripts in cursive script.

In some embodiments, the user device establishes individual predetermined constraints for a set of allowable dimensions for handwritten character input, and divides the currently accumulated plurality of handwritten strokes into a plurality of recognition units based on the individual predetermined constraints, wherein an individual input image is generated from each of the recognition units, which is provided to a handwriting recognition model and recognized as a corresponding output character.

In some embodiments, the user device receives an additional handwritten stroke from the user after segmenting the currently accumulated plurality of handwritten strokes. The user device assigns the additional handwritten stroke to an individual recognition unit of the plurality of recognition units based on a spatial location of the additional handwritten stroke with respect to the plurality of recognition units.

Now, we turn to exemplary user interfaces for providing handwriting recognition and input on a user device. In some embodiments, the exemplary user interfaces are provided on the user device based on a multi-script handwriting recognition model that provides real-time, stroke-order independent handwriting recognition of a user's handwriting input. In some embodiments, the exemplary user interfaces are user interfaces of an exemplary handwriting input interface (802) (e.g., as illustrated in FIGS. 8A and 8B ) that includes a handwriting input area (804), a candidate display area (804), and a text input area (808). In some embodiments, the exemplary handwriting input interface (802) also includes a plurality of control elements (1102), such as a delete button, a space bar, an enter button, a keyboard toggle button, and the like. One or more other areas and/or elements may be provided in the handwriting input interface (802) to enable additional functionality as described below.

As described herein, a multi-script handwriting recognition model can have a very large repertoire of tens of thousands of characters in many different scripts and languages. As a result, for handwriting input, the recognition model is very likely to identify a large number of output characters that all have a fairly good chance of being the characters intended by the user. On a user device with a limited display area, it is advantageous to initially present only a subset of the recognition results while keeping other results available upon user request.

Figures 11a-11g illustrate exemplary user interfaces for displaying a subset of recognition results in the normal field of view of a candidate display region, along with an affordance for invoking an extended field of view of the candidate display region to display the remaining recognition results. Additionally, within the extended field of view of the candidate display region, the recognition results are divided into different categories and displayed on different tabbed pages in the extended field of view.

FIG. 11A illustrates an exemplary handwriting input interface (802). The handwriting input interface includes a handwriting input area (804), a candidate display area (806), and a text input area (808). One or more control elements (1102) are also included in the handwriting input interface (1002).

As illustrated in FIG. 11a, the candidate display area (806) optionally includes an area for displaying one or more recognition results, and an affordance (1104) for invoking an expanded version of the candidate display area (806) (e.g., an expand icon).

FIGS. 11A-11C illustrate that as a user provides one or more handwritten strokes (e.g., strokes (1106, 1108, 1110)) in a handwriting input area (804), the user device identifies and displays a separate set of recognition results corresponding to the currently accumulated strokes in the handwriting input area (804). As illustrated in FIG. 11B , after the user inputs a first stroke (1106), the user device identifies and displays three recognition results (1112, 1114, 1116) (e.g., characters "/", "1", and ","). In some embodiments, a small number of candidate characters are displayed in the candidate display area (806) in an order of recognition confidence associated with each character.

In some embodiments, the top candidate result (e.g., "/") is provisionally displayed in the text input area (808), for example, within a box (1118). The user may optionally confirm that the top candidate is the intended input by a simple confirmation input (e.g., by pressing the "Enter" key or providing a double-tap gesture in the handwriting input area).

FIG. 11C illustrates that as the user inputs two more strokes (1108, 1100) in the handwriting input area (804) before selecting any candidate recognition result, the additional strokes are rendered in the handwriting input area (804) having the initial stroke (1106) and the candidate results are updated to reflect changes to the recognition unit(s) identified from the currently accumulated handwriting inputs. As illustrated in FIG. 11C , based on the three strokes, the user device identified a single recognition unit. Based on the single recognition unit, the user device identified and displayed multiple recognition results (1118-1124). In some embodiments, one or more of the recognition results currently displayed in the candidate display area (806) (e.g., 1118 and 1122) are each a representative candidate character selected from a plurality of similar-looking candidate characters for the current handwriting input.

As illustrated in FIGS. 11C and 11D , when a user selects an affordance (1104) (e.g., using a tap gesture with contact (1126) over the affordance (1104)), the candidate display area changes from the normal field of view (e.g., illustrated in FIG. 11C ) to an extended field of view (e.g., illustrated in FIG. 11D ). In some embodiments, the extended field of view represents all recognition results (e.g., candidate characters) identified for the current handwriting input.

In some embodiments, the initially displayed normal field of view of the candidate display area (806) shows only most common characters used in an individual script or language, while the extended field of view shows all candidate characters, including characters that are rarely used in the script or language. The extended field of view of the candidate display area can be designed in different ways. FIGS. 11D-11G illustrate exemplary designs of an extended candidate display area, according to some embodiments.

As illustrated in FIG. 11d , in some embodiments, the expanded candidate display area (1128) includes one or more tabbed pages (e.g., pages 1130, 1132, 1134, 1136) each presenting a separate category of candidate characters. The tabbed design illustrated in FIG. 11d allows a user to quickly find a desired category of characters, and then locate the character he or she intended to enter within the corresponding tabbed page.

In FIG. 11d , the first tabbed page (1130) displays all candidate characters identified for the current accumulated handwriting input, including both common characters as well as rare characters. As illustrated in FIG. 11d , the tabbed page (1130) displays all of the characters shown in the initial candidate display area (806) of FIG. 11c , as well as a number of additional characters not included in the initial candidate display area (806), such as " ", "β", "巾", etc.)

In some embodiments, the characters displayed in the initial candidate display area (806) include only characters from a set of common characters associated with the script (e.g., all characters in the basic block of the CJK script, as encoded according to the Unicode standard). In some embodiments, the characters displayed in the extended candidate display area (1128) additionally include a set of rare characters associated with the script (e.g., all characters in the extended block of the CJK script, as encoded according to the Unicode standard). In some embodiments, the extended candidate display area (1128) additionally includes candidate characters from other scripts that are not commonly used by users, such as Greek script, Arabic script, and/or emoji script.

In some embodiments, as illustrated in FIG. 11d , the expanded candidate display area (1128) includes individual tabbed pages (1130, 1132, 1134, 1138), each of which corresponds to a respective category of candidate characters (e.g., all characters, rare characters, characters from the Latin script, and characters from the emoji script, respectively). FIGS. 11e-11g illustrate that a user can select each of the different tabbed pages to reveal candidate characters in the corresponding category. FIG. 11e illustrates only the rare characters corresponding to the current handwriting input (e.g., characters from the extended block of the CJK script). FIG. 11f illustrates only the Latin and Greek characters corresponding to the current handwriting input. FIG. 11g illustrates only the emoji characters corresponding to the current handwriting input.

In some embodiments, the extended candidate display area (1128) further includes one or more affordances for sorting candidate characters on individual tabbed pages based on individual criteria (e.g., based on phonetic spelling, based on number of strokes, based on radicals, etc.). The ability to sort candidate characters in each category based on criteria other than recognition confidence scores provides the user with an additional ability to quickly find a desired candidate character for text input.

FIGS. 11H-11K illustrate that in some embodiments, similar-looking candidate characters can be grouped, and only a representative character from each group of similar-looking candidate characters is presented in the initial candidate display area (806). Since the multi-script recognition model described herein can generate many candidate characters that are roughly equally good for a given handwriting input, the recognition model cannot always eliminate one candidate at the expense of other similar-looking candidates. On a device with a limited display area, suddenly displaying a large number of similar-looking candidates does not help the user select the correct character, because fine distinctions may not be readily visible, and even if the user can see the desired character, it may be difficult to select it from a very dense display using a finger or stylus.

In some embodiments, to address the above issues, the user device identifies candidate characters that have a high degree of similarity to each other (e.g., according to a dictionary or glossary of similar-looking characters, or according to some image-based criteria) and groups them into separate groups. In some embodiments, one or more groups of similar-looking characters can be identified from a set of candidate characters for a given handwriting input. In some embodiments, the user device identifies a representative candidate character among multiple similar-looking candidate characters in a same group and displays only the representative candidate in the initial candidate display area (806). If a common character does not appear sufficiently similar to any of the other candidate characters, it is displayed by itself. In some embodiments, as illustrated in FIG. 11h , the representative candidate character for each group (e.g., candidate characters 1118 , 1122 ), " " and "T") are displayed in a different manner (e.g., as bold boxes) than candidate characters that do not belong to any group (e.g., candidate characters (1120, 1124), "乃" and "J"). In some embodiments, the criteria for selecting a representative character for a group is based on the relative usage frequencies of the candidate characters in the group. In some embodiments, other criteria may be used.

In some embodiments, once the representative character(s) are displayed to the user, the user may optionally expand the candidate display area (806) to reveal similarly appearing candidate characters in an expanded view. In some embodiments, selection of a particular representative character may result in an expanded view of only those candidate characters in the same group as the selected representative character.

Various designs are possible for providing expanded views of candidates that appear similar. FIGS. 11h-11k illustrate one embodiment where an expanded view of a representative candidate character is invoked by a predetermined gesture (e.g., an expanded gesture) detected over the representative candidate character (e.g., representative character (1118)). The predetermined gesture (e.g., the expanded gesture) for invoking the expanded view is different from the predetermined gesture (e.g., a tap gesture) for selecting the representative character for text input.

As illustrated in FIGS. 11h and 11i , when a user provides an expand gesture over a first representative character (1118) (e.g., as indicated by two contacts (1138, 1140) moving away from each other), the area displaying the representative character (1118) is expanded and three similar-looking candidate characters (e.g., "乃") are displayed in an expanded field of view (e.g., in expanded boxes (1142, 1144, 1146) respectively) compared to other candidate characters (e.g., "乃") that are not within the same expanded group. ", " " and "巾") are presented.

As shown in Figure 11i, when presented in an enlarged view, three similar-looking candidate characters (e.g., " ", " " and "巾") can be more easily seen by the user. If one of the three candidate characters is the intended character input, the user can select the candidate character, for example, by touching the area where the candidate character is displayed. As illustrated in FIGS. 11J and 11K , the user can select the second character (e.g., " ) shown in box (1144) in the expanded field of view. ") was selected (by contact (1148)). In response, the selected character (e.g., " ") is entered into the text input area (808) at the insertion location indicated by the cursor. As illustrated in FIG. 11k, once a character is selected, the handwriting input in the handwriting input area (804) and the candidate characters in the candidate display area (806) (or the extended field of view of the candidate display area) are cleared for subsequent handwriting input.

In some embodiments, if the user does not see the desired candidate character in the expanded field of view (1142) of the first representative candidate character, the user may optionally use the same gesture to expand other representative characters displayed in the candidate display area (806). In some embodiments, expanding other representative characters in the candidate display area (806) automatically restores the currently presented expanded field of view to the normal field of view. In some embodiments, the user may optionally use the collapse gesture to restore the current expanded field of view to the normal field of view. In some embodiments, the user may scroll the candidate display area (806) (e.g., to the left or right) to reveal other candidate characters that are not visible in the candidate display area (806).

FIGS. 12A and 12B are flowcharts of an exemplary process (1200) in which a first subset of recognition results are presented in an initial candidate display area, while a second subset of recognition results are presented in an extended candidate display area that is hidden from view until specifically invoked by a user. In the exemplary process (1200), the device identifies, from a plurality of handwriting recognition results for a handwriting input, a subset of recognition results having a level of visual similarity exceeding a predetermined threshold. The user device then selects a representative recognition result from the subset of recognition results and displays the selected representative recognition result in a candidate display area of the display. The process (1200) is illustrated in FIGS. 11A through 11K.

As illustrated in FIG. 12A , in an exemplary process (1200), a user device receives a handwriting input from a user (1202). The handwriting input includes one or more handwriting strokes (e.g., 1106, 1108, 1110 of FIG. 11C ) provided in a handwriting input area (e.g., 806 of FIG. 11C ) of a handwriting input interface (e.g., 802 of FIG. 11C ). The user device identifies (1204) a plurality of output characters for the handwriting input (e.g., characters shown in the tabbed page (1130) of FIG. 11C ) based on a handwriting recognition model. The user device divides the plurality of output characters into two or more categories based on predetermined classification criteria (1206). In some embodiments, the predetermined classification criteria determines whether an individual character is a common character or a rare character (1208).

In some embodiments, the user device displays (1210) individual output characters (e.g., common characters) within a first category of two or more categories in an initial view of a candidate display area of the handwriting input interface (e.g., 806 as illustrated in FIG. 11c ), wherein the initial view of the candidate display area is simultaneously provided with an affordance (e.g., 1104 of FIG. 11c ) for invoking an extended view of the candidate display area (e.g., 1128 of FIG. 11d ).

In some embodiments, the user device receives user input selecting an affordance for invoking an expanded field of view (1212), such as illustrated in FIG. 11c. In response to the user input, the user device displays, in the expanded field of view of the candidate display area, individual output characters within a first category of the two or more categories and individual output characters within at least a second category that were not previously displayed in the initial field of view of the candidate display area (1214), such as illustrated in FIG.

In some embodiments, the individual characters in the first category are characters found in a dictionary of common characters, and the individual characters in the second category are characters found in a dictionary of rare characters. In some embodiments, the dictionary of common characters and the dictionary of rare characters are dynamically adjusted or updated based on the usage history associated with the user device.

In some embodiments, the user device identifies (1216) a group of visually similar characters from a plurality of output characters based on a predetermined similarity criterion (e.g., based on a dictionary of similar characters or based on some spatially-derived features). In some embodiments, the user device selects a representative character from the group of visually similar characters based on a predetermined selection criterion (e.g., based on historical frequency of use). In some embodiments, the predetermined selection criterion is based on the relative frequency of use of the characters in the group. In some embodiments, the predetermined selection criterion is based on a preferred input language associated with the device. In some embodiments, the representative candidates are selected based on other factors that indicate the likelihood that each candidate is intended input by the user. These factors include, for example, whether the candidate character belongs to a script for a soft keyboard currently installed on the user's device, or whether the candidate character is among a set of most common characters in a particular language associated with the user or the user device.

In some embodiments, the user device selects, in an initial view of the candidate display area (e.g., 806 of FIG. 11h ), other characters in a group of visually similar characters (e.g., " ", "巾") instead of representative characters (e.g., " ") is displayed (1220). In some embodiments, visual cues (e.g., optional visual highlight, special background) are provided in the initial view of the candidate display area to indicate whether each candidate character is the representative character of the group or a generic candidate character not in any group. In some embodiments, as illustrated in FIG. 11h , the user device receives (1222) a predetermined expansion input (e.g., an expansion gesture) from the user that directs the representative character to be displayed in the initial view of the candidate display area. In some embodiments, as illustrated in FIG. 11i , the user device, in response to receiving the predetermined expansion input, simultaneously displays an expanded view of the representative character in the group of visually similar characters and an individual expanded view of one or more other characters (1224).

In some embodiments, the predetermined expansion input is an expansion gesture detected over a representative character displayed in the candidate display area. In some embodiments, the predetermined expansion input is a contact detected over a representative character displayed in the candidate display area and sustained for longer than a predetermined threshold time. In some embodiments, a sustained contact to expand a group has a longer threshold duration than a tap gesture to select a representative character for text input.

In some embodiments, each representative character is displayed simultaneously with an individual affordance (e.g., an individual expand button) for invoking an expanded view of a group of candidate characters that appear similar to it. In some embodiments, the predetermined expand input is selection of an individual affordance associated with the representative character.

As described herein, in some embodiments, the repertoire of the multi-script handwriting recognition model includes emoji scripts. The handwriting input recognition module can recognize emoji characters based on the user's handwriting input. In some embodiments, the handwriting recognition module presents both emoji characters identified directly from the handwriting, and characters or words in natural human language representing the identified emoji characters. In some embodiments, the handwriting input module recognizes characters or words in natural human language based on the user's handwriting input, and presents both the recognized characters or words, and emoji characters corresponding to the recognized characters or words. In other words, the handwriting input module provides ways to input emoji characters without switching from the handwriting input interface to the emoji keyboard. Additionally, the handwriting input module also provides ways to input regular natural language characters and words by drawing the emoji character by hand. Figures 13A-13E provide exemplary user interfaces illustrating these different ways to input emoji characters and regular natural language characters.

FIG. 13A illustrates an exemplary handwriting input interface (802) invoked under a chat application. The handwriting input interface (802) includes a handwriting input area (804), a candidate display area (806), and a text input area (808). In some embodiments, once the user is satisfied with the text composition in the text input area (808), the user may elect to transmit the text composition to other participants in the current chat session. The conversation history of the chat session is presented in the conversation panel (1302). In this example, the user may enter a chat message (1304) (e.g., "Happy Birthday") displayed in the conversation panel (1302). ") was received.

As illustrated in FIG. 13b, a user provided a handwriting input (1306) for the English word "Thanks" in the handwriting input area (804). In response to the handwriting input (1306), the user device identified a number of candidate recognition results (e.g., recognition results (1308, 1310, 1312)). The top recognition result (1303) was provisionally entered into the text input area (808) within a box (1314).

As illustrated in FIG. 13C , after the user inputs the handwritten word "Thanks" in the handwriting input area (806), the user then draws a stylized exclamation mark (e.g., an elongated circle with a round circle underneath) with strokes (1316) in the handwriting input area (806). The user device recognizes that the additional strokes (1316) form a recognition unit separate from other recognition units previously recognized from the accumulated handwritten strokes (1306) in the handwriting input area (806). Based on the newly input recognition unit (i.e., the recognition unit formed by the strokes (1316)), the user device identifies an emoji character (e.g., a stylized "!") using a handwriting recognition model. Based on the recognized emoji character, the user device presents a first recognition result (1318) (e.g., "Thanks!" with a stylized "!") in the candidate display area (806). Additionally, the user device also identifies a digit "8" that is visually similar to the newly input recognition unit. Based on this recognized digit, the user device presents a second recognition result (1322) (e.g., "Thanks 8") in the candidate display area (806). Additionally, based on the identified emoji character (e.g., the stylized "!"), the user device also identifies a regular character corresponding to the emoji character (e.g., the regular character "!"). Based on this indirectly identified regular character, the user device presents a third recognition result (1320) (e.g., "Thanks!" with the regular character "!") in the candidate display area (806). At this point, the user can select any one of the candidate recognition results (1318, 1320, 1322) and input it into the text input area (808).

As illustrated in FIG. 13D , the user continues to provide additional handwritten strokes (1324) in the handwriting input area (806). This time, the user has drawn a heart symbol followed by a stylized exclamation mark. In response to the new handwritten strokes (1324), the user device recognizes that the newly provided handwritten strokes (1324) form another new recognition unit. Based on the new recognition unit, the user device identifies the emoji character "♥", and alternatively the number "0", as candidate characters for the new recognition unit. Based on these new candidate characters recognized from the new recognition unit, the user device presents two updated candidate recognition results (1326, 1330) (e.g., "Thanks !♥" and "Thanks 80"). In some embodiments, the user device additionally identifies regular character(s) or word(s) (e.g., "Love") corresponding to the identified emoji character (e.g., "♥"). Based on the identified regular character(s) or word(s) for the recognized emoji character, the user device presents a third recognition result (1328) in which the recognized emoji character(s) are replaced with the corresponding regular character(s) or word(s). As illustrated in FIG. 13d , in the recognition result (1328), the emoji character “!” is replaced with a general exclamation mark “!”, and the emoji character “♥” is replaced with regular characters or the word “Love”.

As illustrated in FIG. 13e, a user selects one of the candidate recognition results (e.g., a candidate result (1326) representing the mixed script text "Thanks!♥"), and the text of the selected recognition result is entered into the text input area (808) and subsequently transmitted to other participants in the chat session. The message bubble (1332) represents the text of the message in the conversation panel (1302).

FIG. 14 is a flowchart of an exemplary process (1400) in which a user inputs an emoji character using handwriting input. FIGS. 13A through 13E illustrate an exemplary process (1400) according to some embodiments.

In process (1400), the user device receives handwriting input from the user (1402). The handwriting input includes a plurality of handwriting strokes provided in a handwriting input area of a handwriting input interface. In some embodiments, the user device recognizes a plurality of output characters from the handwriting input based on a handwriting recognition model (1404). In some embodiments, the output characters include at least a first emoji character from a script of a natural human language (e.g., a stylized exclamation mark "!" or emoji character "♥" of FIG. 13D ) and at least a first character (e.g., a character from the word "Thanks" of FIG. 13D ). In some embodiments, for example, as illustrated in FIG. 13D , the user device displays (1406) a recognition result (e.g., result (1326) of FIG. 13D ) that includes a first emoji character from a script of natural human language (e.g., a stylized exclamation mark “!” or emoji character “♥” of FIG. 13D ) and a first character (e.g., a character from the word “Thanks” of FIG. 13D ).

In some embodiments, based on the handwriting recognition model, the user device optionally recognizes (1408) at least a first semantic unit (e.g., the word "thanks") from the handwriting input, wherein the first semantic unit comprises an individual character, word, or phrase that can convey an individual semantic meaning in a particular human language. In some embodiments, the user device identifies (1410) a second emoji character (e.g., a "handshake" emoji character) associated with the first semantic unit (e.g., the word "Thanks") recognized from the handwriting input. In some embodiments, the user device displays (1412) a second recognition result in a candidate display area of the handwriting input interface that includes at least the second emoji character identified from the first semantic unit (e.g., the word "Thanks") (e.g., a recognition result representing the "handshake" emoji character followed by "!" and "♥" emoji characters). In some embodiments, displaying the second recognition result further comprises displaying the second recognition result simultaneously with a third recognition result (e.g., the recognition result "Thanks!♥") that includes at least the first semantic unit (e.g., the word "Thanks").

In some embodiments, the user receives user input selecting a first recognition result to be displayed in the candidate display area. In some embodiments, the user device, in response to the user input, enters text of the selected first recognition result in a text input area of the handwriting input interface, wherein the text includes at least a first emoji character and a first character from a script of a natural human language. In other words, the user can input mixed-script text input using a single handwriting input (even if the handwriting input includes multiple strokes) in the handwriting input area without switching between a natural language keyboard and an emoji character keyboard.

In some embodiments, the handwriting recognition model is trained on a multi-script training corpus including writing samples corresponding to characters in at least three non-overlapping scripts, the three non-overlapping scripts including a set of emoji characters, Chinese characters, and a Latin script.

In some embodiments, the user device identifies (1414) a second semantic unit (e.g., the word "Love") corresponding to a first emoji character (e.g., the "♥" emoji character) recognized directly from the handwriting input. In some embodiments, the user device displays (1416) a fourth recognition result (e.g., 1328 of FIG. 13D ) in the candidate display area of the handwriting input interface that includes at least the second semantic unit (e.g., the word "Love") identified from the first emoji character (e.g., the "♥" emoji character). In some embodiments, as illustrated in FIG. 13D , the user device displays the fourth recognition result (e.g., result (1328) "Thanks! Love") in the candidate display area simultaneously with the first recognition result (e.g., result "Thanks♥!").

In some embodiments, the user device allows the user to input regular text by drawing an emoji character. For example, if the user does not know how to spell the word "elephant," the user may optionally draw a stylized emoji character for "elephant" in the handwriting input area, and if the user device can accurately recognize the handwriting input as an emoji character for "elephant," the user device optionally also presents the word "elephant" in plain text as one of the recognition results displayed in the candidate display area. In another example, instead of writing the Chinese character "猫," the user may draw a stylized cat in the handwriting input area. If the user device identifies an emoji character for "cat" based on the handwriting input provided by the user, the user device optionally also presents the Chinese character "猫" for "cat" in the candidate recognition results along with the emoji character for "cat." By presenting plain text for recognized emoji characters, the user device provides an alternative way to input complex characters or words using a few stylized strokes commonly associated with well-known emoji characters. In some embodiments, the user device stores a dictionary that links emoji characters to their corresponding plain text (e.g., characters, words, phrases, symbols, etc.) in one or more preferred scripts or languages (e.g., English or Chinese).

In some embodiments, the user device recognizes an emoji character based on the visual similarity of the emoji character to an image generated from the handwriting input. In some embodiments, to enable recognition of emoji characters from the handwriting input, the handwriting recognition model used on the user device is trained using a training corpus that includes both handwriting samples corresponding to characters of a natural human language script and also handwriting samples corresponding to a set of artificially designed emoji characters. In some embodiments, emoji characters associated with the same semantic concept may have different appearances when used in mixed input with text from different natural languages. For example, an emoji character for the semantic concept of "Love" may be a "heart" emoji character when presented in plain text in one natural language (e.g., Japanese) and a "kiss" emoji character when presented in plain text in another natural language (e.g., English or French).

As described herein, when performing recognition of multi-character handwriting input, the handwriting input module performs segmentation of the currently accumulated handwriting input in the handwriting input area and divides the accumulated strokes into one or more recognition units. One of the parameters used to determine how to segment the handwriting input may be the way in which strokes are clustered in the handwriting input area and the distance between different clusters of strokes. People have different fonts. Some people tend to write very sparsely with a large distance between strokes or different parts of the same character, while others tend to write very densely with a very small distance between strokes or different characters. Even for the same user, due to imperfect planning, the handwriting characters may deviate from a balanced appearance and may be lopsided, stretched, or compressed in different ways. As described herein, the multi-script handwriting recognition model provides stroke order-independent recognition, so that the user can write characters or parts of characters out of order. As a result, it may be difficult to achieve spatial uniformity and balance between characters in handwriting input.

In some embodiments, the handwriting input model described herein provides a way for a user to inform the handwriting input module whether to merge two adjacent recognition units into a single recognition unit or to split the single recognition unit into two separate recognition units. With the user's assistance, the handwriting input module can revise the initial segmentation and produce the result intended by the user.

Figures 15a through 15j illustrate some exemplary user interfaces and processes through which a user provides predetermined pinch and expand gestures to modify recognition units identified by a user device.

As illustrated in FIGS. 15A and 15B , a user inputs multiple handwritten strokes (1502) (e.g., three strokes) in a handwritten input area (806) of a handwritten input interface (802). The user device identifies a single recognition unit based on the currently accumulated handwritten strokes (1502) and displays three candidate characters (1504, 1506, 1508) (e.g., “巾”, “中”, and “ ") was presented.

FIG. 15c shows that several additional strokes (1510) are additionally entered to the right of the initial handwritten strokes (1502) in the handwriting input area (606). The user device determines (e.g., based on the dimensions and spatial distributions of the plurality of strokes (1502, 1510)) that the strokes (1502) and the strokes (1510) should be considered as two separate recognition units. Based on the division of the recognition units, the user device provides the input images of the first and second recognition units to the handwriting recognition model, and obtains two sets of candidate characters. Thereafter, the user device generates a plurality of recognition results (e.g., 1512, 1514, 1516, 1518) based on different combinations of the recognized characters. Each recognition result includes a recognized character for the first recognition unit and a recognized character for the second recognition unit. As illustrated in FIG. 15c, each of the plurality of recognition results (1512, 1514, 1516, 1518) includes two recognized characters.

In this example, assume that the user actually intended the handwriting input to be recognized as a single character, but inadvertently placed too much space between the left portion (e.g., the left radical "巾") and the right portion (e.g., the right radical "冒") of the handwriting character (e.g., "帽"). Upon viewing the results presented in the candidate display area (806) (e.g., 1512, 1514, 1516, 1518), the user will notice that the user device incorrectly segmented the current handwriting input into two recognition units.

Segmentation may be based on objective criteria, but it would be undesirable for the user to delete the current handwriting input and re-enter the entire character with a smaller distance between the left and right portions.

Instead, as illustrated in FIG. 15d , the user uses a pinch gesture over two clusters of handwritten strokes (1502, 1510) to instruct the handwriting input module that the two recognition units identified by the handwriting input module should be merged into a single recognition unit. The pinch gesture is indicated by two contacts (1520, 1522) moving toward each other on the touch-sensitive surface.

FIG. 15e illustrates that, in response to the user's pinch gesture, the user device revises the segmentation of the currently accumulated handwriting input (e.g., strokes (1502, 1510)) and merges the handwriting strokes into a single recognition unit. As illustrated in FIG. 15e, the user device provides an input image based on the revised recognition unit to the handwriting recognition model, and generates three new candidate characters (1524, 1526, 1528) for the revised recognition unit (e.g., "帽", " ", and "搢") were acquired. In some embodiments, as illustrated in FIG. 15E , the user device optionally adjusts the rendering of the handwriting input within the handwriting input area (806) to reduce the distance between the left cluster and the right cluster of handwriting strokes. In some embodiments, the user device does not change the rendering of the handwriting input displayed in the handwriting input area (608) in response to the pinch gesture. In some embodiments, the user device distinguishes a pinch gesture from an input stroke based on two simultaneous contacts (as opposed to one single contact) detected in the handwriting input area (806).

As illustrated in FIG. 15f, the user inputs two additional strokes (1530) to the right of the previously input handwriting input (i.e., the strokes for the character “帽”). The user device determines that the newly input strokes (1530) are a new recognition unit, and recognizes a candidate character (e.g., “子”) for the newly identified recognition unit. Thereafter, the user device combines the newly identified character (e.g., “子”) with the candidate characters for the previously identified recognition unit, and presents a number of different recognition results (e.g., results (1532, 1534)) in the candidate display area (806).

As illustrated in FIG. 15g, following the handwritten strokes (1530), the user continues to write more strokes (1536) to the right of the strokes (1530) (e.g., three more strokes). Since the horizontal distance between the strokes (1530) and the strokes (1536) is very small, the user device determines that the strokes (1530) and the strokes (1536) belong to the same recognition unit, and provides the input image formed by the strokes (1530, 1536) to the handwriting recognition model. The handwriting recognition model identifies three different candidate characters for the revised recognition unit, and generates two revised recognition results (1538, 1540) for the currently accumulated handwriting input.

In this example, assume that the last two sets of strokes (1530, 1536) were in fact intended as two separate characters (e.g., "子" and "±"). After the user determines that the user device has erroneously combined the two sets of strokes (1530, 1536) into a single recognition unit, the user proceeds to provide an expansion gesture to notify the user device that the two sets of strokes (1530, 1536) should be split into two separate recognition units. As illustrated in FIG. 15h , the user makes two contacts (1542, 1544) around the strokes (1530, 1536), and then moves the two contacts away from each other, generally in a horizontal direction (i.e., along the default writing direction).

FIG. 15i illustrates that, in response to the user's expansion gesture, the user device revises a previous segmentation of the currently accumulated handwriting input and assigns strokes (1530) and strokes (1536) into two consecutive recognition units. Based on the input images generated for the two separate recognition units, the user device identifies one or more candidate characters for a first recognition unit based on the strokes (1530) and one or more candidate characters for a second recognition unit based on the strokes (1536). The user device then generates two new recognition results (1546, 1548) based on different combinations of the recognized characters. In some embodiments, the user device optionally modifies the rendering of the strokes (1536, 1536) to reflect the division of the previously identified recognition units.

As illustrated in FIGS. 15j and 15k , a user has selected one of the candidate recognition results displayed in the candidate display area (806) (as indicated by the contact (1550)), and the selected recognition result (e.g., result (1548)) has been entered into the text input area (808) of the user interface. After the selected recognition result has been entered into the text input area (808), both the candidate display area (806) and the handwriting input area (804) are cleared and ready to display subsequent user input.

FIGS. 16A and 16B are flowcharts of an exemplary process (1600) in which a user notifies a handwriting input module of how to segment or revise an existing segment of a current handwriting input using a predetermined gesture (e.g., a pinch gesture and/or an expand gesture). FIGS. 15J and 15K provide examples of an exemplary process (1600) according to some embodiments.

In some embodiments, the user device receives handwriting input from the user (1602). The handwriting input comprises a plurality of handwriting strokes provided on a touch-sensitive surface coupled to the device. In some embodiments, the user device renders the plurality of handwriting strokes in real time (1604) in a handwriting input area of a handwriting input interface (e.g., handwriting input area (806) of FIGS. 15A-15K ). For example, as illustrated in FIGS. 15D and 15H , the user device receives one of a pinch gesture input and an expand gesture input for the plurality of handwriting strokes.

In some embodiments, for example, as illustrated in FIGS. 15C to 15E , upon receiving a pinch gesture input, the user device generates a first recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as a single recognition unit (1606).

In some embodiments, for example, as illustrated in FIGS. 15g-15i , upon receiving an expansion gesture input, the user device generates a second recognition result based on the plurality of handwritten strokes by processing the plurality of handwritten strokes as two separate recognition units separated by the expansion gesture input (1608).

In some embodiments, for example, as illustrated in FIGS. 15E and 15I , the user device, when generating individual recognition results of the first recognition result and the second recognition result, displays the generated recognition results in the candidate display area of the handwriting input interface.

In some embodiments, a pinch gesture input comprises two simultaneous contacts moving toward each other in an area occupied by multiple handwritten strokes on the touch-sensitive surface. In some embodiments, an expand gesture input comprises two simultaneous contacts moving away from each other in an area occupied by multiple handwritten strokes on the touch-sensitive surface.

In some embodiments, the user device identifies two adjacent recognition units from the plurality of handwritten strokes (e.g., 1614). For example, as illustrated in FIG. 15C , the user device displays (1616) an initial recognition result in the candidate display area that includes individual characters recognized from the two adjacent recognition units (e.g., results (1512, 1514, 1516, 1518) of FIG. 15C ). In some embodiments, when displaying a first recognition result (e.g., results (1524, 1526, or 1528) of FIG. 15E ) in response to the pinch gesture, the user device replaces the initial recognition result with the first recognition result in the candidate display area (1618). In some embodiments, as illustrated in FIG. 15D , the user device receives (1620) a pinch gesture input while the initial recognition result is displayed in the candidate display area. In some embodiments, for example, as illustrated in FIG. 15E , the user device, in response to a pinch gesture input, re-renders a plurality of handwritten strokes (1622) to reduce a distance between two adjacent recognition units in the handwriting input area.

In some embodiments, the user device identifies a single recognition unit from a plurality of handwritten strokes (1624). The user device displays, in the candidate display area, a character recognized from the single recognition unit (e.g., " ""祉" is displayed (1626) an initial recognition result (e.g., result (1538 or 1540) of FIG. 15g) including the word "祉". In some embodiments, when displaying a second recognition result (e.g., result (1546 or 1548) of FIG. 15i) in response to the expansion gesture, as illustrated in FIGS. 15h and 15i , the user device replaces the initial recognition result (e.g., result (1538 or 1540)) with the second recognition result (e.g., result (1546 or 1548)) in the candidate display area (1628). In some embodiments, as illustrated in FIG. 15h , the user device receives an expansion gesture input (1630) while the initial recognition result is displayed in the candidate display area. In some embodiments, as illustrated in FIGS. 15h and 15i , the user device, in response to the expansion gesture input, displays a first recognition result in the handwriting input area. Re-render a plurality of handwritten strokes to increase the distance between a first subset of strokes assigned to a unit and a second subset of handwritten strokes assigned to a second recognition unit (1632).

In some embodiments, the user optionally provides a pinch gesture to notify the user device to treat the multiple strokes as a single recognition unit, immediately after the user provides the strokes and notices that the strokes may be too spread out for accurate segmentation based on standard segmentation processes. The user device can distinguish a pinch gesture from a regular stroke based on two simultaneous contacts present during the pinch gesture. Similarly, in some embodiments, the user optionally provides a dilate gesture to notify the user device to treat the multiple strokes as two separate recognition units, immediately after the user provides the strokes and notices that the strokes may be too close together for accurate segmentation based on standard segmentation processes. The user device can distinguish a dilate gesture from a regular stroke based on two simultaneous contacts present during the pinch gesture.

In some embodiments, the direction of movement of the pinch or expand gesture is optionally used to provide additional guidance as to how to segment strokes under the gesture. For example, if multi-line handwriting input is enabled for the handwriting input area, a pinch gesture with two contacts moving in a vertical direction may notify the handwriting input module to merge two recognition units identified in two adjacent lines into a single recognition unit (e.g., as an upper radical and a lower radical). Similarly, an expand gesture with two contacts moving in a vertical direction may notify the handwriting input module to split a single recognition unit into two recognition units in two adjacent lines. In some embodiments, the pinch and expand gestures may also provide guidance for segmenting sub-portions of a character input, such as different portions of a compound character (e.g., the upper, lower, left, or right portions). , etc.) can be provided to merge two sub-components or split a single component. This is particularly helpful in recognizing complex compound Chinese characters, since users tend to lose their correct proportions and balance when they input complex compound characters by hand. Being able to adjust the proportions and balance of the handwriting input, for example, by pinch and expand gestures, after the handwriting input is completed is particularly helpful in allowing the user to input the correct character without having to make multiple attempts to get the proportions and balance correct.

As described herein, the handwriting input module allows a user to input multi-character handwriting input, and allows non-sequential strokes for multi-character handwriting input within a character, across multiple characters, and even across multiple phrases, sentences, and/or lines in the handwriting input area. In some embodiments, the handwriting input module also provides for character-by-character deletion in the handwriting input area, wherein the order of the character deletions is independent of when the strokes for each character were provided in the handwriting input area and in the reverse writing direction. In some embodiments, the deletion of each recognition unit (e.g., a character or radical) in the handwriting input area is optionally performed stroke-by-stroke, wherein the strokes are deleted in the reverse temporal order in which the strokes were provided within the recognition unit. FIGS. 17A-17H illustrate exemplary user interfaces for responding to a deletion input from a user and providing character-by-character deletion in multi-character handwriting input.

As illustrated in FIG. 17a, a user has provided a plurality of handwritten strokes (1702) in a handwriting input area (804) of a handwriting input user interface (802). Based on the currently accumulated strokes (1702), the user device presents three recognition results (e.g., results (1704, 1706, 1708)) in a candidate display area (806). As illustrated in FIG. 17b, the user has provided an additional plurality of strokes (1710) in the handwriting input area (806). The user device recognizes the three new output characters and replaces the three previous recognition results (1704, 1706, 1708) with the three new recognition results (1712, 1714, 1716). In some embodiments, even if the user device has identified two separate recognition units from the current handwriting input (e.g., strokes (1702) and strokes (1710)), as illustrated in FIG. 17b , the cluster of strokes (1710) does not correspond well to any known characters in the repertoire of the handwriting recognition module. As a result, the candidate characters identified for the recognition unit that includes strokes (1710) (e.g., " ", "凶") all have recognition confidences below a predetermined threshold. In some embodiments, the user device presents a partial recognition result (e.g., result (1712)) that includes only the candidate character for the first recognition unit (e.g., "日") and no candidate character for the second recognition unit in the candidate display area (806). In some embodiments, the user device additionally displays a full recognition result (e.g., result (1714 or 1716)) that includes the candidate characters for both recognition results, regardless of whether the recognition confidences pass the predetermined threshold. Providing a partial recognition result notifies the user that some portion of the handwriting input needs to be revised. Additionally, the user may also elect to first input the accurately recognized portion of the handwriting input and then re-enter the incorrectly recognized portion.

FIG. 17c illustrates that the user has continued to provide an additional handwritten stroke (1718) to the left of the strokes (1710). Based on the relative position and distance of the stroke (1718), the user device determines that the newly added stroke belongs to the same recognition unit as the cluster of handwritten strokes (1702). Based on the revised recognition units, a new character (e.g., " ") and generates a set of new recognition results (1720, 1722, 1724). Again, since none of the candidate characters identified for the strokes (1710) meet the predetermined confidence threshold, the first recognition result (1720) is a partial recognition result.

FIG. 17d illustrates that the user has now input a plurality of new strokes (1726) between strokes (1702) and strokes (1710). The user device assigns the newly input strokes (1726) to the same recognition unit as strokes (1710). Now, the user has input two Chinese characters (e.g., " ") has completed the input of all handwriting strokes, and the accurate recognition result (1728) is displayed in the candidate display area (806).

FIG. 27e illustrates that a user has entered an initial portion of a delete input, for example, by making light contact (1730) on a delete button (1732). If the user maintains contact with the delete button (1732), the user can delete the current handwriting input character by character (or recognition unit by recognition unit). Deletion is not performed on all handwriting input at the same time.

In some embodiments, as illustrated in FIG. 17e, when the user's finger initially touches the delete button (1732) on the touch-sensitive screen, the last recognition unit (e.g., the character ") in the default writing direction (e.g., from left to right) is "Recognition unit) is visually highlighted (e.g., highlighted by a border (1734) or a brightened background, etc.) relative to other recognition unit(s) displayed simultaneously within the handwriting input area (804).

In some embodiments, when the user device detects that the user has maintained contact (1730) on the delete button (1732) for longer than a threshold duration, as illustrated in FIG. 17F , the user device removes the highlighted recognition unit (e.g., within the box (1734)) from the handwriting input area (806). Additionally, as illustrated in FIG. 17F , the user device also revises the recognition results displayed in the candidate display area (608) to delete any output characters generated based on the deleted recognition unit.

FIG. 17f shows that when the user continues to maintain contact (1730) on the delete button (1732) after the last recognition unit (e.g., the recognition unit for the character "腦") in the handwriting input area (806) is deleted, an adjacent recognition unit (e.g., the recognition unit for the character "腦") to the deleted recognition unit is deleted. " recognition unit) to be deleted. As illustrated in FIG. 17F , the remaining recognition units are visually highlighted (e.g., within box (1736)) and ready to be deleted. In some embodiments, the visual highlighting of the recognition unit provides a preview of the recognition unit that will be deleted if the user continues to maintain contact with the delete button. If the user discontinues contact with the delete button before the threshold duration is reached, the visual highlight is removed from the last recognition unit, and the recognition unit is not deleted. As will be appreciated by those skilled in the art, the duration of contact is reset each time a recognition unit is deleted. Additionally, in some embodiments, the intensity of the contact (e.g., the pressure with which the user makes contact (1730) with the touch-sensitive screen) is optionally used to adjust the threshold duration to confirm the user's intent to delete the currently highlighted recognition unit. FIGS. 17F and 17G illustrate an example where the user discontinues contact (1730) on the delete button (1732) before the threshold duration is reached, and the character " "is preserved in the handwriting input area (806). As illustrated in FIGS. 17g and 17h, when the user selects a first recognition result (e.g., result (1738)) for the recognition unit (e.g., as indicated by contact (1740)), the text of the first recognition result (1738) is input into the text input area (808).

FIGS. 18A and 18B are flowcharts of an exemplary process (1800) by which a user device provides character-by-character deletion from multi-character handwriting input. In some embodiments, the deletion of the handwriting input is performed before characters recognized from the handwriting input are identified and entered into a text input area of the user interface. In some embodiments, the deletion of characters from the handwriting input proceeds in the reverse spatial order of the recognition units identified from the handwriting input, and is independent of the temporal sequence in which the recognition units were formed. FIGS. 17A through 17H illustrate an exemplary process (1800) according to some embodiments.

As illustrated in FIG. 18a , in an exemplary process (1800), a user device receives a handwriting input from a user (1802), wherein the handwriting input comprises a plurality of handwritten strokes provided in a handwriting input area of a handwriting input interface (e.g., area (804) of FIG. 17d ). The user device identifies a plurality of recognition units from the plurality of handwritten strokes (1804), each recognition unit comprising a distinct subset of the plurality of handwritten strokes. For example, as illustrated in FIG. 17d , a first recognition unit comprises strokes (1702, 1718), and a second recognition unit comprises strokes (1710, 1726). The user device generates a multi-character recognition result (e.g., result (1728) of FIG. 17d ) comprising individual characters recognized from the plurality of recognition units (1806). In some embodiments, the user device displays a multi-character recognition result (e.g., result (1728) of FIG. 17d ) in a candidate display area of the handwriting input interface. In some embodiments, for example, as illustrated in FIG. 17e , while the multi-character recognition result is displayed in the candidate display area, the user device receives (1810) a delete input (e.g., contact (1730) on a delete button (1732)) from the user. In some embodiments, for example, as illustrated in FIGS. 17e and 17f , in response to receiving the delete input, the user device displays a last character (e.g., a space sequence " "The last character that appears in " ") is removed (1812).

In some embodiments, for example, as illustrated in FIGS. 17A-17D , the user device renders in real time a plurality of handwritten strokes in a handwriting input area of a handwriting input interface as the plurality of handwritten strokes are provided by the user (1814). In some embodiments, in response to receiving a delete input, the user device removes (1816) from the handwriting input area (e.g., the handwriting input area (804) of FIG. 17E ) a discrete subset of the plurality of handwritten strokes corresponding to a last recognition unit (e.g., a recognition unit comprising strokes (1726, 1710)) in a spatial sequence formed by the plurality of recognition units in the handwriting input area. The last recognition unit is the last character (e.g., the character " ) in the multi-character recognition result (e.g., the result (1728) of FIG. 17E ). ") corresponds to.

In some embodiments, the last recognition unit does not include the temporally last handwritten stroke among the multiple handwritten strokes provided by the user (1818). For example, if the user provided the stroke (1718) after he or she provided the strokes (1726, 1710), the last recognition unit including the strokes (1726, 1710) will still be deleted first.

In some embodiments, for example, as illustrated in FIG. 17E , in response to receiving an initial portion of a delete input, the user device visually distinguishes a last recognition unit from other recognition units identified in the handwriting input area (1820). In some embodiments, the initial portion of the delete input is an initial contact detected on a delete button in the handwriting input interface, and the delete input is detected when the initial contact lasts longer than a predetermined threshold amount of time (1822).

In some embodiments, the last recognition unit corresponds to a handwritten Chinese character. In some embodiments, the handwriting input is written in cursive script. In some embodiments, the handwriting input corresponds to multiple Chinese characters written in cursive script. In some embodiments, at least one of the handwritten strokes is divided into two adjacent recognition units among the plurality of recognition units. For example, sometimes, a user may use a long stroke that is connected to multiple characters, and in such a case, the segmentation module of the handwriting input module optionally divides the long stroke into multiple recognition units. When deletion of the handwriting input is performed on a character-by-character basis (or a recognition unit-by-recognition unit basis), only a segment of the long stroke (e.g., a segment within the corresponding recognition unit) is deleted at a time.

In some embodiments, the delete input is a sustained contact on a delete button provided in a handwriting input interface, and removing an individual subset of the plurality of handwriting strokes further comprises removing the subset of handwriting strokes from the handwriting input area at a last recognition unit, stroke by stroke, in a reverse chronological order provided by the user (1824).

In some embodiments, for example, as illustrated in FIGS. 17B and 17C , the user device generates (1826) a partial recognition result comprising a subset of individual characters recognized from the plurality of recognition units, wherein each of the subsets of individual characters satisfies a predetermined confidence threshold. In some embodiments, the user device displays (1828) the partial recognition result (e.g., result (1712) of FIG. 17B and result (1720) of FIG. 17C ) concurrently with the multi-character recognition result (e.g., results (1714, 1722)) in a candidate display area of the handwriting input interface.

In some embodiments, the partial recognition result does not include at least the last character in the multi-character recognition result. In some embodiments, the partial recognition result does not include at least the initial character in the multi-character recognition result. In some embodiments, the partial recognition result does not include at least the middle character in the multi-character recognition result.

In some embodiments, the minimum unit of deletion is a radical, and handwriting inputs are deleted one radical at a time whenever the radical becomes the last recognition unit in the handwriting inputs still remaining in the handwriting input area.

As described herein, in some embodiments, the user device provides both a horizontal writing mode and a vertical writing mode. In some embodiments, the user device allows the user to enter text in both, or either, a left-to-right writing direction and a right-to-left writing direction in the horizontal writing mode. In some embodiments, the user device allows the user to enter text in both, or either, a top-down writing direction and a bottom-up writing direction in the vertical writing mode. In some embodiments, the user device provides various affordances (e.g., writing mode or writing direction buttons) on the user interface to invoke individual writing modes and/or writing directions for current handwriting inputs. In some embodiments, the text input direction in the text input area is the same as the handwriting input direction in the handwriting input direction by default. In some embodiments, the user device allows the user to manually set the input direction in the text input area and the writing direction in the handwriting input area. In some embodiments, the text display direction in the candidate display area is the same as the handwriting input direction in the handwriting input area by default. In some embodiments, the user device allows the user to manually set the text display direction in the text input area independent of the handwriting input direction in the handwriting input area. In some embodiments, the user device associates a writing mode and/or a writing direction of the handwriting input interface with a corresponding device orientation, and a change in the device orientation automatically causes a change in the writing mode and/or writing direction. In some embodiments, a change in the writing direction automatically causes the entry of a top-level recognition result to be input into the text input area.

Figures 19a through 19f illustrate exemplary user interfaces of a user device providing both a horizontal input mode and a vertical input mode.

FIG. 19A illustrates a user device in a horizontal input mode. In some embodiments, as illustrated in FIG. 19A, the horizontal input mode is provided when the user device is in a landscape orientation. In some embodiments, the horizontal input mode is optionally provided in association with when the device is operated in a portrait orientation. In different applications, the association between device orientation and writing mode may be different.

In the horizontal input mode, the user can provide handwritten characters in a horizontal writing direction (e.g., having a default writing direction that runs from left to right or a default writing direction that runs from right to left). In the horizontal input mode, the user device performs a process of segmenting the handwritten input into one or more recognition units along the horizontal writing direction.

In some embodiments, the user device allows only a single line input in the handwriting input area. In some embodiments, as illustrated in FIG. 19A , the user device allows multi-line input (input of two lines) in the handwriting input area. In FIG. 19A , the user has provided multiple handwritten strokes in multiple rows in the handwriting input area (806). Based on the sequence in which the user provided the multiple handwritten strokes and the relative positions of the multiple handwritten strokes and the distances between the multiple handwritten strokes, the user device determines that the user has input two lines of characters. After splitting the handwriting input into two separate lines, the device determines the recognition unit(s) within each line.

As illustrated in FIG. 19A , the user device has recognized individual characters for each recognition unit identified in the current handwriting input (1902) and generated multiple recognition results (1904 and 1906). As further illustrated in FIG. 19A , in some embodiments, if an output character (e.g., the character “I”) for a particular set of recognition units (e.g., the recognition unit formed by the initial stroke) is weak, the user device optionally generates a partial recognition result (e.g., result (1906)) that represents only those output characters having sufficient recognition confidence. In some embodiments, the user may notice from the partial recognition result (1906) that the first stroke may be revised for the recognition model or may be individually deleted or rewritten to generate an accurate recognition result. In this particular example, no editing of the first recognition unit (1904) is necessary because the first recognition unit represents the desired recognition result for the first recognition unit.

In this example, as illustrated in FIGS. 19A and 19B , the user rotates the device to a portrait orientation (e.g., as illustrated in FIG. 19B ). As illustrated in FIG. 19B , in response to the change in device orientation, the handwriting input interface changes from a horizontal input mode to a vertical input mode. In the vertical input mode, the layout of the handwriting input area (804), the candidate display area (806), and the text input area (808) may be different than that illustrated in the horizontal input mode. The particular layouts of the horizontal and vertical input modes may vary to suit different device geometries and application requirements. In some embodiments, with the rotation of the device orientation and the change in input mode, the user device automatically inputs the top result (e.g., result (1904)) as a text input (1910) into the text input area (808). The orientation and position of the cursor (1912) also reflect the change in input mode and direction of entry.

In some embodiments, a change in input mode is optionally triggered by the user touching a particular input mode selection affordance (1908). In some embodiments, the input mode selection affordance is a graphical user interface element that also indicates a current writing mode, a current writing direction, and/or a current paragraph direction. In some embodiments, the input mode selection affordance can cycle through all available input modes and writing directions provided by the handwriting input interface (802). As illustrated in FIG. 19A , the affordance (1908) indicates that the current input mode is a horizontal input mode, that the writing direction is left to right, and that the paragraph direction is top to bottom. In FIG. 19B , the affordance (1908) indicates that the current input mode is a vertical input mode, that the writing direction is top to bottom, and that the paragraph direction is right to left. According to various embodiments, other combinations of writing directions and paragraph directions are possible.

As illustrated in Figure 19c, the user may input a plurality of new strokes (1914) (e.g., two Chinese characters ") in the handwriting input area (804) in the vertical input mode. "Handwriting strokes for" are entered. The handwriting input is entered in a vertical writing direction. The user device divides the handwriting input into two recognition units in the vertical direction and displays two recognition results (1916, 1918), each of which includes two recognized characters laid out in the vertical direction.

FIGS. 19c and 19d illustrate that when a user selects a displayed recognition result (e.g., result (1916)), the selected recognition result is input vertically into the text input area (808).

Figures 19E and 19F illustrate that the user has entered additional lines of handwriting input (1920) in a vertical writing direction. The lines continue from left to right following the paragraph direction of traditional Chinese writing. In some embodiments, the candidate display area (806) also presents recognition results (e.g., results (1922, 1924)) in the same writing direction and paragraph direction as for the handwriting input area. In some embodiments, different writing directions and paragraph directions may be provided by default depending on the primary language associated with the user device, or the language of a soft keyboard installed on the user device (e.g., Arabic, Chinese, Japanese, English, etc.).

FIGS. 19E and 19F also illustrate that when a user selects a recognition result (e.g., result (1922)), the text of the selected recognition result is entered into the text input area (808). Accordingly, as illustrated in FIG. 19F, the current text input in the text input area (808) includes both text entered in a horizontal mode in a left-to-right writing direction, and text entered in a vertical mode in a top-down writing direction. The paragraph direction for the horizontal text is top-down, while the paragraph direction for the vertical text is right-to-left.

In some embodiments, the user device allows the user to separately establish preferred writing directions and paragraph directions for each of the handwriting input area (804), the candidate display area (806), and the text input area (808). In some embodiments, the user device allows the user to establish preferred writing directions and paragraph directions for each of the handwriting input area (804), the candidate display area (806), and the text input area (808), associated with each device orientation.

FIGS. 20A through 20C are flowcharts of an exemplary process (2000) for changing a text input direction and a handwriting input direction of a user interface. FIGS. 19A through 19F illustrate a process (2000) according to some embodiments.

In some embodiments, the user device determines an orientation of the device (2002). The orientation of the device and changes in device orientation may be detected by an accelerometer and/or other orientation-sensitive element within the user device. In some embodiments, the user device provides a handwriting input interface on the device in a horizontal input mode with the device in a first orientation (2004). Individual lines of handwriting input entered in the horizontal input mode are segmented into one or more individual recognition units along a horizontal writing direction. In some embodiments, the device provides a handwriting input interface on the device in a vertical input mode with the device in a second orientation (2006). Individual lines of handwriting input entered in the vertical input mode are segmented into one or more individual recognition units along a vertical writing direction.

In some embodiments, while operating in a horizontal input mode (2008): the device detects a change in device orientation from a first orientation to a second orientation (2010). In some embodiments, in response to the change in device orientation, the device switches from the horizontal input mode to a vertical input mode (2012). This is illustrated, for example, in FIGS. 19A and 19B . In some embodiments, while operating in a vertical input mode (2014): the user device detects a change in device orientation from a second orientation to a first orientation (2016). In some embodiments, in response to the change in device orientation, the user device switches from the vertical input mode to the horizontal input mode (2018). In some embodiments, the association between device orientation and input mode can be the opposite of what is described above.

In some embodiments, while operating in a horizontal input mode (2020): the user device receives a first multi-word handwriting input from the user (2022). In response to the first multi-word handwriting input, the user device presents a first multi-word recognition result in a candidate display area of the handwriting input interface along the horizontal writing direction (2024). This is illustrated, for example, in FIG. 19A . In some embodiments, while operating in a vertical input mode (2026): the user device receives a second multi-word handwriting input from the user (2028). In response to the second multi-word handwriting input, the user device presents a second multi-word recognition result in a candidate display area along the vertical writing direction (2030). This is illustrated, for example, in FIGS. 19C and 19E .

In some embodiments, for example, as illustrated in FIGS. 19A and 19B , the user device receives a first user input selecting a first multi-word recognition result (2032), where the selection is implicitly made by an input for changing an input direction (e.g., rotation of the device or selection of an affordance (1908)). For example, as illustrated in FIGS. 19C or 19E , the user device receives a second user input selecting a second multi-word recognition result (2034). The user device simultaneously displays individual texts of the first multi-word recognition result and the second multi-word recognition result in a text input area of the handwriting input interface (2036), where the individual texts of the first multi-word recognition result are displayed along a horizontal writing direction and the individual texts of the second multi-word recognition result are displayed along a vertical writing direction. This is illustrated, for example, in the text input area (808) of FIG. 19F .

In some embodiments, the handwriting input area allows multiple lines of handwriting input in a horizontal writing direction and has a default top-down paragraph direction. In some embodiments, the horizontal writing direction is left-to-right. In some embodiments, the horizontal writing direction is right-to-left. In some embodiments, the handwriting input area allows multiple lines of handwriting input in a vertical writing direction and has a default left-to-right paragraph direction. In some embodiments, the handwriting input area allows multiple lines of handwriting input in a vertical writing direction and has a default right-to-left paragraph direction. In some embodiments, the vertical writing direction is top-to-down. In some embodiments, the first orientation is landscape by default and the second orientation is portrait by default. In some embodiments, the user device provides a separate affordance in the handwriting input interface to manually switch between a horizontal input mode and a vertical input mode, regardless of the device orientation. In some embodiments, the user device provides a separate affordance in the handwriting input interface for manually switching between two alternative writing orientations. In some embodiments, the user device provides a separate affordance in the handwriting input interface for manually switching between two alternative paragraph orientations. In some embodiments, the affordance is a toggle button that, when invoked one or more times consecutively, cycles through each of the available combinations of input and paragraph orientations.

In some embodiments, the user device receives handwriting input from the user (2038). The handwriting input includes a plurality of handwriting strokes provided in a handwriting input area of the handwriting input interface. In response to the handwriting input, the user device displays one or more recognition results in a candidate display area of the handwriting input interface (2040). While the one or more recognition results are displayed in the candidate display area, the user device detects user input to switch from the current handwriting input mode to an alternative handwriting input mode (2042). In response to the user input (2044): the user device switches from the current handwriting input mode to the alternative handwriting input mode (2046). In some embodiments, the user device clears the handwriting input from the handwriting input area (2048). In some embodiments, the user device automatically inputs a top-ranked recognition result from the one or more recognition results displayed in the candidate display area into a text input area of the handwriting input interface (2050). This is illustrated, for example, in FIGS. 19A and 19B , where the current handwriting input mode is a horizontal input mode and the alternative handwriting input mode is a vertical input mode. In some embodiments, the current handwriting input mode is a vertical input mode and the alternative handwriting input mode is a horizontal input mode. In some embodiments, the current handwriting input mode and the alternative handwriting input mode are modes in which any two different handwriting input orientations or paragraph orientations are provided. In some embodiments, the user input is a rotation of the device from the current orientation to the other orientation (2052). In some embodiments, the user input is an invocation of an affordance to manually switch the current handwriting input mode to the alternative handwriting input mode.

As described herein, the handwriting input module allows a user to input handwriting strokes and/or characters in any chronological order. Therefore, it is advantageous to delete individual handwriting characters in a multi-character handwriting input and re-enter handwriting characters that are identical or different from the deleted characters at the same location, as this will help the user revise a long handwriting input without having to delete the entire handwriting input.

FIGS. 20A-20H illustrate exemplary user interfaces for visually highlighting and/or deleting a recognition unit identified in a currently accumulated plurality of handwriting strokes in a handwriting input area. Allowing a user to individually select, view, and delete any one of a plurality of recognition units identified in a plurality of inputs is particularly useful when multi-character and even multi-line handwriting input is allowed by the user device. Allowing a user to delete a particular recognition unit at the beginning or middle of a handwriting input allows the user to edit a long input without requiring the user to delete all recognition units positioned after an undesirable recognition unit.

As illustrated in FIGS. 21A to 21C , a user has provided a plurality of handwritten strokes (e.g., strokes (2102, 2104, 2106)) in a handwriting input area (804) of a handwriting input user interface (802). While the user continues to provide additional strokes into the handwriting input area (804), the user device updates recognition units identified from the current accumulated handwriting input in the handwriting input area, and revises recognition results according to output characters recognized from the updated recognition units. As illustrated in FIG. 20C , the user device has identified two recognition units from the current handwriting input, and has presented three recognition results (e.g., 2108, 2010, 2112), each of which includes two Chinese characters.

In this example, after the user has entered two handwritten characters, the user notices that the first recognition unit is entered incorrectly, and as a result, the user device fails to identify and present the desired recognition result in the candidate display area.

In some embodiments, when a user provides a tap gesture (e.g., a touch followed by an immediate lift at the same location) on the touch-sensitive display, the user device interprets the tap gesture as an input, causing visual highlighting of individual recognition units currently identified in the handwriting input area. In some embodiments, another predetermined gesture (e.g., a multi-finger swipe gesture over the handwriting input area) is used to cause the user device to highlight individual recognition units in the handwriting input area (804). A tap gesture is sometimes desirable because it is relatively easy to distinguish from a handwriting stroke, which typically involves movement of contact within the handwriting input area (804) and sustained contact of a longer duration. A multi-touch gesture is sometimes desirable because it is relatively easy to distinguish from a handwriting stroke, which typically involves a single contact within the handwriting input area (804). In some embodiments, the user device provides an affordance (2112) in the user interface that can be invoked by the user (e.g., via contact (2114)) to visually highlight individual recognition units (e.g., as illustrated by boxes (2108, 2110)). In some embodiments, the affordance is desirable when there is sufficient screen space to accommodate such an affordance. In some embodiments, the affordance can be invoked multiple times in succession by the user, causing the user device to visually highlight the recognition unit(s) identified along different segmentation chains in the segmentation grid and de-highlight them when all segmentation chains are revealed.

As illustrated in FIG. 21d , when the user provides a gesture necessary to highlight individual recognition units in the handwriting input area (804), the user device additionally displays individual delete affordances (e.g., small delete buttons (2116, 2118)) above each highlighted recognition unit. FIGS. 21e and 21f illustrate that when the user touches (e.g., via contact (2120)) the delete affordance of an individual recognition unit (e.g., the delete button (2116) for the first recognition unit within box (2118)), the individual recognition unit (e.g., within box (2118)) is removed from the handwriting input area (804). In this particular example, the deleted recognition unit is not the last recognition unit entered temporally, nor is it the last recognition unit spatially along the writing direction. In other words, the user can delete any recognition unit regardless of when and where the recognition unit was provided in the handwriting input area. FIG. 21f illustrates that, in response to the deletion of the first recognition unit in the handwriting input area, the user device also updates the recognition results displayed in the candidate display area (806). As illustrated in FIG. 21f, the user device also deleted the candidate character corresponding to the deleted recognition unit from the recognition results. As a result, a new recognition result (2120) appears in the candidate display area (806).

As illustrated in FIGS. 21g and 21h , after the first recognition unit is removed from the handwriting input interface (804), the user has provided a plurality of new handwriting strokes (2122) in the area previously occupied by the deleted recognition unit. The user device has re-segmented the currently accumulated handwriting input in the handwriting input area (804). Based on the recognition units identified from the handwriting input, the user device has re-generated recognition results (e.g., results (2124), 2126)) in the candidate display area (806). FIGS. 21g and 21h illustrate that the user selects one of the recognition results (e.g., result (2124)) (e.g., via contact (2128)) and inputs text of the selected recognition result into the text input area (808).

Figures 22A and 22B are flowcharts for an exemplary process (2200) by which individual recognition units identified in a current handwriting input are visually presented and can be individually deleted, regardless of the time order in which the recognition units were formed. Figures 21A through 21H illustrate a process (2200) according to some embodiments.

In an exemplary process (2200), a user device receives handwriting input from a user (2202). The handwriting input comprises a plurality of handwriting strokes provided on a touch-sensitive surface coupled to the device. In some embodiments, the user device renders the plurality of handwriting strokes in a handwriting input area of a handwriting input interface (e.g., handwriting input area (804)) (2204). In some embodiments, the user device segments the plurality of handwriting strokes into two or more recognition units (2206), each recognition unit comprising a distinct subset of the plurality of handwriting strokes.

In some embodiments, the user device receives an edit request from the user (2208). In some embodiments, the edit request is a contact detected over a predetermined affordance provided in the handwriting input interface (e.g., affordance 2112 of FIG. 21D ) (2210). In some embodiments, the edit request is a tap gesture detected over a predetermined area in the handwriting input interface (2212). In some embodiments, the predetermined area is within the handwriting input area of the handwriting input interface. In some embodiments, the predetermined area is outside the handwriting input area of the handwriting input interface. In some embodiments, another predetermined gesture outside the handwriting input area (e.g., a cross gesture, a horizontal swipe gesture, a vertical swipe gesture, an oblique swipe gesture) can be used as the edit request. Gestures outside the handwriting input area can be easily distinguished from handwritten strokes because they are provided outside the handwriting input area.

In some embodiments, in response to an edit request, the user device visually distinguishes (2214) two or more recognition units in the handwriting input area, for example, using boxes (2108, 2110) of FIG. 21D . In some embodiments, visually distinguishing the two or more recognition units further comprises highlighting individual boundaries between the two or more recognition units in the handwriting input area (2216). In various embodiments, different ways of visually distinguishing recognition units identified in the current handwriting input may be utilized.

In some embodiments, the user device provides means for individually deleting each of the two or more recognition units from the handwriting input area (2218). In some embodiments, the means for individually deleting each of the two or more recognition units is an individual delete button displayed proximate each recognition unit, as illustrated by, for example, the delete buttons (2116, 2118) of FIG. 21D . In some embodiments, the means for individually deleting each of the two or more recognition units is means for detecting a predetermined delete gesture input for each recognition unit. In some embodiments, the user device does not visibly display an individual delete affordance for the highlighted recognition units. Instead, in some embodiments, the user is permitted to use the delete gesture to delete the individual recognition unit under the delete gesture. In some embodiments, because the user device is displaying the recognition units in a visually highlighted manner, the user device does not allow additional handwriting strokes in the handwriting input area. Instead, a predetermined or arbitrary gesture detected over a visually highlighted recognition unit will cause the user device to remove the recognition unit from the handwriting input area, thereby revising the recognition results displayed in the candidate display area. In some embodiments, a tap gesture causes the user device to visually highlight individual recognition units identified in the handwriting recognition area, and the user can then use a delete button to delete the individual recognition units, one by one, in the reverse writing direction.

In some embodiments, for example, as illustrated in FIG. 21e , the user device receives a delete input from the user and via provided means to individually delete a first recognition unit of the two or more recognition units from the handwriting input area (2224). For example, as illustrated in FIG. 21f , the user device, in response to the delete input, removes (2226) a respective subset of handwritten strokes in the first recognition unit from the handwriting input area. In some embodiments, the first recognition unit is a spatially initial recognition unit of the two or more recognition units. In some embodiments, for example, as illustrated in FIGS. 21e and 21f , the first recognition unit is a spatially intermediate recognition unit of the two or more recognition units. In some embodiments, the first recognition unit is a spatially final recognition unit of the two or more recognition units.

In some embodiments, the user device generates a segmentation grid (2228) from the plurality of handwritten strokes, wherein the segmentation grid includes a plurality of alternative segmentation chains, each of which represents a distinct set of recognition units identified from the plurality of handwritten strokes. For example, FIG. 21G illustrates recognition results (2024, 2026), where the recognition result (2024) is generated from one segmentation chain having two recognition units, and the recognition result (2026) is generated from another segmentation chain having three recognition units. In some embodiments, the user device receives two or more consecutive edit requests from the user (2230). For example, the two or more consecutive edit requests may be multiple consecutive taps on the affordance (2112) of FIG. 21G. In some embodiments, in response to each of the two or more consecutive edit requests, the user device visually distinguishes the distinct set of recognition units in the handwriting input area from other segmentation chains among the plurality of alternative segmentation chains (2232). For example, in response to a first tap gesture, two recognition units (e.g., for the characters "帽" and "子", respectively) are highlighted in the handwriting input area (804), and in response to a second tap gesture, three recognition units (e.g., for the characters "巾", "冒", and "子", respectively) are highlighted. In some embodiments, in response to a third tap gesture, the visual highlighting is selectively removed from all of the recognition units, and the handwriting input area returns to a normal state and ready to accept additional strokes. In some embodiments, the user device provides means for individually deleting each of the individual sets of recognition units currently displayed in the handwriting input area (2234). In some embodiments, the means is an individual delete button for each highlighted recognition unit. In some embodiments, the means is means for detecting a predetermined delete gesture over each highlighted recognition unit and invoking a function for deleting the highlighted recognition unit under the predetermined delete gesture.

As described herein, in some embodiments, the user device provides a continuous input mode in the handwriting input area. Since the area of the handwriting input area is limited on a portable user device, it is sometimes desirable to provide a way to cache handwriting inputs provided by the user and to allow the user to reuse screen space without committing previously provided handwriting inputs. In some embodiments, the user device provides a scrolling handwriting input area, where the input area gradually shifts by a predetermined amount (e.g., one recognition unit at a time) as the user gets close enough to the end of the handwriting input area. In some embodiments, it is sometimes advantageous to reuse previously used areas of the input area without dynamically shifting recognition units, since shifting existing recognition units in the handwriting input area can interfere with the user's writing process and possibly prevent accurate segmentation of the recognition units. In some embodiments, when the user reuses an area occupied by a handwriting input that has not yet been entered into the text input area, the top recognition result for the handwriting input area is automatically entered into the text input area, allowing the user to continue to provide new handwriting input without explicitly selecting the top recognition result.

In some conventional systems, a user is allowed to write over existing handwriting input that is still visible in the handwriting input area. In such systems, timing information is used to determine whether a new stroke is part of a previous recognition unit or a new recognition unit. Such timing-dependent systems place strict requirements on the speed and tempo at which the user provides the handwriting input, which are difficult for many users to meet. Additionally, the visual rendering of the handwriting input can be confusing and difficult for the user to decipher. Thus, the writing process can be frustrating and confusing to the user, resulting in an unpleasant user experience.

As described herein, a fading process is used to indicate when a user can reuse an area occupied by a previously written recognition unit and continue writing in the handwriting input area. In some embodiments, the fading process gradually reduces the visibility of each recognition unit provided in the handwriting input area over a threshold amount of time so that existing text does not visually compete with new strokes as new strokes are written over it. In some embodiments, writing over a faded recognition unit automatically causes the top-ranked recognition result for the recognition unit to be entered into the text input area without requiring the user to stop writing and explicitly provide a selection input for the top-ranked recognition result. This implicit and automatic confirmation of the top-ranked recognition result improves the efficiency and speed of the handwriting input interface and reduces the cognitive burden placed on the user to maintain the flow of thought of the current text composition. In some embodiments, writing over a faded recognition unit does not cause automatic selection of the top-ranked search result. Instead, the faded recognition units are stored in the handwriting input stack and combined with the new handwriting input as the current handwriting input. The user can view recognition results generated based on all recognition units accumulated in the handwriting input stack before making a selection.

FIGS. 23A through 23J illustrate exemplary user interfaces and processes in which recognition units provided in different zones of a handwriting input area are gradually faded out from their individual zones after a predetermined amount of time, for example, and a user is permitted to provide new handwriting strokes in a particular zone after the fade-out has occurred in that zone.

As illustrated in FIG. 23A, a user has provided a plurality of handwritten strokes (2302) (e.g., three handwritten strokes for a capital letter “I”) in a handwriting input area (804). The handwritten strokes (2302) are identified by the user device as recognition units. In some embodiments, the handwriting input currently displayed in the handwriting input area (804) is stored in a first layer of a handwriting input stack of the user device. A plurality of recognition results generated based on the identified recognition units are provided in the candidate display area (804).

FIG. 23B illustrates that as the user continues to write one or more strokes (2302) to the right of the strokes (2304), the handwritten strokes (2302) in the first recognition unit begin to gradually fade out in the handwriting input area (804). In some embodiments, an animation is displayed that mimics the gradual fading or disappearance of the visual rendering of the first recognition unit. For example, the animation may create the visual effect of ink evaporating from a whiteboard. In some embodiments, the fading of the recognition unit is not uniform across the entire recognition unit. In some embodiments, the fading of the recognition unit increases over time, eventually causing the recognition unit to become completely invisible in the handwriting input area. However, even if the recognition unit is no longer visible in the handwriting input area (804), in some embodiments, the invisible recognition unit remains at the top of the handwriting input stack, and recognition results generated from that recognition unit continue to be displayed in the candidate display area. In some embodiments, the faded recognition unit is not completely removed from view until new handwriting input is written over it.

In some embodiments, the user device enables new handwriting input to be provided over the area occupied by the faded recognition unit immediately upon the start of the fading animation. In some embodiments, the user device enables new handwriting input to be provided over the area occupied by the faded recognition unit only after the fading has progressed to a predetermined level (e.g., to a level that is completely imperceptible or until recognition is completely invisible in the area).

FIG. 23c illustrates that the first recognition unit (i.e., strokes (2302)) has completed its fading process (e.g., the ink color has stabilized at a very weak level or has become invisible). The user device has identified additional recognition units (e.g., recognition units for the handwritten characters "a" and "m") from additional handwritten strokes provided by the user and updated the recognition results presented in the candidate display area (806).

FIGS. 22d-22f illustrate that, over time, the user provides a plurality of additional handwritten strokes (e.g., 2304, 2306) in the handwriting input area (804). At the same time, previously identified recognition units are gradually faded out of the handwriting input area (804). In some embodiments, it takes a predetermined amount of time for each recognition unit to begin its own fading process after the recognition unit is identified. In some embodiments, the fading process for each recognition unit does not begin until the user begins inputting a second recognition unit subsequently thereto. As illustrated in FIGS. 23b-23f , when the handwriting input is provided in cursive, a single stroke (e.g., stroke 2304 or stroke 2306) may span multiple recognition units (e.g., recognition units for each handwritten character in the word "am" or "back") in the handwriting input area.

FIG. 22g illustrates that even after a recognition unit has started its fading process, the user can return it to its unfaded state by a predetermined recovery input, e.g., a tap gesture, on the delete button (2310) (e.g., as indicated by an immediate lift following contact (2308). When the recognition units are restored, their appearance returns to their normal visibility level. In some embodiments, the recovery of faded recognition units is character-by-character in the reverse direction in the handwriting input area (804). In some embodiments, the recovery of faded recognition units is word-by-word in the handwriting input area (804). As illustrated in FIG. 23g, the recognition units corresponding to the word "back" have been restored from a fully faded state to a not-faded state at all. In some embodiments, the clock for initiating the fading process is reset for each recognition unit when the recognition unit is restored to an unfaded state.

FIG. 22h illustrates that a sustained contact on the delete button causes the last recognition unit in the default writing direction (e.g., the recognition unit for the letter "k" in the word "back") to be deleted from the handwriting input area (804). As the delete input continues to be maintained, more recognition units (e.g., the recognition units for the letters "c", "a", and "b" in the word "back") are deleted one by one in the reverse writing direction. In some embodiments, the deletion of recognition units is word-by-word, such that all letters of the handwritten word "back" are removed simultaneously from the handwriting input area (804). FIG. 22h also illustrates that as the contact (2308) is maintained on the delete button (2310) after the deletion of the recognition unit for the letter "b" in the handwritten word "back", the previously faded recognition unit "m" is also restored.

FIG. 23i illustrates that if the deletion input is interrupted before deletion of the recovered recognition unit "m" in the handwritten word "am" occurs, the recovered recognition unit gradually fades again. In some embodiments, the state of each recognition unit (e.g., a state selected from a set of one or more faded states and a non-faded state) is maintained and updated in the handwriting input stack.

FIG. 23J illustrates that when a user provides one or more strokes (2312) over an area occupied by a faded recognition unit (e.g., a recognition unit for the letter “I”) in the handwriting input area, in some embodiments, the text of the top-level recognition result (e.g., result (2314)) for the handwriting input made prior to the strokes (2312) is automatically entered into the text input area (808), as illustrated in FIGS. 23I and 23J . As illustrated in FIG. 23J , the text “I am” is no longer shown as provisional but instead is committed in the text input area (808). In some embodiments, once text entry is made for a fully or partially faded handwriting input, the handwriting input is removed from the handwriting input stack. The newly entered strokes (e.g., strokes (2312)) become the current input in the handwriting input stack.

As illustrated in FIG. 23j, the text "I am" is no longer shown as provisional but instead is committed in the text input area (808). In some embodiments, once text input is made for a fully or partially faded handwriting input, the handwriting input is removed from the handwriting input stack. The newly entered strokes (e.g., strokes (2312)) become the current input in the handwriting input stack.

In some embodiments, when strokes (2312) are provided over an area occupied by a faded recognition unit (e.g., a recognition unit for the letter “I”) in the handwriting input area, the text of the top-level recognition result (e.g., result (2314)) for the handwriting input made prior to the strokes (2312) is not automatically entered into the text input area (808). Instead, the current handwriting input (both faded and non-faded) in the handwriting input area (804) is cleared and stored in the handwriting input stack. The new strokes (2312) are added to the stored handwriting input in the handwriting input stack. The user device determines recognition results based on the entire currently accumulated handwriting input in the handwriting input stack. The recognition results are displayed in the candidate display area. In other words, even if only a portion of the currently accumulated handwriting input appears in the handwriting input area (804), the recognition results are generated based on the entire handwriting input (both the visible portion and the portions that are no longer visible) stored in the handwriting input stack.

FIG. 23K illustrates that the user has entered more faded strokes (2316) into the handwriting input area (804) over time. FIG. 23L illustrates that a new stroke (2318) has been written over the faded strokes (2312, 2316), causing the text of the top recognition result (2320) for the faded strokes (2312, 2316) to be entered into the text input area (808).

In some embodiments, the user optionally provides handwriting input in multiple lines. In some embodiments, when multi-line input is possible, the same fading process may be used to clear the handwriting input area for new handwriting input.

FIGS. 24A and 24B are flowcharts of an exemplary process (2400) for providing a fading process in a handwriting input area of a handwriting input interface. FIGS. 23A through 23K illustrate a process (2400) according to some embodiments.

In some embodiments, the device receives a first handwriting input from a user (2402). The first handwriting input includes a plurality of handwriting strokes, and the plurality of handwriting strokes form a plurality of recognition units distributed along respective writing directions associated with a handwriting input area of a handwriting input interface. In some embodiments, the user device renders each of the plurality of handwriting strokes in the handwriting input area as the handwriting strokes are provided by the user (2404).

In some embodiments, the user device initiates an individual fading process for each of the plurality of recognition units after the recognition unit is fully rendered (2406). In some embodiments, during the individual fading process, the rendering of the recognition unit in the first handwriting input is faded away. This is illustrated in FIGS. 23A through 23F according to some embodiments.

In some embodiments, for example, as illustrated in FIGS. 23I and 23J and FIGS. 23K and 23L , the user device receives a second handwriting input from the user over an area occupied by a faded recognition unit of a plurality of recognition units in the handwriting input area (2408). In some embodiments, in response to receiving the second handwriting input (2410): the user device renders the second handwriting input in the handwriting input area (2412) and clears all faded recognition units from the handwriting input area (2414). In some embodiments, all recognition units input in the handwriting input area prior to the second handwriting input are cleared from the handwriting input area, regardless of whether the recognition units have begun their fading process. This is illustrated, for example, in FIGS. 23I and 23J and FIGS. 23K and 23L .

In some embodiments, the user device generates one or more recognition results for the first handwriting input (2416). In some embodiments, the user device displays the one or more recognition results in a candidate display area of the handwriting input interface (2418). In some embodiments, in response to receiving the second handwriting input, the user device automatically, without user selection, enters the top recognition result displayed in the candidate display area into a text input area of the handwriting input interface (2420). This is illustrated, for example, in FIGS. 23I and 23J and FIGS. 23K and 23L.

In some embodiments, the user device stores an input stack including the first handwriting input and the second handwriting input (2422). In some embodiments, the user device generates one or more multi-character recognition results each including a separate spatial sequence of characters recognized from a concatenation of the first handwriting input and the second handwriting input (2424). In some embodiments, the user device displays the one or more multi-character recognition results in a candidate display area of the handwriting input interface while rendering of the second handwriting input in the handwriting input area supersedes rendering of the first handwriting input (2426).

In some embodiments, the individual fading process for each recognition unit is initiated a predetermined period of time after the recognition unit has been completed by the user.

In some embodiments, the fading process for each recognition unit begins when the user starts entering strokes for the next recognition unit after the recognition unit.

In some embodiments, the final state of the individual fading process for each recognition unit is a state having a predetermined minimum visibility for the recognition unit.

In some embodiments, the final state of the individual fading process for each recognition unit is a state with zero visibility for the recognition unit.

In some embodiments, after the last recognition unit in the first handwriting input has faded, the user device receives a predetermined recovery input from the user (2428). In response to receiving the predetermined recovery input, the user device restores the last recognition unit from a faded state to an unfaded state (2430). This is illustrated, for example, in FIGS. 23F-23H. In some embodiments, the predetermined recovery input is an initial contact detected on a delete button provided in the handwriting input interface. In some embodiments, a sustained contact detected on the delete button deletes the last recognition unit from the handwriting input area and restores the penultimate recognition unit from a faded state to an unfaded state. This is illustrated, for example, in FIGS. 23G and 23H.

As described herein, the multi-script handwriting recognition model performs stroke-order-independent and stroke-direction-independent recognition of handwritten characters. In some embodiments, the recognition model is trained only on spatially-derived features contained in flat images of writing samples corresponding to different characters in the vocabulary of the handwriting recognition model. Since the images of the writing samples do not contain any temporal information related to the individual strokes contained in the images, the resulting recognition model is stroke-order-independent and stroke-direction-independent.

As exemplified above, stroke-order and stroke-direction independent handwriting recognition offers many advantages over conventional recognition systems that rely on information related to the temporal generation of characters (e.g., the temporal sequences of strokes in the characters). However, in real-time handwriting recognition scenarios, temporal information related to individual strokes is available, and it is sometimes beneficial to utilize this information to improve the recognition accuracy of the handwriting recognition system. The following describes a technique for integrating temporal-derived stroke-distribution information into spatial feature extraction of a handwriting recognition model, wherein the use of temporal-derived stroke-distribution information does not compromise the stroke-order and/or stroke-direction independence of the handwriting recognition system. Based on the stroke-distribution information related to different characters, disambiguation between similar-looking characters that are generated with uniquely different sets of strokes is enabled.

In some embodiments, when handwriting input is converted into an input image (e.g., an input bitmap image) for a handwriting recognition model (e.g., a CNN), the temporal information associated with individual strokes is lost. For example, the Chinese character " ", eight strokes (e.g., labeled #1 to #8 in FIG. 27) can be used to write a character. The sequence and direction of the strokes for a character provide several unique features associated with the character. A naive way to capture stroke order and stroke direction information without compromising the stroke order and stroke direction independence of the recognition system is to explicitly enumerate all possible permutations of stroke order and stroke direction in the training samples. However, even for characters of only moderate complexity, this would amount to over a billion possibilities, which is difficult, if not impossible, to implement in practice. As described herein, a stroke-distribution profile is generated for each writing sample, which extracts from chronological aspects of stroke generation (i.e., temporal information). The stroke-distribution profiles of the writing samples are trained to extract a set of temporal features (e.g., from input bitmap images) that are subsequently combined with the spatial-derived features, without compromising the stroke order and stroke direction independence of the handwriting recognition system. and improve recognition accuracy.

As described herein, temporal information associated with a character is extracted by producing various pixel distributions to characterize each handwritten stroke. Each handwritten stroke of a character gives rise to a deterministic pattern (or profile) when projected along a given direction. While this pattern alone may not be sufficient to clearly recognize a stroke, when combined with other similar patterns, it may be adequate to capture certain characteristics unique to that particular stroke. Integrating this kind of stroke representation with spatial feature extraction (e.g., feature extraction based on input images in a CNN) ultimately provides orthogonal information that may be useful for disambiguating between similar-looking characters in the repertoire of a handwriting recognition model.

FIGS. 25A and 25B are flowcharts of an exemplary process (2500) for integrating temporal and spatial features of handwriting samples during training of a handwriting recognition model, wherein the resulting recognition model remains stroke-order and stroke-direction independent. In some embodiments, the exemplary process (2500) is performed on a server device that provides the trained recognition model to a user device (e.g., the portable device (100)). In some embodiments, the server device includes one or more processors and a memory including instructions for performing the process (2500) when executed by the one or more processors.

In an exemplary process (2500), the device separately trains a set of spatially-derived features and a set of temporally-derived features of a handwriting recognition model (2502), wherein the set of spatially-derived features is trained on a corpus of training images, each of which is an image of a handwriting sample for an individual character of an output character set, and the set of temporally-derived features is trained on a corpus of stroke-distribution profiles, each stroke-distribution profile numerically characterizing the spatial distribution of a plurality of strokes in a handwriting sample for an individual character of the output character set.

In some embodiments, training a set of spatially-guided features separately further comprises training a convolutional neural network having an input layer, an output layer, and a plurality of convolutional layers, including a first convolutional layer, a last convolutional layer, zero or more intermediate convolutional layers between the first convolutional layer and the last convolutional layer, and a hidden layer between the last convolutional layer and the output layer (2504). An exemplary convolutional network (2602) is illustrated in FIG. 26 . The exemplary convolutional network (2602) can be implemented in substantially the same manner as the convolutional network (602) illustrated in FIG. 6 . The convolutional network (2602) includes an input layer (2606), an output layer (2608), a first convolutional layer (2610a), a plurality of convolutional layers including zero or more intermediate convolutional layers and a last convolutional layer (2610n), and a hidden layer (2614) between the last convolutional layer and the output layer (2608). The convolutional network (2602) also includes kernel layers (2616) and sub-sampling layers (2612) according to the arrangement illustrated in FIG. 6. The training of the convolutional network is based on images (2616) of inscription samples in a training corpus (2604). Spatially-derived features are obtained by minimizing recognition errors for training samples in the training corpus, and individual weights associated with different features are determined. The same features and weights that have been trained once are used to recognize new handwriting samples that do not exist in the training corpus.

In some embodiments, training the set of temporal-derived features separately further comprises providing the plurality of stroke-distribution profiles as a statistical model to determine the plurality of temporal-derived parameters and individual weights for the plurality of temporal-derived parameters to distinguish individual characters of the output character set (2506). In some embodiments, a stroke-distribution profile (2620) is derived from each writing sample in the training corpus (2622), as illustrated in FIG. 26 . The training corpus (2622) optionally includes the same writing samples as the corpus (2604), but also includes temporal information associated with the stroke generation in each writing sample. The stroke-distribution profiles (2622) are provided to a statistical modeling process (2624), during which time-derived features are extracted and individual weights for different features are determined by minimizing recognition or segmentation errors based on a statistical modeling method (e.g., CNN, K-Nearest Neighbor, etc.). As illustrated in FIG. 26, the set of time-derived features and the individual weights are converted into a set of feature vectors (e.g., feature vectors (2626) or feature vectors (2628)) and fed into individual layers in the convolutional neural network (2602). Thus, the resulting network includes spatial-derived parameters and temporal-derived parameters that are orthogonal to each other and jointly contribute to the recognition of characters.

In some embodiments, the device combines a set of spatial-derived features and a set of temporal-derived features in a handwriting recognition model (2508). In some embodiments, combining the set of spatial-derived features and the set of temporal-derived features in the handwriting recognition model comprises injecting a plurality of spatial-derived parameters and a plurality of temporal-derived parameters into a convolutional layer or a hidden layer of one of the convolutional layers of the convolutional neural network (2510). In some embodiments, individual weights for the plurality of temporal-derived parameters and the plurality of temporal-derived parameters are injected into a last convolutional layer of the convolutional neural network for handwriting recognition (e.g., the last convolutional layer (2610n) of FIG. 26 ). In some embodiments, multiple time-derived parameters and individual weights for the multiple time-derived parameters are injected into a hidden layer of a convolutional neural network (e.g., hidden layer (2614) of FIG. 26) for handwriting recognition.

In some embodiments, the device provides real-time handwriting recognition of the user's handwriting input using a handwriting recognition model (2512).

In some embodiments, the device generates a corpus of stroke-distribution profiles from a plurality of writing samples (2514). In some embodiments, each of the plurality of writing samples corresponds to a character in the output character set and separately preserves individual spatial information for each constituent stroke of the writing sample when written (2516). In some embodiments, to generate the corpus of stroke-distribution profiles, the device performs the following steps (2518):

For each of the plurality of handwriting samples (2520): the device identifies constituent strokes in the handwriting sample (2522); for each of the identified strokes of the handwriting sample, the device calculates an individual occupancy along each of the plurality of predetermined directions (2524), the occupancy being a ratio between a projection span in each stroke direction and a maximum projection span of the writing sample; for each of the identified strokes of the handwriting sample, the device also calculates an individual saturation ratio for each stroke based on a ratio between the individual number of pixels in each stroke and the total number of pixels in the writing sample (2526). Thereafter, the user device generates a feature vector for the handwriting sample as a stroke-distribution profile of the handwriting sample (2528), the feature vector comprising individual occupancies and individual saturation ratios of at least N strokes in the handwriting sample, where N is a predetermined natural number. In some embodiments, N is less than a maximum number of strokes observed in any single writing sample in the plurality of writing samples.

In some embodiments, for each of the plurality of handwriting samples: the device sorts the individual occupancies of the identified strokes in each of the predetermined directions in descending order; and includes only the N highest occupancies and saturation ratios of the writing sample in the feature vector of the writing sample.

In some embodiments, the plurality of predetermined directions include a horizontal direction, a vertical direction, a positive 45 degree direction, and a negative 45 degree direction of the writing sample.

In some embodiments, to provide real-time handwriting recognition of a user's handwriting input using a handwriting recognition model, the device receives the user's handwriting input; and in response to receiving the user's handwriting input, provides handwriting recognition output to the user substantially simultaneously with the receipt of the handwriting input.

The character " shown in Fig. 27 ", for illustrative purposes, exemplary embodiments are described herein. In some embodiments, each input image of a handwritten character is optionally normalized to a square. The span of each individual handwritten stroke (e.g., strokes (#1, #2, ..., #8)) is measured as it is projected onto the horizontal, vertical, +45 degree diagonal, and -45 degree diagonal of the square. The spans of each stroke Si are recorded for the four projection directions, respectively, as xspan ( i ), yspan ( i ), cspan ( i ), and dspan ( i ). Additionally, the maximum spans observed over the entire image are also recorded. The maximum spans of the character are recorded for the four projection directions, respectively, as xspan , yspan , cspan , and dspan . For illustrative purposes, the four projection directions are optionally considered herein, but in principle any arbitrary number of projections may be used. The set can be used in various embodiments. The characters " in four transparent directions "The maximum spans (e.g., represented as xspan , yspan , cspan , and dspan ) of one of the strokes (e.g., stroke (#4)) and spans (e.g., represented as xspan ( 4 ), yspan ( 4 ), cspan ( 4 ), and dspan (4)) are illustrated in FIG. 27 .

In some embodiments, once the spans have been measured for all strokes 1 through 5, where 5 is the number of individual handwritten strokes associated with the input image, an individual occupancy along each projection direction is computed. For example, for stroke S _i , the individual occupancy R _x ( i ) along the x -direction is computed as R _x ( i ) = xspan ( i ) / xspan . Similarly, individual occupancies along other projection directions can be computed as: R _y ( i ) = yspan ( i )/ yspan , R _c ( i ) = cspan (i)/ cspan , R _d ( i ) = dspan ( i ) /dspan .

In some embodiments, the occupancies of all strokes in each direction are separately sorted in descending order, so that an individual ranking of all strokes in the input image is obtained in terms of their occupancies in that direction for each projection direction. The ranking of strokes in each projection direction reflects the relative importance of each stroke along the associated projection direction. This relative importance is independent of the order and direction in which the strokes are generated in the writing samples. Thus, this ranking based on occupancies is time-derived information that is independent of stroke order and stroke direction.

In some embodiments, each stroke is given a relative weight indicating the importance of that stroke to the entire character. In some embodiments, the weight is measured by the ratio of the number of pixels in each stroke to the total number of pixels in the character. This ratio is referred to as the saturation ratio associated with each stroke.

In some embodiments, a feature vector may be generated for each stroke based on the saturation ratio and occupancy rates of each stroke. For each character, a set of feature vectors containing 5S feature segments is generated. This set of feature segments is referred to as the stroke-distribution profile of the character.

In some embodiments, only a predetermined number of top strokes are used when constructing the stroke-distribution profile of each character. In some embodiments, the predetermined number of strokes is 10. Based on the top 10 strokes, 50 stroke-derived features can be generated for each character. In some embodiments, these features are injected into the last convolutional layer of the convolutional neural network, or into subsequent hidden layers.

In some embodiments, during real-time recognition, an input image of a recognition unit is provided in a handwriting recognition mode trained with both spatial-derived features and temporal-derived features. The input image is processed through each layer of the handwriting recognition model illustrated in FIG. 26. When the processing of the input image reaches a layer requiring stroke-distribution profile input (e.g., the last convolutional layer or the hidden layer), the stroke-distribution profile of the recognition unit is fed into that layer. The processing of the input image and the stroke-distribution profile continues until output segments (e.g., one or more candidate characters) are provided at the output layer (2608). In some embodiments, the stroke-distribution profiles of all the recognition units are computed and provided as input to the handwriting recognition model along with the input images of the recognition units. In some embodiments, the input image of the recognition unit initially experiences the handwriting recognition model (without the benefit of the temporal-trained features). When two or more similar-looking candidate characters are identified with close recognition confidence values, the stroke-distribution profiles of the recognition unit are fed into the handwriting recognition model in a layer trained with temporally-derived features (e.g., the last convolutional layer or the hidden layer). When the input image and stroke-distribution profile of the recognition unit pass through the last layers of the handwriting recognition model, two or more similar-looking candidate characters can be better distinguished due to the differences in their stroke-distribution profiles. Therefore, temporal-derived information related to how a recognition unit is formed by individual handwritten strokes is utilized to improve the recognition accuracy without compromising the stroke-order and stroke-direction independence of the handwriting recognition system.

The foregoing description has been described with reference to specific embodiments for the purpose of explanation. However, the exemplary discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, thereby enabling others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated.

Claims (12)

As a method for providing handwriting recognition,
In an electronic device having a display and a touch-sensitive surface,
A step of receiving a handwriting input, wherein the handwriting input comprises a plurality of handwriting strokes provided on the touch-sensitive surface coupled to the electronic device;
A step of displaying the plurality of handwriting strokes within a handwriting input area of a handwriting input interface;
A step of dividing said plurality of handwritten strokes into two or more recognition units, each recognition unit comprising a distinct subset of said plurality of handwritten strokes;
Step of receiving an editing request;
In response to said editing request, a step of visually distinguishing at least one of said two or more recognition units in said handwriting input area;
A step of displaying an individual delete button in proximity to at least one of the two or more recognition units;
A step of detecting a predetermined deletion gesture input on at least one of the two or more recognition units within the handwriting input area; and
A method comprising: in response to detecting the predetermined delete gesture input, removing a discrete subset of the handwritten strokes corresponding to at least one of the two or more recognition units within the handwriting input area. In the first paragraph,
A method wherein visually distinguishing two or more recognition units further comprises highlighting individual boundaries between the two or more recognition units in the handwriting input area. In the first paragraph,
A method wherein an edit request corresponds to a contact detected on a predetermined affordance displayed on the handwriting input interface. In the first paragraph,
A method wherein an edit request corresponds to a tap gesture detected over a predetermined area in the handwriting input interface. In paragraph 4,
A method wherein the above predetermined area corresponds to an area within the handwriting input area of the handwriting input interface. In paragraph 4,
A method wherein the above predetermined area corresponds to an area outside the handwriting input area of the handwriting input interface. In the first paragraph,
A method wherein at least one of the two or more recognition units spatially corresponds to an initial recognition unit among the two or more recognition units. In the first paragraph,
A method, wherein at least one of the two or more recognition units corresponds to a spatially intermediate recognition unit among the two or more recognition units. In the first paragraph,
A step of generating a segmentation lattice from said plurality of handwritten strokes, said segmentation lattice comprising a plurality of alternative segmentation chains, each of which represents a distinct set of recognition units identified from said plurality of handwritten strokes;
A step of receiving two or more consecutive editing requests;
In response to each of said two or more consecutive edit requests, visually distinguishing said individual set of recognition units in said handwriting input area from other segmentation chains among said plurality of alternative segmentation chains;
detecting a second predetermined deletion gesture input on at least one of the recognition units currently displayed in the handwriting input area; and
A method further comprising, in response to detecting the second predetermined delete gesture input, deleting a respective subset of handwritten strokes corresponding to at least one of the recognition units currently displayed in the handwriting input area. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a device having a display and a touch-sensitive surface,
A non-transitory computer-readable storage medium comprising instructions for performing a method of providing handwriting recognition according to any one of claims 1 to 9, wherein said one or more programs are as follows. As an electronic device that provides handwriting recognition,
display;
touch-sensitive surface;
one or more processors; and
A memory storing one or more programs configured to be executed by said one or more processors,
An electronic device, wherein said one or more programs include instructions for performing a method of providing handwriting recognition according to any one of claims 1 to 9. delete