KR102646428B1 - Method and apparatus for extracting similar letters using artificial intelligence learning model - Google Patents

Abstract

A method for extracting similar characters using an artificial intelligence learning model is disclosed. A method of extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention includes receiving first character data that is to be confirmed from the user, and second character data corresponding to all characters included in a plurality of target documents. It may include extracting third character data from among the character data based on preset similar character judgment conditions according to the format of the first character data and providing the third character data to the user terminal.

Description

Method and device for extracting similar letters using artificial intelligence learning model {METHOD AND APPARATUS FOR EXTRACTING SIMILAR LETTERS USING ARTIFICIAL INTELLIGENCE LEARNING MODEL}

The present invention relates to a method and device for extracting similar characters using an artificial intelligence learning model.

The content described below simply provides background information related to an embodiment of the present invention and does not constitute prior art.

Handwriting refers to the unique habits that appear when writing. Therefore, handwriting appears subtly different from person to person. For this reason, handwriting analysis is used to determine whether a document was written by the person in criminal investigations or wills. Unless you are a handwriting expert, it is not easy to distinguish because they look similar, but the handwriting, such as when written under pressure or when copying someone else's writing, leaves traces that are different from usual, so it is highly credible as evidence.

Existing handwriting appraisal work involves an appraiser visually checking and comparing identical letters to extract and classify them. Accordingly, excessive time is spent on extraction and classification, and omissions or errors may occur. Therefore, a character extraction method that can reduce the labor of the appraiser and shorten the time consumed is required.

The purpose of the present invention is to extract similar letters using an artificial intelligence learning model.

Additionally, the purpose of the present invention is to extract and provide letters similar to letters entered by a user.

Additionally, the purpose of the present invention is to display and provide letters selected by the user among the extracted letters.

In order to achieve the above object, a method of extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention includes the steps of inputting first character data that is to be confirmed from the user, and the entire text contained in the plurality of target documents. Extracting third character data from among second character data corresponding to letters based on preset similar character judgment conditions according to the format of the first character data and providing the third character data to the user terminal. may include.

The second letter data includes extracting a first fragment image of the area where each of the entire letters included in the plurality of target documents is located using a first artificial intelligence learning model, and using a second artificial intelligence learning model. It further includes a label value labeling a letter corresponding to each of the first fragment images, and the first artificial intelligence learning model is configured to determine an area where the letter is expected to exist within the document image corresponding to the plurality of target documents. It is an artificial intelligence learning model that extracts a fragment image for, and the second artificial intelligence learning model may be an artificial intelligence learning model that labels letters corresponding to the fragment image extracted by the first artificial intelligence learning model.

The step of receiving the first character data includes receiving a second piece image corresponding to an area designated by the user among the document images corresponding to the plurality of target documents, or input by the user from the user terminal. It may include receiving text as the first character data.

In the step of extracting the third character data, when the first character data is the second fragment image, the similarity between the second fragment image and the first fragment images is respectively calculated using a third artificial intelligence learning model. A step of calculating and extracting third fragment images whose similarity is greater than or equal to a preset similarity among the first fragment images as the third character data, wherein the third artificial intelligence learning model includes the first fragment image and the third fragment image. It may be an artificial intelligence learning model that calculates the similarity between two piece images.

The step of extracting the third character data includes, when the first character data is the text, extracting fourth character data whose label value corresponding to the second character data is the same as the text among the second character data. It may include extracting a fourth piece image corresponding to the fourth character data and the third character data.

The step of providing the third character data includes generating a grid-shaped character list using fragment images corresponding to the third character data and providing the first character data and the character list to the user terminal. may include.

After providing the third character data, receiving the user's selection of a fifth piece image that is one of the letter list from the user terminal and including the fifth piece image among the document images. It may include displaying a block of a preset size in an area corresponding to the fifth fragment image within the page.

The step of receiving the first character data may include receiving a cursive directly input by the user through the user terminal and extracting the first character data from the cursive.

The step of receiving the first character data includes receiving the user's selection of a sixth piece image that is one of the first piece images corresponding to the second character data and labeling the sixth piece image. It may include receiving the corresponding text as the first character data.

In addition, a similar character extracting device using an artificial intelligence learning model according to an embodiment of the present invention includes a memory in which at least one program is recorded and a processor that executes the program, and the program is an object that the user wishes to check. Inputting one character data, third character data based on preset similar character judgment conditions according to the format of the first character data among second character data corresponding to all characters included in a plurality of target documents It may include instructions for performing the step of extracting and providing the third character data to the user terminal.

According to the present invention, similar letters can be extracted using an artificial intelligence learning model.

Additionally, according to the present invention, letters similar to letters input by the user can be extracted and provided.

Additionally, according to the present invention, letters selected by the user among the extracted letters can be displayed and provided.

Figure 1 is a block diagram showing subjects for extracting similar letters using an artificial intelligence learning model according to an embodiment of the present invention.
Figure 2 is an operation flowchart showing a method of extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention.
Figure 3 is an operation flowchart showing a method of extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention.
Figure 4 is an operation flowchart showing a method of extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention.
Figure 5 is an operation flowchart showing a method of extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention.
Figure 6 is an operation flowchart showing a method of extracting similar letters using an artificial intelligence learning model according to an embodiment of the present invention.
Figure 7 is a diagram showing a computer system according to an embodiment of the present invention.
Figure 8 is a diagram showing the structure of a handwriting recognition function according to an embodiment of the present invention.
Figure 9 is a diagram showing a method for automatically extracting identical characters and a similarity learning structure according to an embodiment of the present invention.
Figure 10 is a diagram showing an example of character region detection using the CRAFT (Character Region Awareness for Text Detection) model.
Figure 11 is a diagram showing the neural network structure of the CRAFT model.
Figure 12 is a diagram showing the process of making BOX into SCORE in the CRAFT model.
Figure 13 is a diagram showing the learning process of the CRAFT model.
Figure 14 is a diagram showing the results of extracting letters using the CRAFT model.
Figure 15 is a diagram showing examples of positive samples and negative samples for metric learning.
Figure 16 is a diagram showing the influence of the embedding space on loss in metric learning.
Figure 17 is a diagram showing a depiction of a metric learning method.
Figure 18 is a diagram showing an example of a method for augmenting data used in a metric learning method.
Figure 19 is a diagram showing a target document displaying an area designated by the user in a similar character extraction method using an artificial intelligence learning model according to an embodiment of the present invention.
Figure 20 is a diagram showing an example of the UI of the result screen provided to the user terminal.

The present invention will be described in detail with reference to the attached drawings as follows. Here, repeated descriptions, known functions that may unnecessarily obscure the gist of the present invention, and detailed descriptions of configurations are omitted. Embodiments of the present invention are provided to more completely explain the present invention to those skilled in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clearer explanation.

Throughout the specification, when a part “includes” a certain element, this means that it may further include other elements rather than excluding other elements, unless specifically stated to the contrary.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a block diagram showing subjects for extracting similar letters using an artificial intelligence learning model according to an embodiment of the present invention.

Referring to FIG. 1, subjects for extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention include a similar character extracting device 110 and a user terminal 120.

The similar character extraction device 110 may refer to a device that receives first character data that the user wishes to check.

The similar character extraction device 110 extracts the third character data using preset similar character judgment conditions according to the format of the first character data among the second character data corresponding to all characters included in the plurality of target documents. It may be an extraction device.

The similar character extraction device 110 may be a device that provides third character data to the user terminal 120.

The user terminal 120 may be a device that receives third character data from the similar character extraction device 110.

The similar character extracting device 110 and the user terminal 120 may be interconnected through a communication network.

A communication network refers to an access path for data to be transmitted and received between the above entities. For example, communication networks include wired networks such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), and ISDNs (Integrated Service Digital Networks), or wireless networks such as wireless LANs, CDMA, Bluetooth, and satellite communications. It may encompass networks, but the scope of communication networks applicable to the present invention is not limited thereto.

Figure 2 is an operation flowchart showing a method of extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention.

Referring to FIG. 2, the method of extracting similar letters using an artificial intelligence learning model according to an embodiment of the present invention can first receive input of first letter data that is to be confirmed from the user (S210).

Next, third character data can be extracted from second character data corresponding to all characters included in a plurality of target documents based on preset similar character judgment conditions according to the format of the first character data. (S220).

Here, the second letter data may include extracting a first fragment image for an area where each of all letters included in the plurality of target documents is located using a first artificial intelligence learning model. In addition, it may further include a label value labeling letters corresponding to each of the first fragment images using a second artificial intelligence learning model.

Here, the label value may be a value corresponding to a letter included in the first fragment image. For example, if the first piece image includes the letter 'A', the label value of the first piece image may be labeled as 'A'.

At this time, the first artificial intelligence learning model is an artificial intelligence learning model that extracts a fragment image for the area where letters are expected to exist within the document image corresponding to the plurality of target documents, and the second artificial intelligence learning model is It may be an artificial intelligence learning model that labels letters corresponding to the fragment image extracted by the first artificial intelligence learning model.

For example, the first artificial intelligence learning model may include a CRAFT (Character Region Awareness for Text Detection) model. The CRAFT model is a model that can extract letters by searching the area where the letters are located on a letter by letter basis.

Next, the third character data can be provided to the user terminal (S230).

According to one embodiment, the step of receiving first character data includes receiving a second piece image corresponding to an area designated by the user among the document images corresponding to the plurality of target documents or from the user terminal. It may include receiving the text input by the user as the first character data.

For example, as shown in Figure 19, the user can drag and select a certain area containing the letter to be input among the document images, and receive the second piece image corresponding to the selected area as the first letter data. Alternatively, the user can input the letters they want to find in document images as text, and receive the input text as first letter data.

According to one embodiment, the step of receiving first character data may include receiving a handwriting directly input by the user through the user terminal and extracting the first character data from the handwriting.

For example, a user can directly input cursive writing using a pen-type input device, and extract a preset area containing the input cursive text as first character data and receive the input. When used in a criminal investigation, it can be used to evaluate the suspect's handwriting by having the suspect write in cursive.

According to one embodiment, the step of receiving first character data includes receiving the user's selection of a sixth piece image that is one of the first piece images corresponding to the second letter data, and the sixth piece image. It may include receiving text corresponding to the labeling of the fragment image as the first character data.

For example, the user may select a fragment image containing a letter to be input among fragment images included in document images, and receive text corresponding to the labeling for the selected fragment image as first character data.

According to one embodiment, before step S210, the first fragment image for the area where each of the entire letters included in the plurality of target documents is located is extracted, and the first fragment image is extracted using a second artificial intelligence learning model. Second letter data can be generated by labeling letters corresponding to each image.

According to one embodiment, before step S220, a first fragment image for an area where each of all letters included in the plurality of target documents is located is extracted, and the first fragment image is extracted using a second artificial intelligence learning model. Second letter data can be generated by labeling letters corresponding to each image.

Figure 3 is an operation flowchart showing a method of extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention.

Referring to FIG. 3, the method of extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention first uses a third artificial intelligence learning model when the first character data is the second fragment image. Thus, the degree of similarity between the second fragment image and the first fragment image can be calculated, respectively (S310).

At this time, the third artificial intelligence learning model may be an artificial intelligence learning model that calculates the similarity between the first fragment image and the second fragment image.

For example, since the second fragment image is not labeled, the similarity can be calculated by comparing the image shapes of each of the second fragment image and the first fragment image using a third artificial intelligence learning model.

Here, the third artificial intelligence learning model can be learned by applying the Proxy Anchor Loss-based metric learning method. Similarity can be calculated through learning using Proxy Anchor Loss, with the same letters as positive samples and different letters as negative samples. It can have fast speed and performance based on proxy. If data is insufficient, data augmentation techniques can be used to overcome the insufficient data. When augmenting data, the data can be augmented by varying the slope, color, background color, sharpness, etc. for the same text, as shown in FIG. 18. Additionally, data can be augmented by adding an environment similar to the horizontal lines of a notebook.

Next, among the first fragment images, third fragment images whose similarity is greater than or equal to a preset similarity can be extracted as the third character data (S320).

Figure 4 is an operation flowchart showing a method of extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention.

Referring to FIG. 4, the method of extracting similar letters using an artificial intelligence learning model according to an embodiment of the present invention first, when the first letter data is the text, the second letter among the second letter data Fourth character data whose label value corresponding to the data is the same as the text can be extracted (S410).

Next, the fourth piece image corresponding to the fourth character data can be extracted as the third character data (S420).

As an optional embodiment, fourth character data and character data similar to the fourth character data may be extracted as third character data.

Specifically, after extracting the fourth letter data, the similarity between the fragment image corresponding to the fourth letter data and the first fragment images can be calculated using a third artificial intelligence learning model, respectively.

Next, based on the calculated similarity, the seventh fragment images and the fourth character data that have a preset similarity or higher among the first fragment images may be extracted as the third character data.

Through this, if a labeling error occurs in the second character data and the same character data is not extracted for the user's text input, there is an effect of allowing additional extraction through the similarity of the fragment image.

Figure 5 is an operation flowchart showing a method of extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention.

Referring to FIG. 5, the method of extracting similar letters using an artificial intelligence learning model according to an embodiment of the present invention can first generate a grid-shaped letter list using fragment images corresponding to the third letter data. There is (S510).

Next, the first character data and the character list may be provided to the user terminal (S520).

Figure 20 is a diagram showing an example of a UI (user interface) of a result screen including first character data provided to a user terminal, a character list, and a document page corresponding to the first character data. The user can check the first character data, character list, and document page through one screen.

Figure 6 is an operation flowchart showing a method of extracting similar letters using an artificial intelligence learning model according to an embodiment of the present invention.

Referring to FIG. 6, in the method of extracting similar letters using an artificial intelligence learning model according to an embodiment of the present invention, after providing the third letter data, a fifth letter from the user terminal is selected from the list of letters. The user's selection of the sculpture image may be received (S610).

Next, a block of a preset size may be displayed in an area corresponding to the fifth fragment image within the page including the fifth fragment image among the document images (S620).

As an optional embodiment, character data in morpheme units may be input, and identical or similar character data may be extracted and provided.

Specifically, the first character data may be input as a fragment image in units of morphemes or as text in units of morphemes from the user.

In addition, when extracting the second character data, the parameters of the first artificial intelligence learning model are adjusted to extract the first fragment image in units of morphemes, and the second artificial intelligence learning model is used to extract the first fragment image corresponding to each of the first fragment images. The second letter data can be extracted by labeling the morpheme.

Next, when the first letter data is a fragment image, the similarity with the first fragment images can be calculated, and the third fragment images with a preset similarity or higher can be extracted as the third letter data. In addition, when the first character data is text, the fourth character data whose labeling corresponding to the second character data is the same as the text can be extracted, and the fourth fragment image corresponding thereto can be extracted as the third character data.

Next, a grid-shaped morpheme list can be created using fragment images of morpheme units corresponding to the third character data and provided to the user terminal.

Figure 7 is a diagram showing a computer system according to an embodiment of the present invention.

An apparatus for extracting similar characters using an artificial intelligence learning model according to an embodiment of the present invention may be implemented in a computer system 1000 such as a computer-readable recording medium.

Referring to FIG. 7, the computer system 1000 includes one or more processors 1010, memory 1030, user interface input device 1040, user interface output device 1050, and storage that communicate with each other through a bus 1020. It may include (1060). Additionally, the computer system 1000 may further include a network interface 1070 connected to the network 1080. The processor 1010 may be a central processing unit or a semiconductor device that executes processing instructions stored in the memory 1030 or storage 1060. Memory 1030 and storage 1060 may be various types of volatile or non-volatile storage media. For example, memory may include ROM 1031 or RAM 1032.

Figure 8 is a diagram showing the structure of a handwriting recognition function according to an embodiment of the present invention.

Referring to Figure 8, the structure of the cursive recognition function can recognize Korean cursive from a file converted to a digital image and store the coordinates of each letter position.

At this time, after matching letters similar to the recognized letters, the position of the letter can be marked with a preset color and shape, and the marked area can be saved as an image.

Figure 9 is a diagram showing a method for automatically extracting identical characters and a similarity learning structure according to an embodiment of the present invention.

Referring to FIG. 9, the method for automatically extracting identical characters according to an embodiment of the present invention can use an optical character recognition method suitable for Hangul, which has a large number of characters due to its combination of initial consonants, middle consonants, and final consonants.

Letters similar to the input letters can be extracted from target documents.

At this time, the similarity between letters of similar shapes can be calculated using an artificial intelligence learning model learned based on metric learning.

Here, the metric learning method may refer to a method of learning to increase the similarity between the same letters and reduce the similarity between different letters.

Figure 10 is a diagram showing an example of character region detection using the CRAFT (Character Region Awareness for Text Detection) model. Figure 11 is a diagram showing the neural network structure of the CRAFT model. Figure 12 is a diagram showing the process of making BOX into SCORE in the CRAFT model. Figure 13 is a diagram showing the learning process of the CRAFT model. Figure 14 is a diagram showing the results of extracting letters using the CRAFT model.

When detecting letters using the YOLO v5 model, which is an existing letter detection model, printed fonts can be detected word by word, but character-by-character detection is difficult, and when printed and cursive fonts are mixed, detection is difficult. Accordingly, a method of detecting handwriting at the character level can be applied using an artificial intelligence learning model based on the CRAFT model.

Referring to Figure 10, the CRAFT model can detect letters included in signs, etc. that can be seen in everyday life. Compared to documents that are generally written horizontally, the structure of the letters themselves is often more complicated, but it is highly readable and can clearly distinguish each letter to convey information even at a glance.

Referring to Figure 11, the CRAFT model may have a structure similar to a simple segmentation neural network in which convolutional layers are stacked based on VGG16-BN and the resolution of the result is increased to a size equivalent to the input image through up sampling.

Here, two masks can be obtained as result values: Region Score, which represents the probability of the area corresponding to the character, and Affinity Score, which represents the probability that the preceding and following characters are connected.

Referring to FIG. 12, the annotation box can be converted into a 2D Gaussian region score and affinity score.

Referring to Figure 13, the CRAFT model can be trained on a dataset given an annotation box on a word-by-word basis. The region score and affinity score obtained from the image cut into words can be trained as pseudo ground truth on the assumption that they are more accurate than the score obtained from the original image. Here, for more accurate correction, you can create a processed image by combining random letters with a random image and train it. This can be helpful for initial learning because it can produce an accurate score.

Figure 15 is a diagram showing examples of positive samples and negative samples for metric learning. Figure 16 is a diagram showing the influence of the embedding space on loss in metric learning. Figure 17 is a diagram showing a depiction of a metric learning method.

Referring to Figures 15 to 17, the metric learning method may be a learning method that trains a network to obtain similar embedding vectors for images of the same class and dissimilar embedding vectors for images of different classes. You can set a standard anchor and learn to embed images of the same class (positive pair) closer and images of different classes (negative pairs) farther away. That is, the positive and negative samples shown in the example of FIG. 15 can be learned to be embedded so that the distance to the positive sample is close and the distance between negative samples is large, as shown in FIG. 16.

The specific implementations described in the present invention are examples and do not limit the scope of the present invention in any way. For the sake of brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connections or connection members of lines between components shown in the drawings exemplify functional connections and/or physical or circuit connections, and in actual devices, various functional connections or physical connections may be replaced or added. Can be represented as connections, or circuit connections. Additionally, if there is no specific mention such as “essential,” “important,” etc., it may not be a necessary component for the application of the present invention.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all scopes equivalent to or equivalently changed from the scope of the claims are within the scope of the spirit of the present invention. It will be said to belong to

110: Similar character extraction device 120: User terminal
1000: computer system 1010: processor
1020: Bus 1030: Memory
1031: Rom 1032: RAM
1040: User interface input device
1050: User interface output device
1060: Storage 1070: Network Interface
1080: Network

Claims (10)

Receiving first character data that is to be confirmed from the user;
extracting third character data from among second character data corresponding to all characters included in a plurality of target documents based on a preset similar character judgment condition according to the format of the first character data; and
Providing the third character data to the user terminal
Including,
The second character data is,
It includes extracting a first fragment image for an area where each of all letters included in the plurality of target documents is located using a first artificial intelligence learning model,
It further includes a label value labeling letters corresponding to each of the first fragment images using a second artificial intelligence learning model,
The first artificial intelligence learning model is,
It is an artificial intelligence learning model that extracts a fragment image for an area where letters are expected to exist within the document image corresponding to the plurality of target documents,
The second artificial intelligence learning model is,
A method of extracting similar letters using an artificial intelligence learning model, which is an artificial intelligence learning model that labels letters corresponding to the fragment image extracted by the first artificial intelligence learning model. delete According to claim 1,
The step of receiving the first character data is,
Receiving an input including a second fragment image corresponding to an area designated by the user among the document images corresponding to the plurality of target documents; or
Receiving the text input by the user as the first character data from the user terminal
A method of extracting similar characters using an artificial intelligence learning model, including. According to clause 3,
The step of extracting the third character data is,
When the first character data is the second fragment image, calculating the similarity between the second fragment image and the first fragment image using a third artificial intelligence learning model; and
Extracting third fragment images among the first fragment images, the similarity of which is greater than or equal to a preset similarity, as the third character data.
Including,
The third artificial intelligence learning model is,
A similar character extraction method using an artificial intelligence learning model, which is an artificial intelligence learning model that calculates the similarity between the first fragment image and the second fragment image. According to clause 3,
The step of extracting the third character data is,
When the first character data is the text, extracting fourth character data from among the second character data whose label value corresponding to the second character data is the same as the text; and
Extracting the fourth piece image corresponding to the fourth character data as the third character data
A method of extracting similar characters using an artificial intelligence learning model, including. According to claim 4 or 5,
The step of providing the third character data is,
generating a grid-shaped letter list using fragment images corresponding to the third letter data; and
Providing the first character data and the character list to the user terminal
A method of extracting similar characters using an artificial intelligence learning model, including. According to clause 6,
After providing the third character data,
Receiving the user's selection of a fifth piece image that is one of the letter list from the user terminal; and
Displaying a block of a preset size in an area corresponding to the fifth fragment image within a page including the fifth fragment image among the document images.
A method of extracting similar characters using an artificial intelligence learning model, including. According to clause 7,
The step of receiving the first character data is,
Receiving handwriting directly input by the user through the user terminal; and
extracting the first character data from the cursive
A method of extracting similar characters using an artificial intelligence learning model, including. According to clause 8,
The step of receiving the first character data is,
Receiving the user's selection of a sixth fragment image that is one of the first fragment images corresponding to the second character data; and
Receiving text corresponding to the labeling of the sixth fragment image as the first character data
A method of extracting similar characters using an artificial intelligence learning model, including. a memory in which at least one program is recorded; and
Processor that executes the program
Including,
The above program is
Receiving first character data that is to be confirmed from the user;
extracting third character data from among second character data corresponding to all characters included in a plurality of target documents based on a preset similar character judgment condition according to the format of the first character data; and
Providing the third character data to the user terminal
Contains instructions to perform,
The second character data is,
It includes extracting a first fragment image for an area where each of all letters included in the plurality of target documents is located using a first artificial intelligence learning model,
It further includes a label value labeling letters corresponding to each of the first fragment images using a second artificial intelligence learning model,
The first artificial intelligence learning model is,
It is an artificial intelligence learning model that extracts a fragment image for an area where letters are expected to exist within the document image corresponding to the plurality of target documents,
The second artificial intelligence learning model is,
A similar character extraction device using an artificial intelligence learning model, which is an artificial intelligence learning model that labels letters corresponding to the fragment image extracted by the first artificial intelligence learning model.

Images