TWI736230B - Image processing method, electronic equipment and storage medium - Google Patents

Abstract

The present invention relates to an image processing method, electronic equipment and storage medium. The method includes: recognizing an image and determining multiple target regions in the image, where the target region is the region where the text to be extracted is located; and determining The relative position feature between each target area in the image; determine the target feature of each target area, the target feature includes the feature of the text to be extracted; through the graph convolutional neural network, the relative The location feature and the target feature are feature extracted to obtain the extracted feature; according to the extracted feature, the field corresponding to the text to be extracted is determined.

Description

Image processing method, electronic equipment and storage medium

The present invention relates to the field of computer technology, in particular to an image processing method, electronic equipment and storage media.

The extraction of key text messages in images plays a very important role in automated office and other scenarios. For example, by extracting key text messages in images, functions such as receipt information extraction, invoice information extraction, and identity information extraction can be realized.

When extracting the text in the image, the recognized text will be mapped to different fields for subsequent operations such as structured storage and display of the text. For example, if the recognized text is "19.88 yuan", it is necessary to determine whether "19.88 yuan" corresponds to the field "total price" or the corresponding field "unit price", so that "19.88 yuan" is subsequently stored as the value of a certain field.

Usually, a template is defined in advance according to the arrangement rules of the text in the image, and the corresponding relationship between the text at a certain position and the field is defined in the template, so that the field corresponding to the recognized text at a certain position can be determined. For example, predefine the field corresponding to the text in the lower right corner of the image as "Total Price", so that it can be determined that the field corresponding to "19.88 Yuan" identified in the lower right corner of the image is "Total Price".

The present invention proposes a technical solution for image processing.

According to one aspect of the present invention, there is provided an image processing method, including: recognizing an image, determining a plurality of target regions in the image, where the target region is the region where the text to be extracted is located; determining the image The relative position feature between each target area in the image; determine the target feature of each target area, the target feature includes the feature of the text to be extracted; through the graph convolutional neural network, the relative position feature and The target feature is subjected to feature extraction to obtain the extracted feature; according to the extracted feature, the field corresponding to the text to be extracted is determined.

In the embodiment of the present invention, a graph convolutional neural network can be used to determine the field corresponding to the text to be extracted in the image based on the relative position characteristics between the target regions and the characteristics of the text to be extracted. The text extraction can be performed without relying on a fixed template. Compared with the method of text extraction based on a template, the accuracy of text extraction is higher when the text is extracted from an image without a suitable template.

In a possible implementation manner, feature extraction is performed on the relative position feature and the target feature through a graph convolutional neural network to obtain the extracted features, including: taking each of the target features as nodes of the graph, Use each of the relative position features as the edges connecting the two nodes to construct a connected graph; through the graph convolutional neural network, the connected graph is iteratively updated, and the connected graph that satisfies the convergence condition after the iterative update is taken as the extraction After the characteristics.

In the embodiment of the present invention, the constructed connected graph includes not only the target feature in the image, but also the relative position feature between the target features in the image, which can characterize the characteristics of the text in the image as a whole, so it can Improve the accuracy of key information extraction results.

When extracting features, graph convolutional neural networks can represent images in the form of connected graphs and extract features. Connected graph is composed of several nodes (Node) and edges (Edge) connecting two nodes. Edges are used to describe the relationship between different nodes. Therefore, the features extracted through the graph convolutional neural network can accurately characterize the relative position between the target regions and the features of the text to be extracted, so as to improve the accuracy of subsequent text extraction.

In a possible implementation manner, determining the field corresponding to the text to be extracted according to the extracted features includes: according to a plurality of predefined categories, the nodes in the connected graph output by the graph convolutional neural network The classification is performed to obtain the category of the node, and the preset category includes: the category of the identifier that the characterizing text belongs to the preset field, and the category of the field value of the characterizing text belonging to the preset field; and the text to be extracted is determined according to the category of the node The identifier or field value corresponding to the preset field.

In the embodiment of the present invention, by predefining the preset category as the identifier or field of the preset field, and classifying the text to be extracted according to the extracted features, the identifier or the field value of the text to be extracted corresponding to the preset field can be obtained, which improves The accuracy of text extraction is improved.

In a possible implementation manner, determining the relative position characteristics between the target areas in the image includes: determining the relative position parameters of the first target area and the second target area in the image; The parameters are characterized to obtain the relative position characteristics of the first target area and the second target area.

In a possible implementation manner, the relative position parameter includes at least one of the following: the lateral distance and the longitudinal distance of the first target area relative to the second target area; the aspect ratio of the first target area; the first target area 2. The aspect ratio of the target area; the relative size relationship between the first target area and the second target area.

In the embodiment of the present invention, the relative position parameter includes the horizontal distance and the vertical distance, the aspect ratio of the first target area, and the relative size relationship between the first target area and the second target area, so that The extraction result of key information is more accurate.

In a possible implementation manner, performing characterization processing on the relative position parameters to obtain the relative position characteristics of the first target area and the second target area includes: mapping the relative position parameters to one through a sine-cosine transformation matrix In the D-dimensional space, a D-dimensional eigenvector is obtained, where D is a positive integer; the D-dimensional eigenvector is converted into a 1-dimensional weight value through a preset weight matrix; the weight value is performed through a preset excitation function Processing to obtain relative position characteristics.

In the embodiment of the present invention, the relative position parameter can be converted into the data format required by the edge of the graph convolutional neural network through the characteristic processing, which is convenient for subsequent feature extraction through the graph convolutional neural network.

In a possible implementation manner, determining the target feature of each target area includes: determining pixel data in the target area, performing feature extraction on the pixel data to obtain visual features; determining text characters in the target area, The text characters are subjected to feature extraction to obtain character features; according to the extracted visual features and character features, the target features of the target area are determined.

In the embodiment of the present invention, considering that there will be interference factors in the image due to the camera angle, light, occlusion, etc., there will usually be more misunderstandings through text detection and recognition, that is, recognition errors may be detected. This may affect the accuracy of key information extraction. Through the extraction of visual information, the visual information is taken into account in the extraction of key information, which will reduce the influence of text misunderstanding on the extraction of key information. Even if the text recognition is wrong, the visual information will not change too much, so the combination of the two can improve the accuracy of the key information extraction results.

In a possible implementation manner, determining the target feature of the target area according to the extracted visual features and character features includes: assigning different weights to the visual features and character features; and assigning weights to the visual features and The character features are fused to obtain the target feature of the target area.

In the embodiment of the present invention, by assigning different weights to visual features and character features, the accuracy of the key information extraction results can be improved.

In a possible implementation, the method is implemented through a pre-built classification network, and the training steps of the classification network are as follows: input sample images into the classification network for processing, and obtain sample images to be extracted The first prediction category of the text, and the correspondence between each category in the first prediction category; training the classification network according to the first prediction category and the label category of the sample image, the The labeling category includes: the type of the identifier that the characterizing text belongs to the preset field, and the type of the field value of the characterizing text belonging to the preset field; training the classification according to the corresponding relationship and the corresponding relationship between the labeled text to be extracted network.

In the embodiment of the present invention, the classification network can be trained more accurately by labeling the classification of the sample image and the corresponding relationship between each classification. When extracting text, the accuracy is higher.

In a possible implementation manner, the image includes at least one of the following: a receipt image, an invoice image, and a business card image.

According to one aspect of the present invention, there is provided an image processing device including: a recognition module for recognizing an image and determining multiple target regions in the image, where the target region is where the text to be extracted is located Region; a relative position feature determination module, used to determine the relative position feature between each target area in the image; a target feature determination module, used to determine the target feature of each target area, the target feature includes The features of the text to be extracted; a graph convolution module, which is used to perform feature extraction on the relative position feature and the target feature through a graph convolutional neural network to obtain the extracted features; a field determination module, It is used to determine the field corresponding to the text to be extracted according to the extracted features.

In the embodiment of the present invention, a graph convolutional neural network can be used to determine the field corresponding to the text to be extracted in the image based on the relative position characteristics between the target regions and the characteristics of the text to be extracted. The text extraction can be performed without relying on a fixed template. Compared with the method of text extraction based on a template, the accuracy of text extraction is higher when the text is extracted from an image without a suitable template.

In a possible implementation, the graph convolution module includes: a first graph convolution sub-module and a second graph convolution sub-module, wherein the first graph convolution sub-module is used to The target feature is the node of the graph, and each of the relative position features is the edge connecting the two nodes to construct a connected graph; the second graph convolution sub-module is used to analyze the connected graph through the graph convolutional neural network The iterative update is performed, and the connected graph that meets the convergence condition after the iterative update is used as the extracted feature.

In the embodiment of the present invention, the constructed connected graph includes not only the target feature in the image, but also the relative position feature between the target features in the image, which can characterize the characteristics of the text in the image as a whole, so it can Improve the accuracy of key information extraction results.

When extracting features, graph convolutional neural networks can represent images in the form of connected graphs and extract features. A connected graph is composed of several nodes and edges connecting two nodes. The edges are used to describe the relationship between different nodes. Therefore, the features extracted through the graph convolutional neural network can accurately characterize the relative position between the target regions and the features of the text to be extracted, so as to improve the accuracy of subsequent text extraction.

In a possible implementation manner, the field determination module includes: a first field determination sub-module and a second field determination sub-module, wherein the first field determination sub-module is used to determine the sub-module according to a plurality of predefined presets. Category, to classify the nodes in the connected graph output by the graph convolutional neural network to obtain the category of the node. The preset category includes: the category representing the identifier of the text belonging to the preset field, and the category representing the text belonging to the preset field The category of the field value; the second field determination sub-module is used to determine the identifier or the field value of the preset field corresponding to the text to be extracted according to the category of the node.

In the embodiment of the present invention, by predefining the preset category as the identifier or field of the preset field, and classifying the text to be extracted according to the extracted features, the identifier or the field value of the text to be extracted corresponding to the preset field can be obtained, which improves The accuracy of text extraction is improved.

In a possible implementation, the relative position feature determination module includes: a first relative position feature determination sub-module and a second relative position feature determination sub-module, wherein the first relative position feature determination sub-module is used for Determine the relative position parameters of the first target area and the second target area in the image; the second relative position feature determination sub-module is used to characterize the relative position parameters to obtain the first target area and the second target area. The relative location characteristics of the target area.

In a possible implementation manner, the relative position parameter includes at least one of the following: the lateral distance and the longitudinal distance of the first target area relative to the second target area; the aspect ratio of the first target area; the first target area 2. The aspect ratio of the target area; the relative size relationship between the first target area and the second target area.

In the embodiment of the present invention, the relative position parameter includes the horizontal distance and the vertical distance, the aspect ratio of the first target area, and the relative size relationship between the first target area and the second target area, so that The extraction result of key information is more accurate.

In a possible implementation, the second relative position feature determination submodule is used to map the relative position parameters to a D-dimensional space through a sine-cosine transformation matrix to obtain a D-dimensional eigenvector, where D is a positive integer Transform the D-dimensional feature vector into a 1-dimensional weight value through a preset weight matrix; process the weight value through a preset excitation function to obtain a relative position feature.

In the embodiment of the present invention, the relative position parameter can be converted into the data format required by the edge of the graph convolutional neural network through the characteristic processing, which is convenient for subsequent feature extraction through the graph convolutional neural network.

In a possible implementation, the target feature determination module includes a first target feature determination sub-module, a second target feature determination sub-module, and a third target feature determination sub-module, wherein the first target feature determination sub-module The module is used to determine the pixel data in the target area, and perform feature extraction on the pixel data to obtain visual features; the second target feature determination sub-module is used to determine the text characters in the target area and compare the text characters Perform feature extraction to obtain character features; the third target feature determination sub-module is used to determine the target features of the target area according to the extracted visual features and character features.

In the embodiment of the present invention, considering that there will be interference factors in the image due to the camera angle, light, occlusion, etc., there will usually be more misunderstandings through text detection and recognition, that is, recognition errors may be detected. This may affect the accuracy of key information extraction. Through the extraction of visual information, the visual information is taken into account in the extraction of key information, which will reduce the influence of text misunderstanding on the extraction of key information. Even if the text recognition is wrong, the visual information will not change too much, so the combination of the two can improve the accuracy of the key information extraction results.

In a possible implementation manner, the third target feature determination sub-module is used to assign different weights to the visual features and character features; to fuse the weighted visual features and character features to obtain the target area Target characteristics.

In the embodiment of the present invention, by assigning different weights to visual features and character features, the accuracy of the key information extraction results can be improved.

In a possible implementation manner, the device is implemented through a pre-built classification network, and the device further includes: a first training module for inputting sample images into the classification network for processing to obtain sample images The first prediction category of the text to be extracted in the image, and the corresponding relationship between each category in the first prediction category; the second training module is used for the first prediction category and the sample image The labeling category is used to train the classification network. The labeling category includes: the category of the identifier that characterizes the text belonging to the preset field, and the category of the field value that characterizes the text belonging to the preset field; the third training module is used to Training the classification network by describing the corresponding relationship and the corresponding relationship between the labeled texts to be extracted.

In the embodiment of the present invention, the classification network can be trained more accurately by labeling the classification of the sample image and the corresponding relationship between each classification. When extracting text, the accuracy is higher.

In a possible implementation manner, the image includes at least one of the following: a receipt image, an invoice image, and a business card image.

According to one aspect of the present invention, there is provided an electronic device including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the instructions stored in the memory to execute The above method.

According to one aspect of the present invention, there is provided a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above-mentioned method when executed by a processor.

According to one aspect of the present invention, there is provided a computer program including computer readable code, and when the computer readable code is run in an electronic device, a processor in the electronic device executes instructions for implementing the above method.

In the embodiment of the present invention, a graph convolutional neural network can be used to determine the field corresponding to the text to be extracted in the image based on the relative position characteristics between the target regions and the characteristics of the text to be extracted. The text extraction can be performed without relying on a fixed template. Compared with the method of text extraction based on a template, the accuracy of text extraction is higher when the text is extracted from an image without a suitable template.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present invention. According to the following detailed description of exemplary embodiments with reference to the accompanying drawings, other features and aspects of the present invention will become clear.

Various exemplary embodiments, features, and aspects of the present invention will be described in detail below with reference to the drawings. The same reference numerals in the drawings indicate elements with the same or similar functions. Although various aspects of the embodiments are shown in the drawings, unless otherwise noted, the drawings are not necessarily drawn to scale.

The dedicated word "exemplary" here means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" need not be construed as being superior or better than other embodiments.

The term "and/or" in this article is only an association relationship describing associated objects, which means that there can be three types of relationships, for example, A and/or B can mean: A alone exists, A and B exist at the same time, and B exists alone. three conditions. In addition, the term "at least one" herein means any one or any combination of at least two of the multiple, for example, including at least one of A, B, and C, and may mean including those made from A, B, and C Any one or more elements selected in the set.

In addition, in order to better illustrate the present invention, numerous specific details are given in the following specific embodiments. Those skilled in the art should understand that the present invention can also be implemented without certain specific details. In some examples, the methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the gist of the present invention.

With the development of artificial intelligence technology, the technology for extracting key information based on images has made great progress. When extracting key information, the text in the image can be recognized. In addition, the recognition of the text will also be determined. The structured message is to determine which field in the structured data corresponds to a certain identified text, so as to facilitate subsequent operations such as structured storage and display of the identified data.

In order to improve the accuracy of key information extraction, an embodiment of the present invention provides an image processing method, which can determine the image based on the relative position feature between the target regions and the feature of the text to be extracted through the image convolutional neural network. The field corresponding to the text to be extracted in the image. This method can perform text extraction without relying on a fixed template. Compared with the method of extracting text information based on a template, the accuracy is higher when extracting text information from an image without a suitable template.

The image processing method provided by the embodiment of the present invention can be applied to the extraction of key information in the image, can realize functions such as receipt information extraction, invoice information extraction, and identity information extraction, and has high application value.

Fig. 1 shows a flowchart of an image processing method according to an embodiment of the present invention. As shown in Fig. 1, the image processing method includes:

Step S11, the image is recognized, and multiple target regions in the image are determined.

The target area is the area where the text to be extracted is located.

Since the distribution of the text to be extracted on the image is often relatively scattered, for example, there is a certain interval between the text "total price" and "19.88 yuan". Therefore, when determining the target area, you can use the text in the image to determine the target area. The distribution relationship on the image is based on the spacing between the texts, and the image is divided to obtain multiple target regions. In addition, the target area may also be divided according to other methods. The specific division method may depend on the specific application scenario of the present invention, which is not limited by the present invention.

After the target area is determined, the area where the text that constitutes a word, a sentence, or expresses a certain meaning can be determined as a target area, for example, the area where the text "total price" is to be extracted is a target Area, the area where "19.88 yuan" is located is a target area.

As for the specific method of determining the target area in the image, the present invention does not limit this.

Step S12: Determine the relative position characteristics between the target regions in the image.

The relative position feature can characterize the relative position relationship between the target areas. The specific relative position feature can be determined according to the center points of the two target areas, or according to a certain vertex of the two target areas. The present invention does not do this. limit. In addition, the relative position feature in the present invention can also be determined based on some other parameters, which will be specifically discussed in the possible implementation of the invention in the following text, and will not be repeated here.

Step S13: Determine the target feature of each target area.

The target feature includes the feature of the text to be extracted. The feature of the text to be extracted is the feature of the text to be extracted. The feature may include the visual feature of the text to be extracted as a whole, the feature of the text character of the text to be extracted, or one of the above two features.

Step S14: Perform feature extraction on the relative position feature and the target feature through the graph convolutional neural network to obtain the extracted feature.

The relative position feature and the target feature are input into the graph convolutional neural network, and feature extraction is performed to obtain the extracted features.

When extracting features, graph convolutional neural networks can represent images in the form of connected graphs and extract features. Connected graph is composed of several nodes (Node) and edges (Edge) connecting two nodes. Edges are used to describe the relationship between different nodes.

Therefore, the features extracted through the graph convolutional neural network can accurately characterize the relative position between the target regions and the features of the text to be extracted, so as to improve the accuracy of subsequent text extraction.

Step S15: Determine the field corresponding to the text to be extracted according to the extracted features.

When determining the field corresponding to the text to be extracted according to the extracted features, it can be implemented through a trained network. The network can classify the text to be extracted based on the extracted features. The classification category is used to characterize the text to be extracted. The corresponding field. After the category of the text to be extracted is determined according to the extracted features, the field corresponding to the text to be extracted is determined.

The training process of the network will be described later, so I won’t repeat it here.

According to the embodiment of the present invention, a graph convolutional neural network can be used to determine the field corresponding to the text to be extracted in the image based on the relative position feature between the target regions and the feature of the text to be extracted. The text extraction can be performed without relying on a fixed template. Compared with the method of text extraction based on a template, the accuracy of text extraction is higher when the text is extracted from an image without a suitable template.

In a possible implementation manner, determining the relative position characteristics between the target areas in the image includes: determining the relative position parameters of the first target area and the second target area in the image; characterizing the relative position parameters Through processing, the relative position characteristics of the first target area and the second target area are obtained.

The first target area and the second target area here are any two target areas in the image.

Wherein, the relative position parameters of the first target area and the second target area in the image include at least one of the following:

The horizontal distance and the vertical distance of the first target area relative to the second target area;

The aspect ratio of the first target area;

The aspect ratio of the second target area;

The relative size relationship between the first target area and the second target area.

Wherein, the horizontal and vertical distances of the first target area relative to the second target area may be the horizontal and vertical distances between the reference point of the first target area and the reference point of the second target area, and the selection of the reference point of the target area It can be the center point of the target area or a vertex of the target area. The selection of a specific reference point is not limited in the present invention.

To facilitate a clearer understanding of the process of determining relative position features, the following describes the process of determining relative position features through specific mathematical expressions. It should be noted that the specific mathematical expressions provided in the present invention are embodiments of the present invention. A possible implementation manner during specific implementation should not be understood as a limitation on the protection scope of the embodiments of the present invention.

For a text to be extracted, the target area is often rectangular, so for the text to be extracted t _i , it can be expressed as t _i => x _i , y _i , h _i , w _i , s _i >, where , x _i, y _i represent the reference point of the target area in a predetermined horizontal and vertical coordinates of the coordinate system, h _i, w _i represent the height and width of the target area, s _i represents the character of the text to be extracted.

Then, in a possible implementation manner, the lateral distance Δ x _ij and the longitudinal distance Δ y _{ij of the} first target area relative to the second target area are expressed as follows:

（1）

(1)

（2）

(2)

The first target area is the area where the text t _i is to be extracted, and the second target area is the area where the text t _{j is to be extracted.}

的表達式如下： In a possible implementation manner, the horizontal distance Δ x _ij and the vertical distance Δ y _ij can also be normalized to obtain the normalized horizontal distance and vertical distance. Specifically, the size parameter of the image can be compared to Δ x _ij and Δ y _ij are normalized. For example, when the width W of the transmitted image is normalized, the relative position parameter is obtained

The expression is as follows:

（3）

(3)

In addition, the high H of the image can also be used for normalization, which will not be repeated here.

By normalizing the horizontal distance Δ x _ij and the vertical distance Δ y _ij , the influence of the enlargement or reduction of the recognized image on the final result is reduced, and the extraction result of the key information is more accurate.

In a possible implementation manner, the aspect ratio of the first target area is w _i /h _i , and the aspect ratio of the second target area is w _j /h _j .

The relative size relationship between the first target area and the second target area may represent the relative size relationship between the size of the first target area and the size of the second target area. Since there are some specific relationships between the text sizes of certain fields, the relative size relationship between the first target area and the second target area is considered in the relative position feature, which can make the extraction result of the key information more accurate.

For example, the size of the text "address" is shorter, and the size of the text "xx city xx street xx road xx number" is longer, so the difference between the two sizes is larger; while the text "total price" and "19.88 yuan" "The gap between the sizes is smaller. Therefore, the relative size relationship of the target area can reflect the field category corresponding to the text to a certain extent.

的表達式如下： In a possible implementation, the relative size relationship

The expression is as follows:

（4）

(4)

In a possible implementation manner, the relative position parameters involved in the above formulas are integrated, and the resulting expression of the integrated relative position parameters is as follows:

（5）

(5)

In this implementation, the relative position parameter includes the normalized horizontal and vertical distances, the aspect ratio of the first target area, and the relative size relationship between the first target area and the second target area. , Can make the extraction result of key information more accurate.

In a possible implementation manner, after the relative position parameters are obtained, the relative position parameters can be characterized to obtain the relative position characteristics of the first target area and the second target area.

Characterizing the relative position parameters to obtain the relative position characteristics of the first target area and the second target area includes: mapping the relative position parameters to a D-dimensional space through a sine-cosine transformation matrix to obtain a D-dimensional feature vector, D is a positive integer; the D-dimensional feature vector is multiplied by a preset weight matrix to obtain a 1-dimensional weight value; the weight value is processed through the preset activation function to obtain the relative position feature.

The sine-cosine transformation matrix here is the transformation matrix used in Fourier sine transformation or cosine transformation.

The specific value of the preset weight matrix here can be determined through network training, and the initial value can be determined through random methods. During network training, the preset weight matrix will be tuned. The training process of the network will be described later, so I won’t repeat it here.

The preset activation function here may be, for example, a linear rectification function (Rectified Linear Unit, ReLU), and the specific activation function may depend on the actual application scenario of the present invention, which is not limited by the present invention.

In order to facilitate the understanding of the process of characterizing relative position parameters, the following specific expressions are used to illustrate the relative position characteristics e _ij after characterization. For details, please refer to formula (6):

（6）

(6)

Where, M represents a sine-cosine transformation matrix, M( r _ij ) represents that the relative position parameter r _ij is mapped to a D-dimensional space through a sine-cosine transformation matrix M, W _m is a preset weight matrix, and ReLU represents a linear rectification function .

In the embodiment of the present invention, the relative position parameter can be converted into the data format required by the edge of the graph convolutional neural network through the characteristic processing, which is convenient for subsequent feature extraction through the graph convolutional neural network.

As mentioned above, the target feature in the embodiment of the present invention may include the visual feature of the text to be extracted as a whole, or the feature of the text character of the text to be extracted.

Then, in a possible implementation manner, determining the target characteristics of each target area includes: determining the pixel data in the target area, extracting the pixel data to obtain the visual characteristics; determining the text characters in the target area, and comparing the text characters Perform feature extraction to obtain text character features; according to the extracted visual features and character features, determine the target features of the target area.

Among them, the visual features can reflect the overall visual information of the text in the target area. When extracting the visual features, it can be extracted through the Region of Interest Align (RoI Align) method. The specific method of extracting the visual features is not limited in the present invention.

In the embodiment of the present invention, considering that there will be interference factors in the image due to the camera angle, light, occlusion, etc., there will usually be more misunderstandings through text detection and recognition, that is, wrong recognition may be recognized. Text characters, which may affect the accuracy of key information extraction. Through the extraction of visual information, the visual information is taken into account in the extraction of key information, which will reduce the influence of text misunderstanding on the extraction of key information. Even if the text recognition is wrong, the visual information will not change too much, so the combination of the two can improve the accuracy of the key information extraction results.

When determining the text characters in the target area, the text characters can be recognized and extracted through text recognition technology. For example, optical character recognition technology (Optical Character Recognition, OCR) can be used to extract features of text characters to obtain text characters. The present invention does not limit the specific method of extracting text characters.

In a possible implementation manner, performing feature extraction on the text characters to obtain character features includes: mapping the text characters to a low-dimensional feature space through one-hot encoding; and then through bidirectional length Time series network (Bi-LSTM) processes the text characters in the low-dimensional feature space to obtain the feature representation of the text, that is, obtain the character features of the text to be extracted.

Through one-hot encoding, the value of discrete features (text characters) can be extended to Euclidean space. A certain value of discrete features corresponds to a point in Euclidean space, which makes the calculation between features more reasonable.

In a possible implementation manner, determining the target feature of the target area according to the extracted visual features and character features includes: assigning different weights to the visual features and character features; and assigning weights to the visual features and The character features are merged (for example, added) to obtain the target feature of the target area.

Considering that visual features and character features may have different effects on the extraction results, hereby assigning different weights to visual features and character features to improve the accuracy of the extraction results. The weights here can be optimized through network training. The specific training process is described in detail later, and will not be repeated here.

In order to facilitate the understanding of the process of characterizing text characters, the following uses specific expressions to illustrate the character characteristics after the characterization process.

For a text to be extracted, _{the process of performing feature extraction on the text character s i} to obtain the character feature t _i can be expressed as the available formula (7).

（7）

(7)

Wherein, W∈R ^{C × D} represents a hot encoded projection matrix, Bi-LSTM showing processing of text characters after hot encoded sequence via a bidirectional network length, s _i ^j denotes the j-th text characters of s _i character.

By assigning a weight α _i to the character feature t _{i and a} weight (1- α _i ) to the visual feature v _i , the target feature n _{i can} be obtained by referring to formulas (8) and (9).

（8）

(8)

（9）

(9)

Among them, W _t ∈ R ^1×Dt and W _v ∈ R ^1×Dv are one-dimensional projection matrices, which can be optimized through network training, and σ is the excitation function. U _t ∈R ^Dh×Dt and U _v ∈R ^Dh×Dt are projection parameters, which can also be optimized through network training.

After the target feature n _i and the relative location feature e _ij are obtained, the relative location feature and the target feature can be extracted through the graph convolutional neural network.

In a possible implementation method, the relative position feature and the target feature are extracted through the graph convolutional neural network, and the extracted features are obtained, including: taking each target feature as the node of the graph, and taking each relative position feature as Connect the edges of the two nodes to construct a connected graph; through the graph convolutional neural network, the connected graph is iteratively updated, and the connected graph that satisfies the convergence condition after the iterative update is used as the extracted feature.

When constructing a connected graph using the relative position feature of the target area as the edge connecting two nodes, the relative position feature will be used as a parameter of the adjacency matrix between the nodes. Of course, the adjacency matrix can also include the semantic similarity of the nodes and other things. Parameters, the present invention does not limit the specific settings of other parameters.

Please refer to FIG. 2, which is a schematic diagram of a connected graph provided by the present invention. In the connected graph, the nodes of the graph are target features, and the edge connecting two nodes is the relative position feature of the target area.

The connected graph constructed by the embodiment of the present invention includes not only the target features in the image, but also the relative position features between the target features in the image, which can characterize the characteristics of the text in the image as a whole, and therefore can improve The accuracy of key information extraction results.

After the connected graph is constructed, the connected graph can be iteratively updated through the graph convolutional neural network, and the connected graph that satisfies the convergence condition after the iterative update is used as the extracted feature. In each iteration process, the feature of any node i is updated by projecting the feature value of each node through the adjacency matrix of each node connected to node i. After multiple iterations, the feature of each node The value will no longer change with the increase of the number of iterations, that is, the eigenvalue of the node remains unchanged. At this time, it can be regarded as meeting the convergence condition, and the connected graph meeting the convergence condition can be used as the extracted feature.

For ease of understanding, the expression of the feature N ^l+1 of node N in the l +1 iteration is as follows:

（10）

(10)

Among them, N ^l is the feature of node N in the lth ^{iteration, W l} is the conversion matrix, which can be optimized through network training, A ^l is the adjacency matrix of the node, and the adjacency matrix A ^l _ij of the nodes i and j The expression is as follows:

（11）

(11)

（12）

(12)

表示歸一化參數，可以透過網路訓練進行優化得到。 Among them, (n ^l _i ) ^T represents the transposition of ^{n l} _i,

Indicates normalized parameters, which can be optimized through network training.

After the extracted features are obtained, in a possible implementation manner, according to the extracted features, determine the field corresponding to the text to be extracted, including: outputting the image convolutional neural network according to a plurality of predefined categories The nodes in the connected graph of the node are classified to obtain the category of the node. The preset category includes: the category of the identifier that characterizes the text belonging to the preset field, and the category of the field value that characterizes the text belonging to the preset field; Type, to determine the identifier or field value of the preset field corresponding to the text to be extracted.

As the recognized text, there may be text that characterizes the identifier of the preset field, and there may also be text that characterizes the field value of the preset field. The text representing the identifier of the preset field is the text in the image used to indicate which field the field value belongs to, and the field value is the specific value under the field. For example, for the preset field "Total Price", the image The recognized text "total price", "total price" or "sub total", etc., are all a specific identification of the preset field "total price"; for the recognized text "19.88 yuan" and "¥: 19.88" And so on, are the field values of the preset fields.

Therefore, for a certain preset field, two categories can be set to correspond to the preset field respectively. Among them, one category is the category that characterizes the text belonging to the preset field, and the other category is the field that characterizes the text belongs to the preset field. The category of the value. When there are multiple different preset fields, each preset field can be set to 2 categories, so there will be multiple characterizing texts belonging to the preset field’s identification category, and multiple characterizing texts belonging to the preset field. The category of the field value.

For example, when identifying a product shopping receipt, the preset fields can be set to "name", "address", "phone number", "date", "time", "product category", "product name", "Commodity unit price", "Single product total price", "Taxes", "Total total price", "Reminder", a total of 12 preset fields, then 24 categories can be preset, which respectively indicate the preset value of each preset field. Set the field identifier and the field value of each preset field. In addition, the category "Others" can be set to distinguish and extract texts that do not belong to the above categories, that is, a total of 25 categories are set.

Examples of the 25 specific preset categories in the above examples are as follows:

Name-identification; name-field value; address-identification; address-field value; telephone number-identification; telephone number-field value; date-identification; date-field value; time-identification; time-field value; product category -Identification; product category-field value; product name-identification; product name-field value; product unit price-identification; product unit price-field value; single product total price-identification; single product total price-field value; tax- Identification; Taxes-Field Value; Total Total Price-Identification; Total Total Price-Field Value; Prompt-Identification; Prompt-Field Value; Others.

In a possible implementation manner, the image processing method of the embodiment of the present invention can be implemented through a pre-built classification network, and the training steps of the classification network are as follows:

Inputting the sample image into the classification network for processing, to obtain the first prediction category of the text to be extracted in the sample image, and the corresponding relationship between each category in the first prediction category;

Train the classification network according to the first prediction category and the label category of the sample image. The label category includes: the category of the identifier that characterizes the text belonging to the preset field, and the category of the character that characterizes the text belonging to the preset field. The category of the field value;

Training the classification network according to the corresponding relationship and the corresponding relationship between the labeled annotation categories.

The classification network can be used to implement the image processing technology of the present invention. The classification network can include the graph convolutional neural network described above. In addition, in order to implement the functions of the present invention, the classification network also includes Other networks may be included, for example, a Bi-LSTM network. The networks included in the classification network of the present invention may be determined according to the specific application scenarios of the embodiments of the present invention, and the present invention does not limit this.

Please refer to Figure 3, which is a schematic structural diagram of a specific implementation of a classification network provided for this application. The network includes a target feature extraction module, a relative position feature extraction module, a convolutional network feature extraction module, and a classification model. Group. Extract the target feature of the image containing the text to be extracted through the target feature extraction module, and extract the relative position feature of the image through the relative position feature extraction module; input the target feature and relative position feature to the convolutional network feature extraction module , Perform iterative update to obtain the features extracted by the iterative; then classify the features extracted by the iterative through the classification module to obtain the predicted category of the node. Since the category characterizes the field corresponding to the text to be extracted, after the category of the text to be extracted is determined according to the extracted features, the field corresponding to the text to be extracted is determined. For the realization of the specific functions of each module, please refer to the relevant discussion in the present invention, which will not be repeated here.

In the above training process, the label category may be the preset category described above, which will not be repeated here.

When training the classification network according to the first prediction category and the label category of the sample image, the parameters in the classification network can be adjusted according to the loss of the first prediction category relative to the label category, so that The classification network has the smallest difference between the predicted category and the labeled category of the sample image.

In addition, during training, using the identification and identification value of whether the two texts belong to the same preset field is also beneficial to the classification accuracy of the classification network. For the convenience of the subsequent description, the two texts that respectively belong to the identifier and the identifier value of the same preset field are referred to as a field pair. For example, the text "total price" and "19.88 yuan" constitute a field pair.

Therefore, when training the classification network, the classification network will also output the correspondence between each category in the first prediction category, and at the same time, the correspondence between the texts will also be marked in the sample image. Then, the classification network can be trained according to the correspondence between the output of the classification network and the correspondence between the labeled texts to be extracted.

The loss function used during training may specifically be a cross entropy loss function (Cross Entropy Loss, CE), and the specific loss function may be selected according to actual requirements, which is not specifically limited in the present invention.

According to the embodiment of the present invention, the trained classification network can be used to determine the field corresponding to the text to be extracted during the extraction of key text messages. For details, please refer to the embodiment provided by the present invention. Therefore, the trained classification network has higher accuracy when extracting text from images without a suitable template.

In a possible implementation manner, the recognized image includes at least one of the following: a receipt image, an invoice image, and a business card image. Of course, in practical applications, the embodiments of the present invention can also be used to recognize other images, and the present invention does not specifically limit this.

According to the embodiment of the present invention, a graph convolutional neural network can be used to determine the field corresponding to the text to be extracted in the image based on the relative position feature between the target regions and the feature of the text to be extracted. The text extraction can be performed without relying on a fixed template. Compared with the method of text extraction based on a template, the accuracy of text extraction is higher when the text is extracted from an image without a suitable template.

According to the embodiment of the present invention, when text extraction is performed, not only the text character features in the target area are used, but also the visual features of the target area are used, which reduces the influence of misrecognition of text characters on the final classification and improves the performance of text extraction. Accuracy; In addition, by establishing the spatial position relationship between the text areas, you can deal with unseen templates without relying on pre-designed templates, and have better scalability.

In a possible implementation, the image processing method can be executed by electronic equipment such as a terminal device or a server, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a mobile phone, Wireless phones, personal digital assistants (Personal Digital Assistants, PDAs), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc., the method can be implemented by a processor calling computer-readable instructions stored in a memory. Alternatively, the method can be executed through a server.

It can be understood that the various method embodiments mentioned in the present invention can be combined with each other to form a combined embodiment without violating the principle and logic. The length is limited, and the present invention will not be repeated. Those skilled in the art can understand that, in the above method of the specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the present invention also provides image processing devices, electronic equipment, computer-readable storage media, and programs, all of which can be used to implement any image processing method provided by the present invention. For the corresponding technical solutions and descriptions, please refer to the corresponding records in the method section. ,No longer.

FIG. 4 shows a block diagram of an image processing device according to an embodiment of the present invention. As shown in FIG. 4, the image processing device 20 includes:

The recognition module 21 is used for recognizing an image and determining multiple target areas in the image, where the target area is the area where the text to be extracted is located;

The relative position feature determining module 22 is used to determine the relative position feature between each target area in the image;

The target feature determining module 23 is configured to determine the target feature of each target area, where the target feature includes the feature of the text to be extracted;

The graph convolution module 24 is configured to perform feature extraction on the relative position feature and the target feature through the graph convolution neural network to obtain the extracted feature;

The field determination module 25 is used to determine the field corresponding to the text to be extracted according to the extracted features.

In the embodiment of the present invention, a graph convolutional neural network can be used to determine the field corresponding to the text to be extracted in the image based on the relative position characteristics between the target regions and the characteristics of the text to be extracted. The text extraction can be performed without relying on a fixed template. Compared with the method of text extraction based on a template, the accuracy of text extraction is higher when the text is extracted from an image without a suitable template.

In a possible implementation manner, the graph convolution module 24 includes: a first graph convolution sub-module and a second graph convolution sub-module, wherein:

The first graph convolution submodule is used to construct a connected graph by taking each of the target features as the nodes of the graph, and using each of the relative position features as the edges connecting the two nodes;

The second graph convolution submodule is used to iteratively update the connected graph through the graph convolutional neural network, and use the connected graph that satisfies the convergence condition after the iterative update as the extracted feature.

In the embodiment of the present invention, the constructed connected graph includes not only the target feature in the image, but also the relative position feature between the target features in the image, which can characterize the characteristics of the text in the image as a whole, so it can Improve the accuracy of key information extraction results.

When extracting features, graph convolutional neural networks can represent images in the form of connected graphs and extract features. Connected graph is composed of several nodes (Node) and edges (Edge) connecting two nodes. Edges are used to describe the relationship between different nodes. Therefore, the features extracted through the graph convolutional neural network can accurately characterize the relative position between the target regions and the features of the text to be extracted, so as to improve the accuracy of subsequent text extraction.

In a possible implementation manner, the field determination module 25 includes: a first field determination sub-module and a second field determination sub-module, wherein:

The first field determines the sub-module, which is used to classify the nodes in the connected graph output by the graph convolutional neural network according to a plurality of pre-defined preset categories to obtain the category of the node, and the preset category includes: characterization The type of the identifier of the text belonging to the preset field, and the type of the field value representing the text belonging to the preset field;

The second field determination submodule is used to determine the identifier or field value of the preset field corresponding to the text to be extracted according to the type of the node.

In the embodiment of the present invention, by predefining the preset category as the identifier or field of the preset field, and classifying the text to be extracted according to the extracted features, the identifier or the field value of the text to be extracted corresponding to the preset field can be obtained, which improves The accuracy of text extraction is improved.

In a possible implementation manner, the relative position feature determination module 22 includes: a first relative position feature determination sub-module and a second relative position feature determination sub-module, wherein:

The first relative position feature determining sub-module is used to determine the relative position parameters of the first target area and the second target area in the image;

The second relative position feature determination sub-module is used to perform characterization processing on the relative position parameters to obtain the relative position features of the first target area and the second target area.

In a possible implementation manner, the relative position parameter includes at least one of the following:

The horizontal distance and the vertical distance of the first target area relative to the second target area;

The aspect ratio of the first target area;

The aspect ratio of the second target area;

The relative size relationship between the first target area and the second target area.

In the embodiment of the present invention, the relative position parameter includes the horizontal distance and the vertical distance, the aspect ratio of the first target area, and the relative size relationship between the first target area and the second target area, so that The extraction result of key information is more accurate.

In a possible implementation, the second relative position feature determination submodule is used to map the relative position parameters to a D-dimensional space through a sine-cosine transformation matrix to obtain a D-dimensional eigenvector, where D is a positive integer Transform the D-dimensional feature vector into a 1-dimensional weight value through a preset weight matrix; process the weight value through a preset excitation function to obtain a relative position feature.

In the embodiment of the present invention, the relative position parameter can be converted into the data format required by the edge of the graph convolutional neural network through the characteristic processing, which is convenient for subsequent feature extraction through the graph convolutional neural network.

In a possible implementation manner, the target feature determination module 23 includes a first target feature determination sub-module, a second target feature determination sub-module, and a third target feature determination sub-module, wherein:

The first target feature determination sub-module is used to determine pixel data in the target area, and perform feature extraction on the pixel data to obtain visual features;

The second target feature determination submodule is used to determine text characters in the target area, and perform feature extraction on the text characters to obtain character features;

The third target feature determination sub-module is used to determine the target feature of the target area according to the extracted visual features and character features.

In the embodiment of the present invention, considering that there will be interference factors in the image due to the camera angle, light, occlusion, etc., there will usually be more misunderstandings through text detection and recognition, that is, recognition errors may be detected. This may affect the accuracy of key information extraction. Through the extraction of visual information, the visual information is taken into account in the extraction of key information, which will reduce the influence of text misunderstanding on the extraction of key information. Even if the text recognition is wrong, the visual information will not change too much, so the combination of the two can improve the accuracy of the key information extraction results.

In a possible implementation manner, the third target feature determination sub-module is used to assign different weights to the visual features and character features; to fuse the weighted visual features and character features to obtain the target area Target characteristics.

In the embodiment of the present invention, by assigning different weights to visual features and character features, the accuracy of the key information extraction results can be improved.

In a possible implementation manner, the device is implemented through a pre-built classification network, and the device further includes:

The first training module is used to input the sample image into the classification network for processing to obtain the first prediction category of the text to be extracted in the sample image, and the corresponding relationship between each category in the first prediction category ；

The second training module is used to train the classification network according to the first prediction category and the label category of the sample image, and the label category includes: a category that characterizes a text belonging to an identifier of a preset field, And the type of the field value representing the text belonging to the preset field;

The third training module is used to train the classification network according to the corresponding relationship and the corresponding relationship between the labeled texts to be extracted.

In the embodiment of the present invention, the classification network can be trained more accurately by labeling the classification of the sample image and the corresponding relationship between each classification. When extracting text, the accuracy is higher.

In a possible implementation manner, the image includes at least one of the following: a receipt image, an invoice image, and a business card image.

In some embodiments, the functions or modules contained in the device provided by the embodiments of the present invention can be used to execute the methods described in the above method embodiments. For specific implementation, refer to the description of the above method embodiments. For brevity, I won't repeat it here.

An embodiment of the present invention also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above-mentioned method when executed by a processor. The computer-readable storage medium may be a volatile computer-readable storage medium or a non-volatile computer-readable storage medium.

An embodiment of the present invention also provides an electronic device, including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

The embodiment of the present invention also provides a computer program product, including computer readable code. When the computer readable code runs on the device, the processor in the device executes the image processing method provided in any of the above embodiments. instruction.

The embodiment of the present invention also provides another computer program product for storing computer-readable instructions, which when executed, cause the computer to perform the operation of the image processing method provided in any of the above-mentioned embodiments.

Electronic devices can be provided as terminals, servers, or other types of devices.

FIG. 5 shows a block diagram of an electronic device 800 according to an embodiment of the present invention. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and other terminals.

5, the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor The device component 814, and the communication component 816.

The processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the foregoing method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations in the electronic device 800. Examples of these data include instructions for any application or method operated on the electronic device 800, contact data, phone book data, messages, images, videos, etc. The memory 804 can be realized by any type of volatile or non-volatile storage devices or their combination, such as static random access memory (SRAM), electronically erasable programmable read-only memory (EEPROM), and erasable In addition to programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, floppy disk or optical disk.

The power supply component 806 provides power for various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, sliding, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or sliding action, but also detect the duration and pressure related to the touch or sliding operation. In some embodiments, the multimedia component 808 includes a front lens and/or a rear lens. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front lens and/or the rear lens can receive external multimedia data. Each front lens and rear lens can be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC). When the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and the surrounding interface modules. The surrounding interface modules may be keyboards, mice, buttons, and so on. These buttons may include, but are not limited to: home button, volume button, start button, and lock button.

The sensor component 814 includes one or more sensors for providing the electronic device 800 with various aspects of state evaluation. For example, the sensor component 814 can detect the on/off state of the electronic device 800 and the relative positioning of the components. For example, the component is the display and the keypad of the electronic device 800. The sensor component 814 can also detect the electronic device 800 or The position of a component of the electronic device 800 changes, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration/deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects when there is no physical contact. The sensor component 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related messages from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), On-site programmable logic gate array (FPGA), controller, microcontroller, microprocessor or other electronic components are used to implement the above methods.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as the memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to complete the above method.

FIG. 6 shows a block diagram of an electronic device 1900 according to an embodiment of the present invention. For example, the electronic device 1900 may be provided as a server. 6, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions that can be executed by the processing component 1922, such as application programs. The application program stored in the memory 1932 may include one or more modules each corresponding to a set of commands. In addition, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input and output (I/O) Interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for enabling the processor to implement various aspects of the present invention.

The computer-readable storage medium may be a tangible device that can hold and store instructions used by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), removable Programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital audio-visual disc (DVD), memory card, floppy disk , Mechanical encoding equipment, such as a punch card on which instructions are stored or a raised structure in a groove, and any suitable combination of the above. The computer-readable storage medium used here is not interpreted as the instantaneous signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (for example, light pulses through fiber optic cables), or through wires The transmitted electrical signal.

The computer-readable program instructions described here can be downloaded from a computer-readable storage medium to various computing/processing devices, or downloaded to an external computer via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network Or external storage device. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. The network interface card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for computer-readable storage in each computing/processing device In the media.

The computer program instructions used to perform the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or any of one or more programming languages. Combined source code or object code, the programming language includes object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions can be executed entirely on the user's computer, partly on the user's computer, executed as a stand-alone software package, partly on the user's computer and partly executed on the remote computer, or entirely on the remote computer or Execute on the server. In the case of a remote computer, the remote computer can be connected to the user’s computer through any kind of network-including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using the Internet). Internet service provider to connect via the Internet). In some embodiments, the electronic circuit is personalized by using the status information of the computer-readable program instructions, such as programmable logic circuit, field programmable logic gate array (FPGA) or programmable logic array (PLA), The electronic circuit can execute computer-readable program instructions to realize various aspects of the present invention.

Herein, various aspects of the present invention are described with reference to flowcharts and/or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart and/or block diagram and the combination of each block in the flowchart and/or block diagram can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to the processors of general-purpose computers, dedicated computers, or other programmable data processing devices to produce a machine that allows these instructions to be executed by the processors of the computer or other programmable data processing devices At this time, a device that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram is produced. It is also possible to store these computer-readable program instructions in a computer-readable storage medium. These instructions make the computer, the programmable data processing device and/or other equipment work in a specific manner, so that the computer-readable medium storing the instructions is It includes an article of manufacture, which includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowchart and/or block diagram.

It is also possible to load computer-readable program instructions on a computer, other programmable data processing device, or other equipment, so that a series of operation steps are executed on the computer, other programmable data processing device, or other equipment to generate a computer realization In this way, instructions executed on a computer, other programmable data processing device, or other equipment realize the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions, and operations of the system, method, and computer program product according to multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, program segment, or part of an instruction, and the module, program segment, or part of an instruction includes one or more logic for implementing the specified Executable instructions for the function. In some alternative implementations, the functions marked in the block may also occur in a different order than the order marked in the drawings. For example, two consecutive blocks can actually be executed basically in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, and the combination of blocks in the block diagram and/or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions. It can be realized, or it can be realized by a combination of dedicated hardware and computer instructions.

The computer program product can be implemented through hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc. Wait.

The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Without departing from the scope and spirit of the illustrated embodiments, many modifications and changes are obvious to those of ordinary skill in the art. The choice of terms used herein is intended to best explain the principles, practical applications, or improvements to technologies in the market of the embodiments, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

20······· Image Processing Device 21······· Identification Module 22······· Relative Position Feature Determination Module 23······· Target feature determination module 24······· Graph Convolution Module 25·······Field Confirmation Module 800····· Electronic Equipment 802·····Processing components 804·····Memory 806····· Power Supply 808····· Multimedia Components 810····· Audio Components 812····· Input/Output Interface 814····· Sensor Assembly 816····· Communication Components 820····· Processor 1900··· Electronic Equipment 1922··· Handling components 1926··· Power Supply 1932··· Memory 1950··· Network Interface 1958··· Input/Output Interface

The drawings here are incorporated into the specification and constitute a part of the specification. These drawings show embodiments in accordance with the present invention and are used together with the specification to illustrate the technical solution of the present invention. Fig. 1 shows a flowchart of an image processing method according to an embodiment of the present invention; Figure 2 shows a schematic structural diagram of a connected graph according to an embodiment of the present invention; Figure 3 shows a schematic structural diagram of a classification network according to an embodiment of the present invention; Fig. 4 shows a block diagram of an image processing apparatus according to an embodiment of the present invention; Figure 5 shows a block diagram of an electronic device according to an embodiment of the present invention; and Fig. 6 shows a block diagram of an electronic device according to an embodiment of the present invention.

Claims (11)

An image processing method, including: recognizing an image, determining multiple target regions in the image, where the target region is the region where the text to be extracted is located; determining the distance between the target regions in the image Determine the target features of each of the target regions, the target features include the features of the text to be extracted; through the graph convolutional neural network, the image is represented in the form of a connected graph, and the Feature extraction is performed on the relative position feature and the target feature to obtain the extracted feature; according to the extracted feature, the field corresponding to the text to be extracted is determined. The method according to claim 1, wherein the feature extraction is performed on the relative position feature and the target feature through a graph convolutional neural network to obtain the extracted features, including: taking each of the target features as a graph Nodes, using each of the relative position features as the edges connecting the two nodes to construct a connected graph; through the graph convolutional neural network, the connected graph is iteratively updated, and the connected graph that satisfies the convergence condition after the iterative update As the extracted feature. The method according to claim 2, wherein, according to the extracted features, determining the field corresponding to the text to be extracted includes: according to a plurality of pre-defined preset categories, the connected graph output by the graph convolutional neural network The nodes are classified to obtain the category of the node. The preset category includes: the category representing the identifier of the text belonging to the preset field, and the table The signature text belongs to the category of the field value of the preset field; according to the category of the node, it is determined that the text to be extracted corresponds to the identification or field value of the preset field. The method according to claim 1, wherein determining the relative position characteristics between the target areas in the image includes: determining the relative position parameters of the first target area and the second target area in the image; The relative position parameters are characterized to obtain the relative position characteristics of the first target area and the second target area. The method according to claim 4, wherein the relative position parameter includes at least one of the following: the horizontal distance and the vertical distance of the first target area relative to the second target area; the aspect ratio of the first target area; The aspect ratio of the second target area; the relative size relationship between the first target area and the second target area. The method according to any one of claim 4 or 5, wherein, performing characterization processing on the relative position parameter to obtain the relative position characteristics of the first target area and the second target area includes: transmitting the relative position parameter through The sine-cosine transformation matrix is mapped to a D-dimensional space to obtain a D-dimensional eigenvector, where D is a positive integer; the D-dimensional eigenvector is converted into a 1-dimensional weight value through a preset weight matrix; The weight value is processed through the preset activation function to obtain the relative position feature. The method according to claim 1, wherein determining the target feature of each target area includes: determining pixel data in the target area, performing feature extraction on the pixel data to obtain visual features; determining text characters in the target area , Perform feature extraction on the text characters to obtain character features; determine the target feature of the target area according to the extracted visual features and character features. The method according to claim 7, wherein determining the target feature of the target area according to the extracted visual features and character features includes: assigning different weights to the visual features and character features; and assigning weights to the visual The feature and character feature are merged to obtain the target feature of the target area. The method according to claim 1, wherein the method is implemented through a pre-built classification network, and the training steps of the classification network are as follows: input the sample image into the classification network for processing, and obtain the sample image The first prediction category of the text to be extracted, and the correspondence between each category in the first prediction category; training the classification network according to the first prediction category and the label category of the sample image, The labeling category includes: the category of the identifier that the characterizing text belongs to the preset field, and the category of the field value of the characterizing text belonging to the preset field; According to the corresponding relationship and the corresponding relationship between the labeled texts to be extracted, the classification network is trained. An electronic device, comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the instructions stored in the memory to execute any of the request items 1 to 9 The method described in one item. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the method described in any one of request items 1 to 9.

Images