WO2024041032A1 - Method and device for generating editable document based on non-editable graphics-text image - Google Patents

Abstract

The present invention relates to a method and device for generating an editable document based on a non-editable graphics-text image. The method comprises: acquiring a non-editable graphics-text image; extracting contour features and text features from the non-editable graphics-text image; generating initial structured data according to the contour features and the text features; determining a relationship between two elements in the non-editable graphics-text image on the basis of a pre-trained element relationship classification model, the contour features and the text features, the pre-trained element relationship classification model being determined by training on the basis of a data set composed of contour features and/or text features of the two elements and a relationship tag of the two elements; supplementing the relationship between two elements to the initial structured data to obtain final structured data; and generating an editable document of the non-editable graphics-text image on the basis of the final structured data. On this basis, non-editable graphics-text data can be efficiently and accurately converted into an editable document.

Description

Method and device for generating editable documents based on non-editable graphics and text images Technical field

The present invention relates to the field of computer technology, and in particular to methods and devices for generating editable documents based on non-editable graphics and text images.

Background technique

Since graphic and text images cannot be edited, graphic and text data cannot be quickly changed based on existing elements. If you want to save uneditable graphic and text data as an editable document, you still need to manually create it from scratch, resulting in low utilization of graphic data and time-consuming and laborious manual production. Therefore, how to efficiently and accurately convert non-editable graphics and text images into editable documents is a technical problem that currently needs to be solved.

Contents of the invention

The present invention provides a method and device for generating editable documents based on non-editable graphic and textual images to solve the problem in the prior art that saving uneditable graphic and textual data into editable documents is time-consuming and inefficient. Problem to achieve efficient and accurate conversion of non-editable graphics and text images into editable documents.

A method for generating editable documents based on non-editable graphic and text images. The method includes: obtaining non-editable graphic and text images; extracting contour features from the non-editable graphic and text images; and extracting contour features from the non-editable graphic and text images. Extract text features from edited graphic images; wherein the outline features include the shape, position, size and color within the outline; the text features include text box coordinates, text content, text color and font size ; Generate initial structured data based on the outline features and the text features; determine two of the non-editable graphic and text images based on the pre-trained element relationship classification model and the outline features and the text features. The relationship between elements; wherein, the pre-trained element relationship classification model is trained based on a data set composed of the outline features or text features of the two elements, the outline features and/or text features, and the relationship labels of the two elements. Determine; supplement the initial structured data based on the relationship between the two elements to obtain the final structured data; generate an editable image corresponding to the non-editable graphic image based on the final structured data. Edit the document.

In one embodiment, the pre-trained element relationship classification model includes multiple pre-trained binary classification models. Correspondingly, the pre-trained element relationship classification model and the contour features and the Determining the relationship between two elements in the non-editable graphic and text-based image includes: based on each pre-trained two-classification model and the contour feature and the text feature, determining the non-editable text feature. The classification result of the relationship between every two elements in the graphic image; based on the classification result with the largest probability value among the multiple determined classification results, determine the relationship between every two elements in the non-editable graphic image Final classification result.

In one embodiment, the process of determining a pretrained element relationship classification model after training based on a data set consisting of outline features and/or text features of two elements and relationship labels of the two elements includes: obtaining multiple The outline features and text features of non-editable graphic and text images; determine the data set based on the outline features and/or text features corresponding to every two elements in each non-editable graphic and text image; and based on every two elements Determine the relationship label of each two elements; based on the relationship label of each two elements and their corresponding two-classification model, determine the samples of the data set as positive samples and negative samples; train the correspondence based on the positive samples and negative samples The two-classification model is obtained to obtain the pre-trained two-classification model.

In one embodiment, determining the text features in the non-editable graphic and text-based image based on text detection and text recognition methods includes: determining the non-editable graphic and text-based image based on a preset text box detection algorithm. The text boxes included in the image and their coordinates; based on the preset text recognition algorithm, determine the text content included in each text box; determine each text based on the coordinates of each text box and the pixel histogram in each text box The color of the text in the box; based on the coordinates of each text box, determines the font size within the text box.

In one embodiment, determining the contour features in the non-editable graphic and text-based image based on the contour detection and shape recognition method includes: determining the non-editable graphic and text-based image based on a preset contour detection algorithm. At least one contour included in the image; a shape recognition model based on a pre-trained residual neural network, identifying the shape of each contour in the at least one contour; determining a relative shape of the shape based on a minimum circumscribed rectangle size of the shape of each contour The size of each contour is determined based on the coordinates of the preset position of the shape of each contour; the color of each contour is determined based on the color corresponding to the centroid coordinates of each contour.

In one embodiment, based on a preset contour detection algorithm, it is determined that the non-editable At least one contour included in the graphic image; including: based on a preset contour detection algorithm, determining a set of contours included in the non-editable graphic image; according to the coordinates of each contour and the text box , filter out the contours that coincide with the text box; determine at least one contour based on the remaining contours.

In one embodiment, based on the final structured data, generating an editable document corresponding to the non-editable graphic image includes: obtaining and displaying the structured data; based on the final structured data data, generate an image corresponding to the outline feature and a text corresponding to the text feature at a corresponding position on the canvas, and determine an initial editable document corresponding to the non-editable graphic and text image.

In one embodiment, after creating an image corresponding to the outline feature and a text corresponding to the text feature at a corresponding position on the canvas based on the final structured data, the method further includes: In response to the user's operation, the initial editable document is added, modified or deleted.

The present invention also provides a device for generating editable documents based on non-editable graphic and textual images. The device includes: an acquisition module for acquiring non-editable graphic and textual images; and a determining module for generating the non-editable graphic and textual images from the non-editable graphic and textual images. Extract contour features from the graphic and text images; extract text features from the non-editable graphic and text images; wherein the contour features include the shape, position, size and color within the contour; the text features including text box coordinates, text content, text color and font size; the first generation module is used to generate structured data according to the outline features and text features; the second generation module is used to generate structured data based on the final structured data , generating an editable document corresponding to the non-editable graphic image.

The present invention also provides a computer device, including a memory and a processor. Computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by the processor, they cause the processor to execute the above-mentioned non-editable-based method. Steps of a method for generating editable documents from graphics and text images.

The present invention also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, they cause one or more processors to execute the above-mentioned non-editable graphics-based image generation. Steps to edit a document.

The above-mentioned method and device for generating editable documents based on non-editable graphics and text images can determine the non-editable graphics and text types by determining the outline features and text features of the non-editable graphics and text images. The elements and corresponding attributes contained in the image are generated, and initial structured data is generated based on these elements and corresponding attributes, and then the relationship between the two elements is further determined based on the pre-trained element relationship classification model, and the two elements are The relationship between them is supplemented to the initial structured data to obtain the final structured data, and then based on the final structured data, editable documents corresponding to the non-editable graphics and text images are generated, thereby no longer relying on Manually convert non-editable graphic images into editable documents, thereby efficiently and accurately converting non-editable graphic images into editable documents.

Description of drawings

Figure 1 is one of the flow diagrams of a method for generating editable documents based on non-editable graphics and text images provided by the present invention;

Figure 2 is a schematic flow chart of a method for generating editable documents based on non-editable graphics and text images provided by the present invention.

Figure 3 is the third schematic flowchart of the method for generating editable documents based on non-editable graphics and text images provided by the present invention;

Figure 4 is a schematic flow chart of the fourth method for generating editable documents based on non-editable graphics and text images provided by the present invention;

Figure 5 is a schematic flow chart of a method for generating editable documents based on non-editable graphic images provided by the present invention.

Figure 6 is a schematic flow chart of the method for generating editable documents based on non-editable graphic images provided by the present invention.

Figure 7 is a schematic diagram of a graphic image provided by the present invention;

Figure 8 is a schematic diagram of the initial structured data provided by the present invention;

Figure 9 is a schematic diagram of the final structured data provided by the present invention;

Figure 10 is a schematic diagram of the display interface of the graphic editor provided by the present invention;

Figure 11 is a schematic framework diagram of a device for generating editable documents based on non-editable graphics and text images provided by the present invention;

Figure 12 is a schematic diagram of the electronic device provided by the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention.

It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of this disclosure should have the usual meanings understood by those with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprising" mean that the elements or things appearing before the word include the elements or things listed after the word and their equivalents, without excluding other elements or things. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right", etc. are only used to express relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

In order to facilitate understanding, the technical terms involved in this application are explained.

(1)Connected text box network (connectionist text proposal network, CTPN)

CTPN is a text detection algorithm proposed in ECCV 2016. CTPN is a deep neural network that combines a convolutional neural network (CNN) and a long short serm memory network (LSTM). It can effectively detect horizontally distributed text in complex scenes.

In the present invention, CTPN is used to detect text boxes in non-editable graphic images.

(2) Non maximum suppression (NMS)

Non-maximum suppression is to suppress elements that are not maximum values and search for local maximum values. In the common object detection algorithms in recent years (such as rcnn, sppnet, fast-rcnn, faster-rcnn, etc.), many rectangular boxes that may be objects are eventually found from a picture, and then each rectangular box is To do category classification probability. NMS is used to filter out a part of the rectangular frames from the plurality of rectangular frames. In the present invention, NMS is used to filter the text boxes in the detected non-editable graphic and text images, so that the text boxes detected by CTPN are closer to the text boxes in the original non-editable graphic and text images.

(3) Text line construction algorithm

In the present invention, it is used to connect text boxes filtered by NMS to form a text detection box.

(4)OpenCV

OpenCV is a cross-platform computer vision and machine learning software library released under the Apache2.0 license (open source) and can run on Linux, Windows, Android and Mac OS operating systems. OpenCV is lightweight and efficient. It consists of a series of C functions and a small number of C++ classes. It also provides interfaces in Python, Ruby, MATLAB and other languages, and implements many common algorithms in image processing and computer vision.

(5) Convolutional recurrent neural network structure (convolutional recurrent neural network, CRNN)

It is mainly used to recognize text sequences of variable length end-to-end without cutting individual words first. Instead, it converts text recognition into a timing-dependent sequence learning problem, which is image-based sequence recognition.

(6) Connectionist temporal classification (CTC)

CTC is used to solve the alignment problem of input data and given labels. It can be used to perform end-to-end training and output sequence results of variable length. It can be understood that since there are certain character intervals between words in text images of natural scenes, and text images may also have problems such as deformation, the same text will have different expressions, and the same text will appear in the text recognition results. Therefore, the CTC model can be used to remove spaced characters and repeated characters in text recognition results.

(7)Structured data

Structured data, also called row data, is data logically expressed and implemented by a two-dimensional table structure. It strictly follows data format and length specifications and is mainly stored and managed through relational databases.

It can be understood that since the elements in graphic data come in various forms and each type of elements has different attributes such as shape, size, font color, etc., therefore, it is necessary to manually convert uneditable graphic data into editable graphic data. It is relatively time-consuming and energy-consuming. Therefore, the present invention provides a method for generating editable documents based on non-editable graphic and textual images, solving the problem that in the past graphic and textual data can only be saved as uneditable images, and at the same time solving the problem of using When readers try to edit graphic data in documents, they often need to create it from scratch to efficiently convert non-editable graphic data into editable graphic data.

The method and device for generating editable documents based on non-editable graphics and text images provided by the present invention will be described below with reference to the accompanying drawings.

Figure 1 is a schematic flowchart of a method for generating editable documents based on non-editable graphics and text images provided by the present invention. It can be understood that the method of generating an editable document based on a non-editable graphic image can be executed by a device that generates an editable document based on a non-editable graphic image. The device for generating an editable document based on a non-editable graphic image may be a computer device.

As shown in Figure 1, in one embodiment, a method of generating editable documents based on non-editable graphic images is proposed, which may include the following steps:

Step 110: Obtain non-editable graphic and text images.

Among them, graphic images include images and text. Among them, images and text have related properties. For example, an image has properties such as the shape and position and size of the outline and the color within the outline, and text has text box coordinates, text content, text color, and font size. Specifically, for graphics and text images, please refer to Figure 7 below, which will not be described in detail here.

Non-editable graphic images are images in which the shape, color, and font size and color of the image or graphics cannot be directly adjusted, such as images in PNG or JPG format.

Step 120: Extract contour features from non-editable graphic and text images; extract text features from non-editable graphic and text images.

Among them, the outline features include the shape, position, size and color of the outline; the text features include text box coordinates, text content, text color and font size.

Among them, the contour detection and shape recognition methods are used to detect the relevant attribute characteristics of the contour included in the non-editable graphic image, such as shape, position, size and color within the contour. Text detection and recognition methods are used to detect relevant attribute characteristics of text included in non-editable graphic images, such as text box coordinates, text content, text color and font size.

It can be understood that by determining the outline features and text features, the elements and corresponding attributes contained in the graphic image can be determined, and initial structured data can be generated based on these elements and corresponding attributes.

Step 130: Generate initial structured data based on contour features and text features.

It can be understood that the initial structured data is generated based on the contour features and text features to facilitate subsequent root analysis. Determine the final structured data based on the initial structured data.

Specifically, the outline features and text features can be saved in the form of a dictionary. For example, dictionary S = {contour C1: (position coordinates, color attribute value, shape attribute value, shape size), contour C2: (position coordinates, color attribute value, shape attribute value, shape size), ..., contour Ck (position coordinates, color attribute value, shape attribute value, shape size)}. For another example, image shape J={image 1: (image path, shape position coordinates, shape size), image 2: (image path, shape position coordinates, shape size),..., image z: (image path, Shape position coordinates, shape size)}. Among them, contour C1, contour C2, contour Ck and image 1, image 2 or image z are one element respectively.

Step 140: Determine the relationship between two elements in the non-editable graphic and text image based on the pre-trained element relationship classification model and the contour features and text features.

Wherein, the pre-trained element relationship classification model is determined after training based on a data set composed of outline features or text features of two elements, outline features and/or text features, and relationship labels of the two elements. Among them, the elements are outlines or texts included in non-editable graphic images, such as graphics, images, text boxes or text content.

In addition, the relationship tag of two elements is used to identify the relationship between the two elements, which can be, for example, surrounding, containing, associated, or independent. Surrounding can be understood as an element surrounding another element, such as text above an arrow. Containment can be understood as an element containing another element, such as text within a certain outline. Association can be understood as contact between two elements, such as arrows connected to text boxes, arrows connected to other shapes, etc. Independence can be understood as relationships other than the above three types of relationships.

It can be understood that the initial structured data includes features such as the shape, position, size, and color within the outline, text box coordinates, text content, text color, and font size, in order to more comprehensively and accurately represent non-editable graphics and text. The relationship between elements in class images can be further determined based on the pre-trained element relationship classification model. The relationship between two elements in non-editable graphic and text class images can be further determined, and the relationship between the two elements can be supplemented to the above From the initial structured data, the final structured data is obtained.

Step 150: Supplement the initial structured data based on the relationship between the two elements to obtain the final structured data.

Among them, the final structured data can be referred to Figure 9 below, which will not be described in detail here.

Step 160: Generate an editable document corresponding to the non-editable graphic and text image based on the final structured data.

The editable document corresponding to a non-editable graphic image is an image in which the shape, color, and font size and color can be directly adjusted, such as an image in visio format.

The method provided by the present invention for generating editable documents based on non-editable graphic and text images can determine the elements contained in the non-editable graphic and text images and their corresponding elements by determining the outline features and text features of the non-editable graphic and text images. attributes, and generate initial structured data based on these elements and corresponding attributes, and then further determine the relationship between the two elements based on the pre-trained element relationship classification model, and supplement the relationship between the two elements to the initial From the structured data, the final structured data is obtained, and then based on the final structured data, editable documents corresponding to non-editable graphic and text images are generated, thereby no longer relying on manual conversion of non-editable graphic and text images into Editable documents, thereby efficiently and accurately converting non-editable graphics and text images into editable documents.

In one embodiment, the pre-trained element relationship classification model includes multiple binary classification models. Correspondingly, as shown in Figure 2, based on the pre-trained element relationship classification model as well as contour features and text features, it is determined that the inability to Editing the relationship between two elements in a graphic image includes the following steps:

Step 210: Based on each pre-trained binary classification model and the contour features and text features, determine the classification result of the relationship between the two elements in the non-editable graphic and text image.

Among them, the binary classification model is used to determine whether the relationship between two elements is a relationship corresponding to a preset relationship label. For example, it can be used to determine whether the relationship between two elements is an inclusion relationship. Among them, the two-classification model can be a support vector machine (SVM).

The classification result may be, for example, that the probability that the relationship between the two elements is an inclusion relationship is 90%, and the probability that the relationship between the two elements is not an inclusion relationship is 10%.

In addition, assuming that a non-editable graphic image has t outlines and c texts, it is necessary to calculate the corresponding relationships of [(t+c)*(t+c-1)/2] element combinations. classification results.

Step 220: Based on the classification result with the largest probability value among the determined multiple classification results, determine the final classification result of the relationship between each two elements in the non-editable graphic and text image.

It can be understood that there can be a two-classification model corresponding to each relationship label. Therefore, multiple relationship labels correspond to multiple binary classification models. Therefore, to determine the relationship between two elements, one can Among the classification results based on multiple binary classification models, the classification result with the largest probability value is determined as the final classification result. For example, the classification results of multiple binary classification models for the relationship between two elements of the same group are: the probability of being a contained relationship is 90%, the probability of being an associated relationship is 20%, and the probability of being an independent relationship is 15%. , the probability of being a wraparound relationship is 10%, then the classification result corresponding to 90% is determined to be the final classification result, that is, the relationship between two elements is determined to be an inclusion relationship.

In one of the embodiments, as shown in Figure 3, the process of determining a pre-trained element relationship classification model after training based on a data set consisting of outline features and/or text features of two elements and relationship labels of two elements. , including the following steps:

Step 310: Obtain the outline features and text features of multiple non-editable graphic and text images.

Among them, the process of obtaining the outline features and text features of non-editable graphics and text images can be referred to the relevant descriptions in the previous article. For the sake of simplicity, they will not be described again here.

In addition, after obtaining the contour features and text features, the feature values in the contour features and text features can be normalized, and One-hot one-hot encoding is used to encode the feature values.

Step 320: Determine a data set based on the contour features and/or text features corresponding to each two elements in each non-editable graphic image; and determine the relationship label of each two elements based on the relationship between each two elements.

It can be understood that in order to determine the relationship between every two elements in each non-editable graphic image, the contour features and/or text features corresponding to each two elements contained in each non-editable graphic image can be used as samples, and the relationship between each two elements is used as the label of the corresponding sample to determine the data set for training the two-classification model.

In addition, after the relationship label of each two elements, One-hot one-hot encoding can be used to encode the relationship label to facilitate subsequent processing.

Step 330: Based on the relationship labels of each two elements and their corresponding binary classification models, the samples of the data set are determined as positive samples and negative samples.

It can be understood that there can be a two-classification model corresponding to each relationship label. Therefore, the corresponding two-classification model can be, for example: a two-classification model used to determine whether the relationship between two elements is a surround relationship, or a two-classification model used to determine whether the relationship between two elements is an inclusion relationship. model, or a binary classification model used to determine whether the relationship between two elements is an association relationship, or a A binary classification model used to determine whether the relationship between two elements is independent.

Specifically, samples with the same relationship label category corresponding to the two-classification model can be classified as positive samples, and samples with different relationship label categories corresponding to the two-classification model can be classified as negative samples. For example, for a binary classification model that is used to determine whether the relationship between two elements is an association relationship, a sample in which the relationship between the two elements is an association relationship is determined as a positive sample, and a sample in which the relationship between the two elements is a surrounding relationship is determined as a positive sample. Samples with relationships, inclusions or independent relationships are determined as negative samples.

It can also be understood that since there can be a two-classification model corresponding to each relationship label, the positive samples and negative samples corresponding to the two-classification model corresponding to each relationship label are different.

Step 340: Train the corresponding two-classification model based on the positive samples and negative samples to obtain a pre-trained two-classification model.

Specifically, the positive samples and negative samples can be divided into training sets, verification sets, and test sets respectively according to preset proportions, and then the corresponding two-classification model is trained based on the training set to obtain a pre-trained two-classification model.

It can be understood that, combined with the relevant description in Figure 2, after obtaining the pre-trained two-classification model, the relationship between each two elements in the non-editable graphic and text image can be determined based on the pre-trained two-classification model.

In one embodiment, as shown in Figure 4, determining text features in non-editable graphic and text images based on text detection and text recognition methods includes the following steps:

Step 410: Determine the text box and its coordinates included in the non-editable graphic image based on a preset text box detection algorithm.

Specifically, the CTPN text detection network can be used as the preset text box detection algorithm to detect text on non-editable graphic images to obtain the initial text box, and then use the NMS algorithm to filter out the redundant text in the initial text box. boxes, and finally combined with the text line construction algorithm, the text boxes belonging to the same text sequence are connected to obtain the connected text boxes.

It is understandable that there may be no text in non-editable graphic images. Therefore, if the corresponding initial text box is not detected based on the preset text box detection algorithm, there is no need to perform subsequent correlation on the initial text box. Just process it.

It can also be understood that when using the CTPN text detection network to text non-editable graphic and text images, Before this detection, non-editable graphics and text images can be preprocessed to facilitate subsequent detection of text boxes and text. For example, non-editable graphics and text images can be grayscaled and binarized. Among them, grayscale can be achieved through the OpenCV function cv2.cvtColor, and binarization can be achieved through the OpenCV function cv2.threshold.

In addition, after obtaining the text box, in order to subsequently generate a text box that conforms to the tilt angle and area of the text, the connected text boxes need to be corrected, and finally a text box of p text boxes contained in the non-editable graphic image is obtained. Coordinate set T = {T ₁ , T ₂ ,..., T _t }. It can be understood that the connected text boxes can be expressed in the form of coordinates corresponding to the text boxes, for example, the coordinates of the diagonal lines corresponding to the text boxes. Therefore, among them, the elements T _t in the coordinate set T are recorded in the form of (x ₁ , y ₁ , _{x 2} , y ₂ ), and x ₁ and x ₂ in (x ₁ , y ₁ , x 2 , y _{2 )} represents the abscissa coordinate of the diagonal line corresponding to the text box, and y ₁ and y ₂ represent the ordinate coordinate of the diagonal line corresponding to the text box.

Step 420: Determine the text content included in each text box based on a preset text recognition algorithm.

Specifically, the CRNN text recognition algorithm can be used as the preset text recognition algorithm to identify the text content included in each text box. First, the CRNN text recognition algorithm can use the MobileNetv3 network to extract features from the image area corresponding to the input text detection box set T to obtain a feature map. Among them, the height of the input image can be 32, and the width can be any number greater than 0. After the MobileNetv3 network, the height of the feature map becomes 1. Secondly, the Im2Seq network layer can be used in the CRNN text recognition algorithm to transform the resulting feature map into the shape of a feature sequence to input into the subsequent sequence model. Then, the obtained feature sequence is input into the BiLSTM model. The BiLSTM model learns the feature sequence and uses the fully connected layer to obtain the model prediction label distribution results. Among them, the predicted label distribution result includes the recognized text and the corresponding probability. Finally, the predicted label distribution result can also be input into the CTC layer and decoded to obtain the text content recognized by the coordinate set T of the text box.

Step 430: Determine the text color in each text box based on the coordinates of each text box and the pixel histogram in each text box.

Specifically, according to the coordinate set T of the text box, in the non-editable graphic image, the pixel histogram in each text box can be obtained by combining the coordinates of each text box, thereby obtaining the text color value.

Step 440: Determine the size of the font in the text box based on the coordinates of each text box.

It can be understood that the height of each text box can be determined according to the coordinates of each text box, so that Determine the relative size of fonts within text boxes based on the height of each text box.

In one embodiment, as shown in Figure 5, based on the contour detection and shape recognition method, determining the contour features in the non-editable graphic image includes the following steps:

Step 510: Determine at least one contour included in the non-editable graphic image based on a preset contour detection algorithm.

Specifically, step 510 may include step 5101 to step 5103, which are described below.

Step 5101: Based on a preset contour detection algorithm, determine a set of contours included in the non-editable graphic image.

Specifically, OpenCV's function cv2.findContours can be used to perform contour detection and obtain a collection of contours included in non-editable graphics and text images. The set of contours may be, for example, C={C ₁ , C ₂ ,..., C _c }, where C ₁ , C ₂ , and C _c each represent a contour.

It can be understood that before performing contour detection on the non-editable graphic and text images, the non-editable graphic and text images can be pre-processed to facilitate subsequent contour detection. For example, non-editable graphics and text images can be grayscaled and binarized.

Step 5102: Based on the coordinates of each contour and the text box, filter out the contours that overlap the text box.

It can be understood that the contours obtained based on the preset contour detection algorithm include not only the contours of some images, but also the contours of text boxes. Therefore, the contours of the text boxes need to be removed to obtain the contours of the images in the non-editable graphic and text images.

Specifically, the above set of contours C={C ₁ , C ₂ ,..., C _c } can be traversed, and the currently traversed contour is recorded as C _x . For the currently traversed contour C _x , find whether the contour C _x The degree of overlap with each text box is greater than the preset threshold. If so, the outline C _x is deleted and an updated set of outlines C is obtained.

Step 5103: Determine at least one contour based on the remaining contours.

Specifically, the contours in the updated contour set C are connected. Specifically, the OpenCV function approxPolyDP can be used to perform contour approximation.

Step 520: Identify the shape of each contour in at least one contour based on the shape recognition model of the pre-trained residual neural network.

The residual neural network may be ResNet50, for example. Specifically, the data set can be randomly divided, Obtain the training set, verification set and test set, use the training set to train the ResNet50 model, and the verification set and test set are used to adjust the parameters of the ResNet50 model and test the performance respectively. And the cross entropy of multi-label classification is used as the loss function to train the shape recognition model based on the residual neural network, and the pre-trained shape recognition model of the residual neural network is obtained.

It can be understood that the classification categories corresponding to the shape recognition model of the pre-trained residual neural network can be predetermined. The classification categories are determined based on common shapes, and the shapes of the relevant classification categories are set in the subsequent visual editing platform. This makes it easy to create editable shapes. Specifically, shapes can be divided into three categories: arrow shape categories, basic shape categories, and image shape categories. Among them, the arrow shape categories include: up arrow, down arrow, straight arrow, curved arrow, bidirectional arrow, straight line and curve. Basic shape classes include: circle, triangle, square, sector, ellipse, parallelogram and rhombus. The image shape category includes non-arrow and non-basic images presented in graphic files. The classification of the defined shape recognition model is consistent with the shape types provided by the graphic editor of the present invention.

It can also be understood that before identifying the shape of each contour, each contour can be pre-processed. For example, data enhancement can be performed on the contour, including operations such as scaling down and enlarging, and color transformation.

Step 530: Determine the relative size of the shape based on the minimum circumscribed rectangle size of the shape of each outline, and determine the position of each outline based on the coordinates of the preset position of the shape of each outline.

The coordinates of the preset position of each contour shape may be the position coordinates of the upper left corner and the lower right corner of each contour shape, or may be other coordinates on the shape of each contour that can identify the contour.

It can be understood that step 530 provides a possible method for determining the size and position of the outline, and other methods can also be used for determination, and the present invention is no longer limited in this regard.

Step 540: Determine the centroid coordinates of each contour in at least one contour.

Specifically, the set of updated contours C = {C ₁ , C ₂ , ..., C _c } can be traversed, and the centroid coordinates corresponding to each contour in the set of updated contours C can be calculated as the coordinates of the contours Centroid attribute value. Specifically, OpenCV's function cv2.moments can be used to calculate the first-order geometric moment to obtain the centroid position of the specified contour.

Step 550: Determine the color of each contour based on the color corresponding to the centroid coordinate of each contour.

Specifically, the color value of the centroid coordinate is obtained according to the centroid coordinate of each contour as the color attribute value of the contour.

In one embodiment, as shown in Figure 6, generating an editable document corresponding to a non-editable graphic image based on the final structured data includes the following steps:

Step 610: Obtain and display the final structured data.

It can be understood that this step can be implemented by a graphic editor. Specifically, the final structured data can be as shown in Figure 9 below, and will not be described in detail here. Among them, the graphic editor can be a self-built graphic editor based on Electron.

Step 620: Based on the final structured data, generate images corresponding to outline features and text corresponding to text features at corresponding positions on the canvas, and determine the initial editable document corresponding to the non-editable graphic and text images.

Specifically, when the editor renders the visual interface, it first creates a shape based on the shape of the outline in the outline feature, then determines the position of the shape on the canvas based on the position coordinates of the outline, and then fills the shape color and adjusts the shape size. After the outline is created, the text box will be created, and the text box will be created based on the coordinates of the text box in the structured data, and the recognized text content will be generated in the text box, and the text color and size will be adjusted, and the above generated and The created content is saved, thereby generating an initial editable document corresponding to the non-editable graphic and text image.

In one of the embodiments, after creating the image corresponding to the outline feature and the text corresponding to the text feature at corresponding positions on the canvas, the method further includes:

In response to user operations, the initial editable document is added, modified, or deleted.

It can be understood that after the graphic editor completes editing, the initial editable document can be saved in a format editable by other graphic editors, and further edits such as adding, modifying, and deleting can be performed in other graphic editors. Implement compatibility between editors. In one possible embodiment, the other editor may be Visio.

Figure 7 is a schematic diagram of graphic images. As shown in Figure 7, it includes CT scan images, arrows, multiple text contents, and background graphics corresponding to the text contents.

FIG. 8 is a schematic diagram of the initial structured data corresponding to the graphic image in FIG. 7 . As shown in FIG. 8 , it includes graphic related information (shapes), image related information (figures) and text features (texts) included in the graphic image as shown in FIG. 7 .

Among them, the related information of graphics (shapes) and the related information of images (figures) can correspond to The contour characteristics of the previous article. Figures correspond to the CT scan images in Figure 7, where figures: [{id: element 21, path: "fig1.jpg", size: (a21, w21), position: ((x41, y41), (x42, y42))}], where "path: "fig1.jpg"" is the saving path of the image, and "position: ((x41, y41), (x42, y42))" is the image in the non-editable graphic and text category The relative position in , "size: (a21, w21)" is the size of the image.

In addition, {id: element 1, type: RightDirectionalConnector, color: (r1, g1, b1), size: (a1, w1), position: ((x1, y1), (x2, y2))} in shapes , "type: RightDirectionalConnector" is the shape of the outline, "color: (r1, g1, b1)" is the color of the outline, "size: (a1, w1)" is the size of the outline, "position: ((x1, y1) , (x2, y2))}” is the position of the outline.

It can be understood that other items in shapes are similar to the aforementioned item with the id of element 1. For the sake of brevity, they will not be described again here.

Similarly, {id in texts: element 13, content: "Image Encoder", color: (r13, g13, b13), size: (a13, w13), position: ((x25, y25), (x26, y26))}, "content: "Image Encoder"" is the content of the text, "color: (r13, g13, b13)" is the color of the text, "size: (a13, w13)" is the font size, "position: ((x25, y25), (x26, y26))" is the position coordinate of the text box. The other items in texts are similar to the aforementioned item with the id of element 13. For the sake of brevity, they will not be described again here.

FIG. 9 is a schematic diagram of the final structured data corresponding to the graphic image in FIG. 7 . As shown in Figure 9, in addition to the graphics-related information, image-related information and text features included in the graphic image as shown in Figure 8, it also includes the relationship between the two elements. Taking element 1 as an example, its corresponding final structured data is: "Self attributes: [type: RightDirectionalConnector, color: (r1, g1, b1), size: (a1, w1), position: ((x1, y1 ), (x2, y2))], relationship: [[combination object: element 2, combination relationship: association], [combination object: element 19, combination relationship: surround], [combination object: element 21, combination relationship: association ]]". Compared with Figure 8, the final structured data shown in Figure 9 also has more elements related to element 1 and the relationship between the two elements: "Relationship: [[Combined object: element 2, combined relationship: association] , [Combined object: Element 19, Combination relationship: Surround], [Combined object: Element 21, Combination relationship: Association]]".

Figure 10 is a schematic diagram of the display interface of the graphic editor provided by the present invention. As shown in Figure 10, it can be to open and display the editable document corresponding to the non-editable graphic image, and make some simple adjustments to the content in the editable document, such as adjusting the size, color and position of the font in the image, and you can select lines The thickness, you can also adjust the overall layout of graphics, text and images in the editable document, and add or modify some new shapes.

The device for generating editable documents based on non-editable graphic and text images provided by the present invention is described below. The device for generating editable documents based on non-editable graphic and text images described below is different from the device based on non-editable graphic and text images described above. Methods for generating editable documents from class images can be referenced in correspondence with each other.

As shown in Figure 11, in one embodiment, a device for generating an editable document based on a non-editable graphic image is provided. The device for generating an editable document based on a non-editable graphic image may include:

The acquisition module 1110 is used to acquire non-editable graphic and text images;

The first determination module 1120 is used to extract outline features from the non-editable graphic and text images; extract text features from the non-editable graphic and text images; wherein the outline features include the shape of the outline, Position, size and color within the outline; the text features include text box coordinates, text content, text color and font size;

The first generation module 1130 is used to generate initial structured data according to the outline features and the text features;

The second determination module 1140 is used to determine the relationship between two elements in the non-editable graphic image based on the pre-trained element relationship classification model and the outline features and the text features; wherein, The above-mentioned pre-trained element relationship classification model is determined after training based on a data set composed of the outline features or text features of the two elements, the outline features and/or text features, and the relationship labels of the two elements;

The supplement module 1150 is used to supplement the initial structured data based on the relationship between the two elements to obtain the final structured data;

The second generation module 1160 is configured to generate an editable document corresponding to the non-editable graphic image based on the final structured data.

The device provided by the present invention for generating editable documents based on non-editable graphic and text images can determine the elements contained in the non-editable graphic and text images and their corresponding elements by determining the outline features and text features of the non-editable graphic and text images. attributes, and generate initial The initial structured data is then further determined based on the pre-trained element relationship classification model to determine the relationship between the two elements, and the relationship between the two elements is supplemented to the initial structured data to obtain the final structure. ized data, and then based on the final structured data, editable documents corresponding to the non-editable graphic and text images are generated, thereby no longer relying on manual conversion of non-editable graphic and text images into editable documents, thereby achieving efficient, Accurately convert non-editable graphics and text images into editable documents.

In one embodiment, the second determining module 1140 includes:

A first determination unit, configured to determine the classification result of the relationship between each two elements in the non-editable graphic and text image based on each pre-trained two-classification model and the contour features and the text features;

The second determination unit is used to determine the final classification result of the relationship between each two elements in the non-editable graphic and text image based on the classification result with the largest probability value among the plurality of determined classification results.

In one embodiment, the second determination module 1140 further includes:

The acquisition unit is used to acquire the outline features and text features of multiple non-editable graphic and text images;

The third determination unit is used to determine the data set based on the contour features and/or text features corresponding to each two elements in each non-editable graphic image; and determine each two elements based on the relationship between each two elements. relationship tag;

The fourth determination unit is used to determine the samples of the data set as positive samples and negative samples based on the relationship labels of each two elements and their corresponding binary classification models;

The training unit is used to train the corresponding two-classification model based on positive samples and negative samples to obtain a pre-trained two-classification model.

In one embodiment, the first determining module 1120 includes:

The fifth determination unit is used to determine the text box and its coordinates included in the non-editable graphic image based on a preset text box detection algorithm;

The sixth determination unit is used to determine the text content included in each text box based on the preset text recognition algorithm;

The seventh determination unit is used to determine the text color in each text box based on the coordinates of each text box and the pixel histogram in each text box;

The eighth determination unit is used to determine the size of the font in the text box based on the coordinates of each text box.

In one embodiment, the first determining module 1120 further includes:

A ninth determination unit, configured to determine at least one contour included in the non-editable graphic image based on a preset contour detection algorithm;

a recognition unit configured to recognize the shape of each contour in the at least one contour based on a shape recognition model based on a pre-trained residual neural network;

a tenth determination unit, configured to determine the relative size of the shape based on the minimum circumscribed rectangle size of the shape of each outline, and determine the position of each outline based on the coordinates of the preset position of the shape of each outline;

An eleventh determination unit, configured to determine the color of each contour based on the color corresponding to the centroid coordinate of each contour.

In one embodiment, the ninth determining unit includes:

A twelfth determination unit, configured to determine a set of contours included in the non-editable graphic image based on a preset contour detection algorithm;

A filtering unit configured to filter out the contours that coincide with the text box according to the coordinates of each contour and the text box;

A thirteenth determination unit, configured to determine at least one contour based on the remaining contours.

In one embodiment, the second generation module 1160 includes:

An acquisition and display unit is used to acquire and display the structured data;

A generating unit configured to generate, based on the final structured data, an image corresponding to the outline feature and a text corresponding to the text feature at a corresponding position on the canvas, and determine the image corresponding to the non-editable graphic and text image. Initial editable document.

Figure 12 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 12, the electronic device may include: a processor (processor) 1210, a communications interface (communications interface) 1220, a memory (memory) 1230 and a communication bus 1240. Among them, the processor 1210, the communication interface 1220, and the memory 1230 complete communication with each other through the communication bus 1240. The processor 1210 can call logical instructions in the memory 1230 to execute a method of generating an editable document based on a non-editable graphic image. The method includes: obtaining a non-editable graphic image; and generating an editable document from the non-editable graphic image. Extract outline features from the class image; extract text features from the non-editable graphic image; wherein the outline features include the shape, position, size and color of the outline; the text features include Text box coordinates, text content, text color and font size; generating initial structured data according to the outline features and the text features; based on the pre-trained element relationship classification model and the outline features and the text features, Determine the relationship between two elements in the non-editable graphic image; wherein the pre-trained element relationship classification model is based on the outline features or text features of the two elements, and The data set consisting of the relationship tags of the two elements is determined after training; based on the relationship between the two elements, it is added to the initial structured data to obtain the final structured data; based on the final structured data, a generated The editable document corresponding to the non-editable graphic image.

In addition, the above-mentioned logical instructions in the memory 1230 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), magnetic disk or optical disk and other media that can store program code. .

On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are read by a computer, When executed, the computer can execute the method for generating an editable document based on a non-editable graphic image provided by the present invention. The method for generating an editable document based on a non-editable graphic image includes: obtaining the non-editable graphic image. image; extract contour features from the non-editable graphic and text images; extract text features from the non-editable graphic and text images; wherein the contour features include the shape, position, size and content of the contour color; the text features include text box coordinates, text content, text color and font size; generate initial structured data according to the outline features and text features; based on the pre-trained element relationship classification model and the The outline features and the text features are used to determine the relationship between the two elements in the non-editable graphic image; wherein the pre-trained element relationship classification model is based on the relationship between the two elements. The data set composed of contour features or text features, contour features and/or text features, and the relationship labels of the two elements is determined after training; based on the relationship between the two elements, it is added to the initial structured data to obtain the final Structured data; based on the final structured data, generate an editable document corresponding to the non-editable graphic image.

On the other hand, the present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. The computer program is implemented when executed by a processor to perform the non-editable graphics-based image generation provided by the present invention. A method for editing an editable document. The method for generating an editable document based on a non-editable graphic image includes: obtaining a non-editable graphic image; extracting outline features from the non-editable graphic image; Text features are extracted from the non-editable graphic images; wherein the outline features include the shape, position, size and color of the outline; the text features include text box coordinates, text content, text color and Font size; generate initial structured data according to the outline features and the text features; determine the non-editable graphic and text images based on a pre-trained element relationship classification model and the outline features and the text features The relationship between two elements in; wherein, the pre-trained element relationship classification model is based on a data set composed of the outline features or text features of the two elements, the outline features and/or text features, and the relationship labels of the two elements. Determine after training; add to the initial structured data based on the relationship between the two elements to obtain the final structured data; generate the non-editable graphic and text image correspondence based on the final structured data editable document.

The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the part of the above technical solution that essentially contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disc, optical disc, etc., including a number of instructions for causing a computer device to (can be a personal computer, server, or network device, etc.) to perform the methods described in various embodiments or certain parts of the embodiments.

It can be understood that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still make various modifications to the foregoing embodiments. The technical solutions described in the embodiments may be modified, or some of the technical features thereof may be equivalently substituted; however, these modifications or substitutions shall not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.

Claims (10)

1. A method of generating editable documents based on non-editable graphic images, characterized in that the method includes:

   Get non-editable graphic and text images;

   Extract outline features from the non-editable graphic and text images; extract text features from the non-editable graphic and text images; wherein the outline features include the shape, position, size and color within the outline. ;The text features include text box coordinates, text content, text color and font size;

   Generate initial structured data according to the outline features and the text features;

   Based on the pre-trained element relationship classification model and the outline features and the text features, the relationship between the two elements in the non-editable graphic image is determined; wherein the pre-trained element relationship classification The model is determined after training based on a data set composed of the outline features and/or text features of the two elements, and the relationship labels of the two elements;

   Supplement the initial structured data based on the relationship between the two elements to obtain the final structured data;

   Based on the final structured data, an editable document corresponding to the non-editable graphic image is generated.
2. The method of generating editable documents based on non-editable graphics and text images according to claim 1, wherein the pre-trained element relationship classification model includes a plurality of pre-trained two-classification models, corresponding to , based on the pre-trained element relationship classification model and the outline features and the text features, determining the relationship between the two elements in the non-editable graphic image includes:

   Based on each pre-trained binary classification model and the contour features and the text features, determine the classification result of the relationship between each two elements in the non-editable graphic image;

   Based on the classification result with the largest probability value among the multiple determined classification results, the final classification result of the relationship between each two elements in the non-editable graphic and text image is determined.
3. The method for generating editable documents based on non-editable graphics and text images as claimed in claim 2, characterized in that a data set is composed of outline features and/or text features of two elements and relationship tags of the two elements. The process of determining the pre-trained element relationship classification model after training includes:

   Obtain the outline features and text features of multiple non-editable graphic and text images;

   Determine the data set based on the contour features and/or text features corresponding to each two elements in each non-editable graphic image; and determine the relationship label of each two elements based on the relationship between each two elements;

   Based on the relationship labels of each two elements and their corresponding binary classification models, the samples of the data set are determined as positive samples and negative samples;

   Based on the positive samples and negative samples, the corresponding two-classification model is trained to obtain the pre-trained two-classification model.
4. The method of generating an editable document based on a non-editable graphic image according to claim 1, wherein the text features in the non-editable graphic image are determined based on the text detection and text recognition method, include:

   Based on a preset text box detection algorithm, determine the text box and its coordinates included in the non-editable graphic image;

   Based on a preset text recognition algorithm, determine the text content included in each text box;

   Determine the text color in each text box based on the coordinates of each text box and the pixel histogram in each text box;

   Based on the coordinates of each text box, determine the size of the font within the text box.
5. The method for generating editable documents based on non-editable graphics and text images according to claim 4, wherein the contour features in the non-editable graphics and text images are determined based on the contour detection and shape recognition method. ,include:

   Determine at least one contour included in the non-editable graphic and text-based image based on a preset contour detection algorithm;

   A shape recognition model based on a pre-trained residual neural network, identifying the shape of each of the at least one contour;

   determining the relative size of the shape based on the size of the smallest enclosing rectangle of the shape of each outline, and determining the position of each outline based on the coordinates of a preset position of the shape of each outline;

   The color of each contour is determined based on the color corresponding to the centroid coordinate of each contour.
6. The method for generating an editable document based on a non-editable graphic image according to claim 5, characterized in that, based on a preset contour detection algorithm, at least one of the non-editable graphic images is determined. Outline; includes:

   Based on a preset contour detection algorithm, determine a set of contours included in the non-editable graphic image;

   According to the coordinates of each contour and the text box, filter out the contours that coincide with the text box;

   At least one contour is determined based on the remaining contours.
7. The method of generating an editable document based on a non-editable graphic image as claimed in claim 1, wherein an editable document corresponding to the non-editable graphic image is generated based on the final structured data, include:

   Obtain and display said final structured data;

   Based on the final structured data, an image corresponding to the outline feature and a text corresponding to the text feature are generated at corresponding positions on the canvas, and an initial editable document corresponding to the non-editable graphic image is determined.
8. A device for generating editable documents based on non-editable graphic images, characterized in that the device includes:

   Acquisition module, used to obtain non-editable graphic and text images;

   The first determination module is used to extract contour features from the non-editable graphic image; extract text features from the non-editable graphic image; wherein the contour features include the shape and position of the contour , size and color within the outline; the text features include text box coordinates, text content, text color and font size;

   A first generation module, configured to generate initial structured data according to the outline features and the text features;

   The second determination module is used to determine the relationship between two elements in the non-editable graphic image based on the pre-trained element relationship classification model and the outline features and the text features; wherein, the The pre-trained element relationship classification model is determined after training based on a data set composed of the outline features or text features of the two elements, the outline features and/or text features, and the relationship labels of the two elements;

   A supplementary module, used to supplement the initial structured data based on the relationship between the two elements to obtain the final structured data;

   The second generation module is configured to generate an editable document corresponding to the non-editable graphic and text image based on the final structured data.
9. A computer device, including a memory and a processor, characterized in that computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the processor executes the instructions as claimed in claim 1 The steps of the method for generating an editable document based on a non-editable graphic image according to any one of claims 1 to 7.
10. A storage medium storing computer-readable instructions, characterized in that, when the computer-readable instructions are executed by one or more processors, they cause the one or more processors to execute any one of claims 1 to 7 The steps of the method for generating editable documents based on non-editable graphic and textual images described in the claim.