CN106897732B - A Multi-Oriented Text Detection Method in Natural Images Based on Linked Text Fields - Google Patents

Abstract

The invention discloses multi-direction Method for text detection in a kind of natural picture based on connection text section, text section and connection are two steps crucial in the detection method, be defined as follows: text section refers to marking off many single multidirectional bounding box regions on picture, they surround a part of a text item or word；Connection refers to connecting adjacent field, it is meant that they belong to the same word or in short.The full convolutional neural networks that text section and connection are used in combination an end-to-end training are equally spaced detected with a variety of scales.Last testing result is first to connect multiple text section composition new regions, obtained from being then combined to these new regions.Detection method proposed by the present invention all achieves brilliant effect in terms of these in accuracy rate, speed and model ease compared with the existing technology, high-efficient and strong robustness, complicated picture background can be overcome, in addition also can in detection image non-Latin text long text.

Description

A Multi-Oriented Text Detection Method in Natural Images Based on Linked Text Fields

technical field

The invention belongs to the technical field of computer vision, and more specifically relates to a multi-directional text detection method in natural pictures based on connected text fields.

Background technique

Reading text in natural images is a challenging and popular task with many practical applications in photo optical recognition, geolocation, and image retrieval. In the text reading system, text detection is to locate the text area with a bounding box at the word level or text strip level, and it is usually regarded as a very critical first step. In a sense, text detection can also be regarded as a special object detection, which takes words, characters or text strips as the detection target.

Although existing technologies have achieved great success in applying object detection methods to text detection, there are still several obvious deficiencies in object detection methods in locating text regions. First, the aspect ratio of words or text strips is usually much larger than that of general objects, and it is difficult for previous methods to generate bounding boxes of this ratio; second, some non-Latin texts have no gaps between adjacent words. Contains spaces, such as Chinese characters. Existing technologies can only detect words, and are not applicable to this kind of text, because this kind of text without spaces cannot provide visual information for dividing different words. Third, in large natural scene images, the text may be in any direction, but most of the existing technologies can only detect horizontal text. Therefore, text detection in natural scene pictures is still one of the difficulties in the field of computer vision technology.

Contents of the invention

The purpose of the present invention is to provide a multi-directional text detection method in natural pictures based on connected text fields. The method has high text detection accuracy, fast speed, simple model, and strong robustness, and can overcome complex picture backgrounds. Also detects long text in non-Latin scripts.

In order to achieve the above object, the present invention solves the scene text detection problem from a brand new perspective, and provides a multi-directional text detection method in a natural picture based on connected text fields, comprising the following steps:

(1) Training the text field connection detection network model, including the following sub-steps:

(1.1) Mark the text content of all text images in the training image set with the entry level, and the label is the four point coordinates of the rectangular initial bounding box of the entry to obtain the training data set;

(1.2) Define the text field detection model that can predict the output text field and connection according to the entry label, the network model is composed of a cascaded convolutional neural network and a convolution predictor, and the text field is calculated according to the above-mentioned training data set and connected tags, design a loss function, combine online amplification and online negative sample hard example mining techniques, use the reverse conduction method to train the network, and obtain a text field detection model, including the following sub-steps:

(1.2.1) Constructing a text field detection convolutional neural network model: the first few layers of convolutional units for extracting features come from the pre-trained VGG-16 network, and the first few layers of convolutional units are convolutional layer 1 to pooling layer 5, Fully connected layer 6 and fully connected layer 7 are converted to convolutional layer 6 and convolutional layer 7 respectively, followed by some additional convolutional layers for extracting deeper features for detection, including convolutional layers 8. Convolutional layer 9, convolutional layer 10, and the last layer is convolutional layer 11; the last 6 different convolutional layers output feature maps of different sizes, which is convenient for extracting high-quality features of multiple scales and detecting text Segments and connections are performed on these six feature maps of different sizes; for these six convolutional layers, a filter of size 3×3 is added after each layer as a convolutional predictor to jointly detect text fields and connect;

(1.2.2) Generate text field bounding box labels from marked word bounding boxes: For the original training image set Itr, record the scaled training image set as Itr′, w _I , h _I are the width and height of Itr′ respectively, Can take 384×384 or 512×512 pixels, the i-th picture Itr _i ′ is used as the model input, and all word bounding boxes marked on Itr _i ′ are recorded as W _i =[W _i1 ,...,W _ip ], where W _ij is the jth word bounding box on the i-th picture, the word bounding box can be word level or entry level, j=1,..., p, p is the word bounding box on the i-th picture The total number of ; Note that the feature maps output by the last 6 convolutional layers constitute a set Itro _i ′=[Itro _i1 ′,...,Itro _i6 ′], where Itro _il ′ is the lth in the last 6 convolutional layers The feature map output by the layer, w _l and h _l are the width and height of the feature map respectively, and the coordinates (x, y) on Itro _il ′ correspond to the coordinates of the center point on Itr _i ′ (x _a , y _a ) Horizontal initial bounding boxes B _ilq , they satisfy the following formula:

The width and height of the initial bounding box B _ilq are both set to a constant a _l to control the proportion of the output text field, l=1,...,6; record the feature map Itro _il 'corresponding to the l-th layer output The set of initial bounding boxes is B _il =[B _il1 ,...,B _ilm ], q=1,...,m, where m is the number of initial bounding boxes on the feature map output by layer l; as long as the initial bounding boxes The center of the box B _ilq is contained inside any marked word bounding box W _ij on Itr′, and the size a _l of B _ilq and the height h of the marked word bounding box W _ij satisfy: Then this initial bounding box B _ilq is marked as a positive class, the value of the label is 1, and it matches the word bounding box W _ij with the closest height; otherwise, when B _ilq and all word bounding boxes W _i do not satisfy the above two When a condition is met, B _ilq is marked as a negative class, and the value of the label is 0; the text field is generated on the initial bounding box, which is the same as the label category of the initial bounding box; wherein, the proportional constant 1.5 is an empirical value;

(1.2.3) Generate a text field on the labeled initial bounding box generated in the step (1.2.2) and calculate the offset of the positive text field: the negative text field bounding box s ^- is the negative initial bounding box B ^- ; the bounding box s of the positive text field ⁺ is obtained from the initial bounding box B ⁺ of the positive class through the following steps: a) Record the initial bounding box B of the positive class ⁺ the matched tag word bounding box W and the horizontal direction The angle is θ _s , with the central point of B ⁺ as the center, rotate W clockwise by θ _s angle; b) crop W, and remove the part beyond the left and right of B ⁺ ; c) center the center point of B ⁺ , the cropped word The bounding box W′ is rotated counterclockwise by the θ _s angle, and the geometric parameters x _s , y _s , w _s , h _s , θ _s of the text field s ⁺ the real label are obtained; d) Calculate the offset of the text s ⁺ relative to B ⁺ Quantity (Δx _s , Δy _s , Δw _s , Δh _s , Δθ _s ), the calculation formula is as follows:

x _s ＝a _l Δx _s +x _a

y _s = a _l Δy _s + y _a

w _s = a _l exp(Δw _s )

h _s = a _l exp(Δh _s )

θ _s = Δθ _s

Among them, x _s , y _s , w _s , h _s , and θ _s are respectively the abscissa of the center point, the ordinate of the center point, width, height, and the angle between the text field bounding box s ⁺ and the horizontal direction; x _a , y _a , w _a , h _a are the abscissa, ordinate, width, and height of the center point of the horizontal initial bounding box B ⁺ respectively; Δx _s , Δy _s , Δw _s , Δh _s , and Δθ _s are text fields The offset of bounding box s ⁺ central point abscissa x _s relative to initial bounding box B ⁺ , the offset of vertical coordinate y _s relative to initial bounding box, the offset change of width w _s , and the offset change of height h _s amount, the offset of angle θ _s ;

(1.2.4) Calculate the connection label for the bounding box of the text field generated in step (1.2.3): the text field s is generated on the initial bounding box B, so the connection labels between s and their corresponding initial bounding box B The connection labels between are the same; for the feature map set Itro _i ′=[Itro _i1 ′,...,Itro _i6 ′], if in the initial bounding box set B _il of the same feature map Itro _il ′, two initial bounding box The labels of are all positive classes, and matches the same word, then The intra-layer connection between is marked as a positive class, otherwise it is marked as a negative class; if the initial bounding box in the initial bounding box set B _il corresponding to the feature map Itro _il ′ The initial bounding box in the initial bounding box set B _i(l-1) corresponding to Itro _i(l-1) ′ labels are all positive and match the same word bounding box W _ij , then The cross-layer connections between are marked as positive class, otherwise marked as negative class;

(1.2.5) Take the scaled training image set Itr′ as the input of the text field detection model, predict the text field s output: initialize the weight and bias of the model, set the learning rate of the first 60,000 training iteration steps to 10 ^-3 , Then the learning rate decays to 10 ^-4 ; for the last 6 convolutional layers, at the coordinates (x, y) on the feature map Itro _il ′ of the first layer, (x, y) corresponds to the input image Itr _i ′ with (x _a , y _a ) is the center point coordinates, the initial bounding box B _ilq with a _l as the size, the 3×3 convolutional predictor will predict B _ilq is divided into positive and negative scores c _s , c _s is a two-dimensional vector, and its value range is a decimal between 0-1. Also predicted 5 numbers As the geometric offset when divided into the positive text field s ⁺ , where Respectively, the predicted text field bounding box s ⁺ center point abscissa relative to the offset of the positive initial bounding box B ⁺ , the vertical coordinate relative to the offset of the positive initial bounding box B ⁺ , the offset change in height, Width offset variation, angle offset;

(1.2.6) Predict the output of intra-layer connection and cross-layer connection based on the predicted text field: for intra-layer connection, at the coordinate point (x, y) on the same feature map Itro _il ′, take x-1 ≤x′≤x+1, y-1≤y′≤y+1, the neighbor points (x′, y′) in the range of y-1≤y′≤y+1, when these 8 points correspond to the input image Itr _i ′, the result of , y) and the adjacent text field s ^{(x′, y′, l)} in the layer connected to the reference text field s ^{(x, y, l)} corresponding to y), the 8 adjacent text fields in the layer can be expressed as a set:

A 3×3 convolutional predictor predicts s ^{(x, y, l)} and the set of neighbors in the layer The positive and negative scores c _l1 of the connection, c _l1 is a 16-dimensional vector, where w is a superscript, indicating the connection within the layer;

For cross-layer connection, a cross-layer connection connects the text fields corresponding to two points on the feature map output by two consecutive convolutional layers; because each layer of convolutional layer passes through, the width and height of the feature map will be reduced by half , the width w _l and height h _l of the l-th layer output feature map Itro _il ′ are half of the width w _l-1 and height h _l-1 of the l-1 layer feature map Itro _i(l-1) ′, and The initial bounding box scale a _l corresponding to Itro _il ′ is twice the initial bounding box scale a _l-1 corresponding to _{Itro i(l-1)} ′, for (x, y), take 4 cross-layer neighbor points (x', y') in the range of 2x≤x'≤2x+1, 2y≤y'≤2y+1 on the feature map Itro _i(l-1) ', (x, y) on Itro _il ′ corresponds to the initial bounding box on the input image Itr _i ′, and the 4 cross-layer neighbor points on Itro _i(l-1) ′ correspond to the 4 initial bounding boxes on the input image Itr _i ′ The spatial positions of the bounding boxes coincide, and the four cross-layer neighbor text fields s ^{(x′, y′, l-1)} can be expressed as a set:

The 3×3 convolutional predictor will predict the base text field s ^{(x, y, l)} of layer l and the set of adjacent text fields on layer l-1 The positive and negative scores c _l2 of the cross-layer connection, c _l2 is an 8-dimensional vector:

in, Indicates that the predictor predicts the positive and negative scores of the connection between s ^{(x, y, l)} and all 4 adjacent text fields, and c is a superscript, indicating a cross-layer connection;

All intralayer connections and all cross-layer connections Constitute the connection set N _s ;

(1.2.7) Take the text field label obtained in step (1.2.3) and step (1.2.4), the real offset of the positive text field, and the connection label as the output reference, and the text predicted by step (1.2.5) The category and score of the segment, the offset of the predicted text field, and the connection score predicted in step (1.2.6) are the predicted output, and the target loss function between the predicted output and the output benchmark is designed, and the reverse conduction is used for the text field connection detection model The method is continuously trained to minimize the loss of text field classification, text field offset regression and connection classification, and the target loss function is designed for the text field connection detection model. The target loss function is the weighted sum of three losses:

where y _s is the label of all text fields, c _s is the predicted text field score, y _l is the predicted connection label, c _l is the predicted connection score, which is composed of intra-layer connection score c _l1 and cross-layer score c _l2 ; If the i-th initial bounding box is marked as a positive class, then y _s (i) = 1, otherwise 0; L _conf (y _s , c _s ) is the softmax loss of the predicted text field score c _s , L _conf (y _s , c _l ) is the softmax loss of the predicted connection score c _l , is the predicted text field geometry parameter s and the ground truth label The smoothed L between ₁ regression loss; n _s is the number of positive initial bounding boxes, used to normalize text field classification and regression loss; n _l is the total number of positive class connections, used to normalize the connection classification loss ; λ ₁ and λ ₂ are weight constants, which are 1 in practice;

(1.2.8) During the training process of step (1.2.7), the online amplification method is used to amplify the training data I _tr online, and the online negative sample hard case mining strategy is used to balance positive samples and negative samples. Before the training images I _tr are scaled to the same size and loaded in batches, they are randomly cropped into image blocks, and the jaccard overlap coefficient o of each image block and the true bounding box of the text field is the smallest; for multi-directional text, The data augmentation is carried out on the minimum enclosing rectangle of the multi-directional text bounding box, the overlap coefficient o of each sample is randomly selected from 0, 0.1, 0.3, 0.5, 0.7 and 0.9, and the size of the image block is the size of the original image Between 0.1 and 1 times; the training image is not flipped horizontally; in addition, text fields and connection negative samples occupy most of the training samples, and the online negative sample hard case mining strategy is used to balance the positive samples and negative samples, and the text field and connection are separated For mining, control the ratio between negative samples and positive samples not to exceed 3:1.

(2) Use the above-mentioned trained convolutional neural network to perform text field and connection detection on the text image to be detected, including the following sub-steps:

(2.1) Perform text field detection on the text image to be detected. The feature maps output by different convolutional layers can predict text fields of different scales, and the feature maps output by the same convolutional layer can predict text fields of the same scale: the image to be detected The i-th text image Itst _i to be detected in the collection Itst is scaled to a uniform size, and the specific size can be set manually according to the situation of the text image to be detected, and the zoomed text image to be detected is Itst _i ′. Input the image Itst _i ′ into the text field connection detection model trained in step (1.2), and obtain the set Itsto _i ′=[Itsto _i1 ′,...,Itsto _i6 ′], where Itsto _il ′ is the feature map output by the l-th layer in the last 6 convolutional layers, l=1,...,6, the coordinates (x, y on each output feature map Itsto _il ′ ), the 3×3 convolution predictor will predict the initial bounding box B _ilq corresponding to (x, y) is predicted to be the score c _s of positive and negative text fields, and also predicts 5 numbers As the geometric offset when it is predicted as a positive text field s ⁺ ;

(2.2) The text fields on all feature layers detected by the text image to be detected are connected and detected, and the connections include intra-layer connections and cross-layer connections: predict intra-layer connections and cross-layer connections based on the text fields predicted in (2.1) Connection, for intra-layer connections, at the coordinate point (x, y) on the same feature map Itsto _il ′, the 3×3 convolutional predictor predicts s ^{(x, y, l)} and its 8 neighbor text fields The positive and negative scores c _l1 of the intra-layer connection between them; for the cross-layer connection, the 3×3 convolutional predictor will predict the reference text field s ^{(x, y, l)} of the l-th layer and the 4 on the l-1 layer neighbor text fields The cross-layer connection positive and negative scores c _l2 , c _l1 and c _l2 constitute the predicted connection score c _l ;

(2.3) Combine the detected text field confidence score and connection confidence score, where the text field confidence score includes the positive and negative category score and offset score of the text field, and use the convolution predictor to output the softmax standardized score: in ( 2.1) Based on the predicted text field, the intra-layer connection and cross-layer connection are predicted. For the intra-layer connection, at the coordinate point (x, y) on the same feature map Itsto _il ′, the 3×3 convolutional predictor predicts s ^{(x, y, l)} and 8 neighbor text fields The positive and negative scores c _l1 of the intra-layer connection between them; for the cross-layer connection, the 3×3 convolutional predictor will predict the reference text field s ^{(x, y, l)} of the l-th layer and the 4 on the l-1 layer neighbor text field The cross-layer connection positive and negative scores c _l2 , c _l1 and c _l2 constitute the predicted connection score c _l .

(3) Combining text fields and connections to obtain an output bounding box, including the following sub-steps:

(3.1) According to the standardized score detected in (2.3), filter the text field and connection output by the convolutional predictor, use the filtered text field as a node, and use the connection as an edge to construct a connection graph: For step (2) to be Detect a fixed number of text fields s and connection N _s generated by inputting the text image to the text field detection model, and filter through their scores; set different filtering thresholds for the text field s and connection N _s , respectively α and β; The filtering threshold can be artificially set different values according to different data. In practice, when performing multi-directional text image detection, α=0.9, β=0.7 can be used, and when multilingual long text image detection is performed, α=0.9 can be used. β=0.5, when performing horizontal text detection, you can take α=0.6, β=0.3; use the filtered text field s' as a node, and the filtered connection N _s ' as an edge, and use them to construct a graph;

(3.2) Perform a depth-first search on the graph to find connected components. Each component is denoted as a set B, which contains text fields connected by connections;

(3.3) The text field set S obtained by performing depth-first search on the graph in step (3.2) is combined into a complete word through the following steps, including:

(3.3.1) Input: |S| is the number of text fields in the set S, where is the i-th text field, i is the superscript, are respectively the central abscissa and ordinate of the i-th text field bounding box s ⁽ⁱ⁾ , are the width and height of text field bounding box s ⁽ⁱ⁾ , respectively, is the angle between the text field bounding box s ⁽ⁱ⁾ and the horizontal direction;

(3.3.2) where θ _b is the offset angle of the output bounding box, is the offset angle of the i-th text field bounding box s in the set, obtained from the average offset angle of all text fields in the set S;

(3.3.3) Find the intercept b of the straight line tan(θ _b )x+b, so that all text fields in the set S reach the center point The sum of the distances is the smallest;

(3.3.4) Find the two endpoints (x _p , y _p ) and (x _q , y _q ) of the straight line, p represents the first endpoint, q represents the second endpoint, x _p and y _p are the first The abscissa and ordinate of the first endpoint, x _q and y _q are respectively the abscissa and ordinate of the second endpoint;

(3.3.5) b represents the output bounding box, x _b and y _b are the horizontal and vertical coordinates of the center of the output bounding box, respectively;

(3.3.6) Where w _b is the width of the output bounding box, w _p and w _q are respectively the width of the bounding box centered on point p and the width of the bounding box centered on q;

(3.3.7) h _b is the height of the output bounding box, is the height of the i-th text field bounding box s in the set, obtained from the average height of all text fields in the text field set S;

(3.3.8)b:=(x _b , y _b , w _b , h _b , θ _b ), b is the output bounding box, represented by coordinate parameters, size parameters, and angle parameters;

(3.3.9) Output the combined bounding box b.

Through the above technical solutions conceived by the present invention, compared with the prior art, the present invention has the following technical effects:

(1) Can detect multi-directional text: the text in the natural scene picture is usually in any direction or distorted, the text area of the present invention can be described locally through the text field bounding box, and the text field bounding box can be set to any direction, so Text that can contain multi-directional or distorted shapes.

(2) flexibility is high: the inventive method also can detect the text strip of any length, because the combination text field only depends on the connection of prediction, therefore both can detect word, also can detect text strip;

(3) Robustness is strong: what the present invention method adopts is to carry out partial description with text field bounding box, the method for this partial description can overcome complex natural picture background, capture text feature from picture;

(4) High efficiency: the text field detection model of the inventive method is trained end-to-end, and it can process more than 20 images per second with a size of 512x512, because the text fields and connections are all performed once in the full convolution CNN model Obtained by forward propagation, no offline scaling and rotation of the input image is required;

(5) Strong versatility: Some non-Latin texts do not contain spaces between adjacent words, such as Chinese characters. Existing technologies can only detect words, and are not applicable to this kind of text, because this kind of text without spaces cannot provide visual information for dividing different words. In addition to Latin scripts, the present invention is also capable of detecting long texts in non-Latin scripts, because the method of the present invention does not need to use spaces to provide visual division information.

Description of drawings

Fig. 1 is the flow chart of multidirectional text detection in the natural picture based on text field connection of the present invention;

Fig. 2 is the schematic diagram that the present invention calculates each parameter of text field true label;

Fig. 3 is a schematic diagram of the output composition of the convolutional predictor of the present invention;

Fig. 4 is a text field connection detection model network connection diagram of the present invention;

Fig. 5 is a result diagram of detecting text fields and connecting output bounding boxes of the text image to be detected by using the trained text field connection detection network model in an embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

Below at first explain and illustrate with regard to the technical terms of the present invention:

Convolutional Neural Network (CNN): A neural network that can be used for image classification, regression, and other tasks. Networks usually consist of convolutional layers, downsampling layers, and fully connected layers. The convolutional layer and the downsampling layer are responsible for extracting the features of the image, and the fully connected layer is responsible for classification or regression. The parameters of the network include the parameters and offsets of the convolution kernel and the fully connected layer. The parameters can be learned from the data through the reverse conduction algorithm;

VGG16: The runner-up of ILSVRC in 2014 is VGGNet, which contains 16 CONV/FC layers, has a very uniform architecture, is quite attractive, and only performs 3x3 convolution and 2x2 pooling layers from the beginning to the end, becoming a classic convolutional neural network. Model. Their pretrained models are available plug-and-play with Caffe. It demonstrates that the depth of the network is a key component of good performance.

Depth First Search (DFS): It is an algorithm for traversing or searching a tree or graph. Traverse the nodes of the tree along the depth of the tree, searching the branches of the tree as deep as possible. When all edges of node v have been explored, the search will backtrack to the start node of the edge where node v was found. This process continues until all nodes reachable from the source node have been found. If there are undiscovered nodes, select one of them as the source node and repeat the above process, and the whole process is repeated until all nodes are visited. It belongs to the classic algorithm in graph theory. Using the depth-first search algorithm can generate the corresponding topological sorting table of the target graph. Using the topological sorting table can easily solve many related graph theory problems, such as the maximum path problem and so on.

As shown in Figure 1, the text detection method under the natural scene based on spatial transformation of the present invention comprises the following steps:

(1) Training the text field connection detection network model, including the following sub-steps:

(1.1) Mark the text content of all text images in the training image set with the entry level, and the label is the four point coordinates of the rectangular initial bounding box of the entry to obtain the training data set;

(1.2) Define the text field detection model that can predict the output text field and connection according to the entry label, the network model is composed of a cascaded convolutional neural network and a convolution predictor, and the text field is calculated according to the above-mentioned training data set and connected tags, design a loss function, combine online amplification and online negative sample hard example mining techniques, use the reverse conduction method to train the network, and obtain a text field detection model, including the following sub-steps:

(1.2.1) Constructing a text field detection convolutional neural network model: the first few layers of convolutional units for extracting features come from the pre-trained VGG-16 network, and the first few layers of convolutional units are convolutional layer 1 to pooling layer 5, Fully connected layer 6 and fully connected layer 7 are converted to convolutional layer 6 and convolutional layer 7 respectively, followed by some additional convolutional layers for extracting deeper features for detection, including convolutional layers 8. Convolutional layer 9, convolutional layer 10, and the last layer is convolutional layer 11; the last 6 different convolutional layers output feature maps of different sizes, which is convenient for extracting high-quality features of multiple scales and detecting text Segments and connections are performed on these six feature maps of different sizes; for these six convolutional layers, a filter of size 3×3 is added after each layer as a convolutional predictor to jointly detect text fields and connect;

(1.2.2) Generate text field bounding box labels from marked word bounding boxes: For the original training image set Itr, record the scaled training image set as Itr′, w _I , h _I are the width and height of Itr′ respectively, Can take 384×384 or 512×512 pixels, the i-th picture Itr _i ′ is used as the model input, and all word bounding boxes marked on Itr _i ′ are recorded as W _i =[W _i1 ,...,W _ip ], where W _ij is the jth word bounding box on the i-th picture, the word bounding box can be word level or entry level, j=1,..., p, p is the word bounding box on the i-th picture The total number of ; Note that the feature maps output by the last 6 convolutional layers constitute a set Itro _i ′=[Itro _i1 ′,...,Itro _i6 ′], where Itro _il ′ is the lth in the last 6 convolutional layers The feature map output by the layer, w _l and h _l are the width and height of the feature map respectively, and the coordinates (x, y) on Itro _il ′ correspond to the coordinates of the center point on Itr _i ′ (x _a , y _a ) Horizontal initial bounding boxes B _ilq , they satisfy the following formula:

The width and height of the initial bounding box B _ilq are both set to a constant a _l to control the proportion of the output text field, l=1,...,6; record the feature map Itro _il 'corresponding to the l-th layer output The set of initial bounding boxes is B _il =[B _il1 ,...,B _ilm ], q=1,...,m, where m is the number of initial bounding boxes on the feature map output by layer l; as long as the initial bounding boxes The center of the box B _ilq is contained inside any marked word bounding box W _ij on Itr′, and the size a _l of B _ilq and the height h of the marked word bounding box W _ij satisfy: Then this initial bounding box B _ilq is marked as a positive class, the value of the label is 1, and it matches the word bounding box W _ij with the closest height; otherwise, when B _ilq and all word bounding boxes W _i do not satisfy the above two When a condition is met, B _ilq is marked as a negative class, and the value of the label is 0; the text field is generated on the initial bounding box, which is the same as the label category of the initial bounding box; wherein, the proportional constant 1.5 is an empirical value;

(1.2.3) Generate a text field on the labeled initial bounding box generated in the step (1.2.2) and calculate the offset of the positive text field: the negative text field bounding box s ^- is the negative initial bounding box B ^- ; the bounding box s of the positive text field ⁺ is obtained from the initial bounding box B ⁺ of the positive class through the following steps: a) Record the initial bounding box B of the positive class ⁺ the matched tag word bounding box W and the horizontal direction The angle is θ _s , with the central point of B ⁺ as the center, rotate W clockwise by θ _s angle; b) crop W, and remove the part beyond the left and right of B ⁺ ; c) center the center point of B ⁺ , the cropped word The bounding box W′ is rotated counterclockwise by the θ _s angle, and the geometric parameters x _s , y _s , w _s , h _s , θ _s of the text field s ⁺ the real label are obtained; d) Calculate the offset of the text s ⁺ relative to B ⁺ Quantity (Δx _s , Δy _s , Δw _s , Δh _s , Δθ _s ), the calculation formula is as follows:

x _s ＝a _l Δx _s +x _a

y _s = a _l Δy _s + y _a

w _s = a _l exp(Δw _s )

h _s = a _l exp(Δh _s )

θ _s = Δθ _s

Among them, x _s , y _s , w _s , h _s , and θ _s are respectively the abscissa of the center point, the ordinate of the center point, width, height, and the angle between the text field bounding box s ⁺ and the horizontal direction; x _a , y _a , w _a , h _a are the abscissa, ordinate, width, and height of the center point of the horizontal initial bounding box B ⁺ respectively; Δx _s , Δy _s , Δw _s , Δh _s , and Δθ _s are text fields The offset of bounding box s ⁺ central point abscissa x _s relative to initial bounding box B ⁺ , the offset of vertical coordinate y _s relative to initial bounding box, the offset change of width w _s , and the offset change of height h _s amount, the offset of angle θ _s ;

(1.2.4) Calculate the connection label for the bounding box of the text field generated in step (1.2.3): the text field s is generated on the initial bounding box B, so the connection labels between s and their corresponding initial bounding box B The connection labels between are the same; for the feature map set Itro _i ′=[Itro _i1 ′,...,Itro _i6 ′], if in the initial bounding box set B _il of the same feature map Itro _il ′, two initial bounding box The labels of are all positive classes, and matches the same word, then The intra-layer connection between is marked as a positive class, otherwise it is marked as a negative class; if the initial bounding box in the initial bounding box set B _il corresponding to the feature map Itro _il ′ The initial bounding box in the initial bounding box set B _i(l-1) corresponding to Itro _i(l-1) ′ labels are all positive and match the same word bounding box W _ij , then The cross-layer connections between are marked as positive class, otherwise marked as negative class;

(1.2.5) Take the scaled training image set Itr′ as the input of the text field detection model, predict the text field s output: initialize the weight and bias of the model, set the learning rate of the first 60,000 training iteration steps to 10 ^-3 , Then the learning rate decays to 10 ^-4 ; for the last 6 convolutional layers, at the coordinates (x, y) on the feature map Itro _il ′ of the first layer, (x, y) corresponds to the input image Itr _i ′ with (x _a , y _a ) is the center point coordinates, the initial bounding box B _ilq with a _l as the size, the 3×3 convolutional predictor will predict B _ilq is divided into positive and negative scores c _s , c _s is a two-dimensional vector, and its value range is a decimal between 0-1. Also predicted 5 numbers As the geometric offset when divided into the positive text field s ⁺ , where Respectively, the predicted text field bounding box s ⁺ center point abscissa relative to the offset of the positive initial bounding box B ⁺ , the vertical coordinate relative to the offset of the positive initial bounding box B ⁺ , the offset change in height, Width offset variation, angle offset;

(1.2.6) Predict the output of intra-layer connection and cross-layer connection based on the predicted text field: for intra-layer connection, at the coordinate point (x, y) on the same feature map Itro _il ′, take x-1 ≤x′≤x+1, y-1≤y′≤y+1, the neighbor points (x′, y′) in the range of y-1≤y′≤y+1, when these 8 points correspond to the input image Itr _i ′, the result of , y) The 8 intra-layer neighbor text fields s ^{(x′, y′, l)} connected with the reference text field s ^{(x, y, l)} corresponding to y), the 8 intra-layer neighbor text fields can be expressed as a set:

A 3×3 convolutional predictor predicts s ^{(x, y, l)} and the set of neighbors in the layer The positive and negative scores c _l1 of the connection, c _l1 is a 16-dimensional vector, where w is a superscript, indicating the connection within the layer;

For cross-layer connection, a cross-layer connection connects the text fields corresponding to two points on the feature map output by two consecutive convolutional layers; because each layer of convolutional layer passes through, the width and height of the feature map will be reduced by half , the width w _l and height h _l of the l-th layer output feature map Itro _il ′ are half of the width w _l-1 and height h _l-1 of the l-1 layer feature map Itro _i(l-1) ′, and The initial bounding box scale a _l corresponding to Itro _il ′ is twice the initial bounding box scale a _l-1 corresponding to _{Itro i(l-1)} ′, for (x, y), take 4 cross-layer neighbor points (x', y') in the range of 2x≤x'≤2x+1, 2y≤y'≤2y+1 on the feature map Itro _i(l-1) ', (x, y) on Itro _il ′ corresponds to the initial bounding box on the input image Itr _i ′, and the 4 cross-layer neighbor points on Itro _i(l-1) ′ correspond to the 4 initial bounding boxes on the input image Itr _i ′ The spatial positions of the bounding boxes coincide, and the four cross-layer neighbor text fields s ^{(x′, y′, l-1)} can be expressed as a set:

The 3×3 convolutional predictor will predict the base text field s ^{(x, y, l)} of layer l and the set of adjacent text fields on layer l-1 The positive and negative scores c _l2 of the cross-layer connection, c _l2 is an 8-dimensional vector:

in, Indicates that the predictor predicts the positive and negative scores of the connection between s ^{(x, y, l)} and all 4 adjacent text fields, and c is a superscript, indicating a cross-layer connection;

All intralayer connections and all cross-layer connections Constitute the connection set N _s ;

(1.2.7) Take the text field label obtained in step (1.2.3) and step (1.2.4), the real offset of the positive text field, and the connection label as the output reference, and the text predicted by step (1.2.5) The category and score of the segment, the offset of the predicted text field, and the connection score predicted in step (1.2.6) are the predicted output, and the target loss function between the predicted output and the output benchmark is designed, and the reverse conduction is used for the text field connection detection model The method is continuously trained to minimize the loss of text field classification, text field offset regression and connection classification, and the target loss function is designed for the text field connection detection model. The target loss function is the weighted sum of three losses:

where y _s is the label of all text fields, c _s is the predicted text field score, y _l is the predicted connection label, c _l is the predicted connection score, which is composed of intra-layer connection score c _l1 and cross-layer score c _l2 ; If the i-th initial bounding box is marked as a positive class, then y _s (i) = 1, otherwise 0; L _conf (y _s , c _s ) is the softmax loss of the predicted text field score c _s , L _conf (y _s , c _l ) is the softmax loss of the predicted connection score c _l , is the predicted text field geometry parameter s and the ground truth label The smoothed L between ₁ regression loss; n _s is the number of positive initial bounding boxes, used to normalize text field classification and regression loss; n _l is the total number of positive class connections, used to normalize the connection classification loss ; λ ₁ and λ ₂ are weight constants, which are 1 in practice.

(1.2.8) During the training process of step (1.2.7), the online amplification method is used to amplify the training data I _tr online, and the online negative sample hard case mining strategy is used to balance positive samples and negative samples. Before the training images I _tr are scaled to the same size and loaded in batches, they are randomly cropped into image blocks, and the jaccard overlap coefficient o of each image block and the true bounding box of the text field is the smallest; for multi-directional text, The data augmentation is carried out on the minimum enclosing rectangle of the multi-directional text bounding box, the overlap coefficient o of each sample is randomly selected from 0, 0.1, 0.3, 0.5, 0.7 and 0.9, and the size of the image block is the size of the original image Between 0.1 and 1 times; the training image is not flipped horizontally; in addition, text fields and connection negative samples occupy most of the training samples, and the online negative sample hard case mining strategy is used to balance the positive samples and negative samples, and the text field and connection are separated For mining, control the ratio between negative samples and positive samples not to exceed 3:1.

(2) Use the above-mentioned trained convolutional neural network to perform text field and connection detection on the text image to be detected, including the following sub-steps:

(2.1) Perform text field detection on the text image to be detected. The feature maps output by different convolutional layers can predict text fields of different scales, and the feature maps output by the same convolutional layer can predict text fields of the same scale: the image to be detected The i-th text image Itst _i to be detected in the collection Itst is scaled to a uniform size, and the specific size can be set manually according to the situation of the text image to be detected, and the zoomed text image to be detected is Itst _i ′. Input the image Itst _i ′ into the text field connection detection model trained in step (1.2), and obtain the set Itsto _i ′=[Itsto _i1 ′,...,Itsto _i6 ′], where Itsto _il ′ is the feature map output by the l-th layer in the last 6 convolutional layers, l=1,...,6, the coordinates (x, y on each output feature map Itsto _il ′ ), the 3×3 convolution predictor will predict the initial bounding box B _ilq corresponding to (x, y) is predicted to be the score c _s of positive and negative text fields, and also predicts 5 numbers As the geometric offset when it is predicted as a positive text field s ⁺ ;

(2.2) The text fields on all feature layers detected by the text image to be detected are connected and detected, and the connections include intra-layer connections and cross-layer connections: predict intra-layer connections and cross-layer connections based on the text fields predicted in (2.1) Connection, for intra-layer connections, at the coordinate point (x, y) on the same feature map Itsto _il ′, the 3×3 convolutional predictor predicts s ^{(x, y, l)} and its 8 neighbor text fields The positive and negative scores c _l1 of the intra-layer connection between them; for the cross-layer connection, the 3×3 convolutional predictor will predict the reference text field s ^{(x, y, l)} of the l-th layer and the 4 on the l-1 layer neighbor text field The cross-layer connection positive and negative scores c _l2 , c _l1 and c _l2 constitute the predicted connection score c _l ;

(2.3) Combine the detected text field confidence score and connection confidence score, where the text field confidence score includes the positive and negative category score and offset score of the text field, and use the convolution predictor to output the softmax standardized score: in ( 2.1) Based on the predicted text field, the intra-layer connection and cross-layer connection are predicted. For the intra-layer connection, at the coordinate point (x, y) on the same feature map Itsto _il ′, the 3×3 convolutional predictor predicts s ^{(x, y, l)} and 8 neighbor text fields The positive and negative scores c _l1 of the intra-layer connection between them; for the cross-layer connection, the 3×3 convolutional predictor predicts the reference text field s ^{(x, y, l)} of the l-th layer and the 4 on the l-1 layer Neighbor Text Field The cross-layer connection positive and negative scores c _l2 , c _l1 and c _l2 constitute the predicted connection score c _l .

(3) Combining text fields and connections to obtain an output bounding box, including the following sub-steps:

(3.1) According to the standardized score detected in (2.3), filter the text field and connection output by the convolutional predictor, use the filtered text field as a node, and use the connection as an edge to construct a connection graph: For step (2) to be Detect a fixed number of text fields s and connection N _s generated by inputting the text image to the text field detection model, and filter through their scores; set different filtering thresholds for the text field s and connection N _s , respectively α and β; The filtering threshold can be artificially set different values according to different data. In practice, when performing multi-directional text image detection, α=0.9, β=0.7 can be used, and when multilingual long text image detection is performed, α=0.9 can be used. β=0.5, when performing horizontal text detection, you can take α=0.6, β=0.3; use the filtered text field s' as a node, and the filtered connection N _s ' as an edge, and use them to construct a graph;

(3.2) Perform a depth-first search on the graph to find connected components. Each component is denoted as a set B, which contains text fields connected by connections;

(3.3) The text field set S obtained by performing depth-first search on the graph in step (3.2) is combined into a complete word through the following steps, including:

(3.3.1) Input: |S| is the number of text fields in the set S, where is the i-th text field, i is the superscript, are respectively the central abscissa and ordinate of the i-th text field bounding box s ⁽ⁱ⁾ , are the width and height of text field bounding box s ⁽ⁱ⁾ , respectively, is the angle between the text field bounding box s ⁽ⁱ⁾ and the horizontal direction;

(3.3.2) where θ _b is the offset angle of the output bounding box, is the offset angle of the i-th text field bounding box s in the set, obtained from the average offset angle of all text fields in the set S;

(3.3.3) Find the intercept b of the straight line tan(θ _b )x+b, so that all text fields in the set S reach the center point The sum of the distances is the smallest;

(3.3.4) Find the two endpoints (x _p , y _p ) and (x _q , y _q ) of the straight line, p represents the first endpoint, q represents the second endpoint, x _p and y _p are the first The abscissa and ordinate of the first endpoint, x _q and y _q are respectively the abscissa and ordinate of the second endpoint;

(3.3.5) b represents the output bounding box, x _b and y _b are the horizontal and vertical coordinates of the center of the output bounding box, respectively;

(3.3.6) Where w _b is the width of the output bounding box, w _p and w _q are respectively the width of the bounding box centered on point p and the width of the bounding box centered on q;

(3.3.7) h _b is the height of the output bounding box, is the height of the i-th text field bounding box s in the set, obtained from the average height of all text fields in the text field set S;

(3.3.8)b:=(x _b , y _b , w _b , h _b , θ _b ), b is the output bounding box, represented by coordinate parameters, size parameters, and angle parameters;

(3.3.9) Output the combined bounding box b.

Claims (7)

1. a kind of multidirectional text detection method in the natural picture based on connection text field, it is characterized in that, described method comprises the steps: (1) Training the text field connection detection network model, including the following sub-steps: (1.1) Mark the text content of all text images in the training image set with the entry level, and the label is the four point coordinates of the rectangular initial bounding box of the entry to obtain the training data set; (1.2) Define the text field connection detection network model that can predict the output text field and connection according to the entry label, the text field connection detection network model is composed of a cascade convolutional neural network and a convolution predictor, according to the above training The data set is calculated to obtain the label of the text field and the connection, and the loss function is designed. Combined with the online amplification and online negative sample mining method, the text field connection detection network is trained by the reverse conduction method, and the text field connection detection network model is obtained; (2) Utilize the well-trained above-mentioned text field connection detection network model to carry out text field and connection detection on the text image to be detected, including the following sub-steps: (2.1) Perform text field detection on the text image to be detected, predict text fields of different scales from the feature maps output by different convolution layers, and predict text fields of the same scale from the feature maps output by the same convolution layer; (2.2) text fields on all feature layers detected by the text image to be detected are connected and detected, and the connections include intra-layer connections and cross-layer connections; (2.3) Combining the confidence score and the connection confidence score of the detected text field, wherein the Chinese field confidence score includes the text field positive and negative category score and offset score, and uses the convolution predictor to output the softmax standardized score; (3) Combining text fields and connections to obtain an output bounding box, including the following sub-steps: (3.1) According to the standardized score detected in (2.3), filter the text field and connection output by the convolutional predictor, use the filtered text field as a node, and use the connection as an edge to construct a connection graph; (3.2) Perform a depth-first search on the graph to find interconnected components. Each component is denoted as a set S, which contains text fields connected by connections; (3.3) Combine the text fields in a collection into a complete term, calculate the bounding box of the complete term and output it. 2. the multi-directional text detection method in the natural picture based on connecting text field according to claim 1, is characterized in that, described step (1.2) is specially: (1.2.1) Constructing a text field detection convolutional neural network model: the first few layers of convolutional units for extracting features come from the pre-trained VGG-16 network, and the first few layers of convolutional units are convolutional layer 1 to pooling layer 5, Fully connected layer 6 and fully connected layer 7 are converted to convolutional layer 6 and convolutional layer 7 respectively, followed by some additional convolutional layers for extracting deeper features for detection, including convolutional layers 8. Convolutional layer 9, convolutional layer 10, and the last layer is convolutional layer 11; the last 6 different convolutional layers output feature maps of different sizes, which is convenient for extracting high-quality features of multiple scales and detecting text Segments and connections are performed on these six feature maps of different sizes; for these six convolutional layers, a filter of size 3×3 is added after each layer as a convolutional predictor to jointly detect text fields and connect; (1.2.2) Generate a text field bounding box label from the labeled word bounding box: For the original training image set Itr, record the scaled training image set as Itr', w _I , h _I are the width and height of Itr' respectively, Taking the i-th picture Itr _i ' as the model input, all word bounding boxes marked on Itr _i ' are written as W _i ＝[W _i1 ,...,W _ip ], where W _ij is the i-th picture on the j word bounding boxes, the word bounding boxes are word level or entry level, j=1,...,p, p is the total number of word bounding boxes on Itr _i '; the output of the 6 layers of convolutional layers after the note The feature map constitutes a set Itro _i '=[Itro _i1 ',...,Itro _i6 '], where Itro _il ' is the feature map output by the lth layer in the last 6 convolutional layers, w _l and h _l are the The width and height of the feature map, the coordinates (x, y) on Itro _il ' correspond to the horizontal initial bounding box B _ilq with (x _a , y _a ) as the center point coordinates on Itr _i ', and they satisfy the following formula: The width and height of the initial bounding box B _ilq are set to a constant a _l to control the proportion of the output text field, l=1,...,6; record the feature map Itro _il 'corresponding to the l-th layer output The set of initial bounding boxes is B _il =[B _il1 ,...,B _ilm ], q=1,...,m, where m is the number of initial bounding boxes on the feature map output by layer l; as long as the initial bounding boxes The center of the box B _ilq is contained inside any marked word bounding box W _ij on Itr', and the size a _l of B _ilq and the height h of the marked word bounding box W _ij satisfy: Then this initial bounding box B _ilq is marked as a positive class, the value of the label is 1, and matches the word bounding box W _ij with the closest height; otherwise, when B _ilq and all word bounding boxes W _i do not satisfy the above two When a condition is met, B _ilq is marked as a negative class, and the value of the label is 0; the text field is generated on the initial bounding box, which is the same as the label category of the initial bounding box; (1.2.3) Generate a text field on the labeled initial bounding box generated in the step (1.2.2) and calculate the offset of the positive text field: the negative text field bounding box s ^- is the negative initial bounding box B ^- ; the bounding box s of the positive text field ⁺ is obtained from the initial bounding box B ⁺ of the positive class through the following steps: a) Record the initial bounding box B of the positive class ⁺ the matched tag word bounding box W and the horizontal direction The angle is θ _s , with the central point of B ⁺ as the center, rotate W clockwise by θ _s angle; b) crop W, and remove the part beyond the left and right of B ⁺ ; c) center the center point of B ⁺ , the cropped word The bounding box W' is rotated counterclockwise by the angle θ _s , and the geometric parameters x _s , y _s , w _s , h _s , θ _s of the text field s ⁺ the real label are obtained; d) Calculate the offset of the text s ⁺ relative to B ⁺ Quantity (Δx _s , Δy _s , Δw _s , Δh _s , Δθ _s ), the calculation formula is as follows: x _s ＝a _l Δx _s +x _a y _s = a _l Δy _s + y _a w _s = a _l exp(Δw _s ) h _s = a _l exp(Δh _s ) θ _s = Δθ _s Among them, x _s , y _s , w _s , h _s , and θ _s are respectively the abscissa of the center point, the ordinate of the center point, width, height, and the angle between the text field bounding box s ⁺ and the horizontal direction; x _a , y _a , w _a , h _a are the abscissa, ordinate, width, and height of the center point of the horizontal initial bounding box B ⁺ respectively; Δx _s , Δy _s , Δw _s , Δh _s , and Δθ _s are text fields The offset of bounding box s ⁺ central point abscissa x _s relative to initial bounding box B ⁺ , the offset of vertical coordinate y _s relative to initial bounding box, the offset change of width w _s , and the offset change of height h _s amount, the offset of angle θ _s ; (1.2.4) Calculate the connection label for the bounding box of the text field generated in step (1.2.3): the text field s is generated on the initial bounding box B, so the connection labels between s and their corresponding initial bounding box B The connection labels between are the same; for the feature map set Itro _i '=[Itro _i1 ',...,Itro _i6 '], if in the initial bounding box set B _il of the same feature map Itro _il ', two initial bounding box The labels of are all positive classes, and matches the same word, then The intra-layer connection between is marked as a positive class, otherwise it is marked as a negative class; if the initial bounding box in the initial bounding box set B _il corresponding to the feature map Itro _il ' The initial bounding box in the initial bounding box set B _i(l-1) corresponding to Itro _i(l-1) ' labels are all positive and match the same word bounding box W _ij , then The cross-layer connections between are marked as positive class, otherwise marked as negative class; (1.2.5) Take the scaled training image set Itr' as the input of the text field detection model, and predict the text field s output: initialize the weight and bias of the model, and set the learning rate of the first 60,000 training iteration steps to 10 ^-3 , Afterwards, the learning rate decays to 10 ^-4 ; for the last 6 convolutional layers, at the coordinates (x, y) on the l-th layer feature map Itro _il ', (x, y) corresponds to the input image Itr _i ' with (x _a , y _a ) is the center point coordinates, the initial bounding box B _ilq with a _l as the size, the 3×3 convolutional predictor will predict B _ilq is divided into positive and negative scores c _s , c _s is a two-dimensional vector, the value range is a decimal between 0-1; at the same time, 5 numbers are predicted As the geometric offset when divided into the positive text field s ⁺ , where Respectively, the predicted text field bounding box s ⁺ center point abscissa relative to the offset of the positive initial bounding box B ⁺ , the vertical coordinate relative to the offset of the positive initial bounding box B ⁺ , the offset change in height, Width offset variation, angle offset; (1.2.6) Predict intra-layer connection and cross-layer connection output based on the predicted text field: For intra-layer connection, at the coordinate point (x, y) on the same feature map Itro _il ', take x-1 ≤x'≤x+1, y-1≤y'≤y+1 neighbor points (x', y'), when these 8 points correspond to the input image Itr _i ', then obtained with (x , y) is an intra-layer neighbor text field s ^(x',y',l) connected to the reference text field s ^(x,y,l ), and the eight intra-layer neighbor text fields can be expressed as a set: The 3×3 convolution predictor will predict s ^{(x, y, l)} and the set of neighbors in the layer The positive and negative scores c _l1 of the connection, c _l1 is a 16-dimensional vector, where w is a superscript, indicating the connection within the layer; For cross-layer connection, a cross-layer connection connects the text fields corresponding to two points on the feature map output by two consecutive convolutional layers; because each layer of convolutional layer passes through, the width and height of the feature map will be reduced by half , the width w _l and height h _l of the l-th layer output feature map Itro _il ' is half of the width w _l-1 and height h _l-1 of the l-1 layer feature map Itro _i(l-1) ', and The initial bounding box scale a _l corresponding to Itro _il 'is 2 times the initial bounding box scale a _l-1 corresponding to _{Itro i(l-1)} ', for the (x, y), take 4 cross-layer neighbor points (x', y') in the range of 2x≤x'≤2x+1, 2y≤y'≤2y+1 on the feature map Itro _i(l-1) ', The (x, y) on Itro _il 'corresponds to the initial bounding box on the input image Itr _i ', and the 4 cross-layer neighbor points on Itro _i(l-1) ' correspond to the 4 initial bounding boxes on the input image Itr _i ' The spatial positions of the bounding boxes coincide, and the four cross-layer neighbor text fields s ^{(x', y', l-1)} can be expressed as a set: The 3×3 convolutional predictor will predict the reference text field s ^{(x, y, l)} of layer l and the set of adjacent text fields on layer l-1 The positive and negative scores c _l2 of the cross-layer connection, c _l2 is an 8-dimensional vector: in, Indicates that the predictor predicts the positive and negative scores of the connection between s ^{(x, y, l)} and all 4 adjacent text fields, and c is a superscript, indicating a cross-layer connection; All intralayer connections and all cross-layer connections Constitute the connection set N _s ; (1.2.7) Take the text field label obtained in step (1.2.3) and step (1.2.4), the real offset of the positive text field, and the connection label as the output reference, and the text predicted by step (1.2.5) The category and score of the segment, the offset of the predicted text field, and the connection score predicted in step (1.2.6) are the predicted output, and the target loss function between the predicted output and the output benchmark is designed, and the reverse conduction is used for the text field connection detection model The method is continuously trained to minimize the loss of text field classification, text field offset regression and connection classification, and the target loss function is designed for the text field connection detection model. The target loss function is the weighted sum of three losses: where y _s is the label of all text fields, c _s is the predicted text field score, y _l is the predicted connection label, c _l is the predicted connection score, which is composed of intra-layer connection score c _l1 and cross-layer score c _l2 ; If the i-th initial bounding box is marked as a positive class, then y _s (i)=1, otherwise it is 0; L _conf (y _s ,c _s ) is the softmax loss of the predicted text field score c _s , L _conf (y _s , c _l ) is the softmax loss of the predicted connection score c _l , is the predicted text field geometry parameter s and the ground truth label The smoothing L between ₁ regression loss; n _s is the number of positive initial bounding boxes, used to normalize the text field classification and regression loss; n _l is the total number of positive connections, used to normalize the connection classification loss ; λ ₁ and λ ₂ are weight constants; (1.2.8) In the training process of step (1.2.7), the online amplification method is used to amplify the training data I _tr online, and the online negative sample difficult example mining strategy is used to balance the positive samples and negative samples. Before the training images I _tr are scaled to the same size and loaded in batches, they are randomly cropped into image blocks, and the jaccard overlap coefficient o of each image block and the real bounding box of the text field is the smallest; for multi-directional text, the data Amplification is performed on the minimum enclosing rectangle of the multi-directional text bounding box, the overlap coefficient o of each sample is randomly selected from 0, 0.1, 0.3, 0.5, 0.7, and 0.9, and the size of the image block is 0.1 of the original image size Between -1 times; the training image is not flipped horizontally; in addition, the text field and connection negative samples occupy most of the training samples, and the online negative sample hard case mining strategy is used to balance the positive samples and negative samples, and the text field and connection are separated. Mining, control the ratio between negative samples and positive samples not to exceed 3:1. 3. the multidirectional text detection method in the natural picture based on connection text field according to claim 1 or 2, is characterized in that, described step (2.1) is specially: The i-th text image Itst _i to be detected in the image set Itst to be detected is scaled to a uniform size, and the specific size can be artificially set according to the situation of the text image to be detected, and the text image to be detected after the scaling is recorded as Itst _i '; The image Itst _i ' is input to the text field connection detection model trained in step (1.2), and the set Itsto _i '=[Itsto _i1 ',...,Itsto _i6 composed of the feature maps respectively output by the last 6 convolutional layers is obtained '], where Itsto _il 'is the feature map output by the lth layer in the last 6 convolutional layers, l=1,...,6, the coordinates (x,y) on each output feature map Itsto _il ' , the 3×3 convolution predictor will predict the initial bounding box B _ilq corresponding to (x, y) is predicted as the score c _s of the positive and negative text fields, and also predicts 5 numbers As the geometric offset when it is predicted as the positive text field s ⁺ . 4. the multidirectional text detection method in the natural picture based on connecting text field according to claim 1 or 2, is characterized in that, described step (2.2) is specially: Predict the intra-layer connection and cross-layer connection based on the text field predicted in (2.1). For the intra-layer connection, at the coordinate point (x, y) on the same feature map Itsto _il ', the 3×3 convolutional predictor predicts Output s ^{(x, y, l)} and its 8 neighbor text fields The positive and negative scores c _l1 of the intra-layer connection between them; for the cross-layer connection, the 3×3 convolution predictor will predict the reference text field s ^(x,y,l) of the l-th layer and the 4 on the l-1 layer neighbor text field The cross-layer connection positive and negative scores c _l2 , c _l1 and c _l2 constitute the predicted connection score c _l . 5. the multidirectional text detection method in the natural picture based on connecting text field according to claim 1 or 2, is characterized in that, described step (2.3) is specially: According to the results of step (2.1) and step (2.2), at the coordinates (x, y) on each feature map Itsto _il ', the score c _s of the predicted text field and the offset of the text field The four items of intra-layer connection score c _l1 and cross-layer connection score c _l2 are concatenated into a 33-dimensional vector, and an additional softmax layer is added after the output channel of the convolutional predictor to standardize the text field score and connection score respectively. 6. the multidirectional text detection method in the natural picture based on connecting text field according to claim 1 or 2, is characterized in that, described step (3.1) is specifically; For the fixed number of text fields s and connection N _s generated by inputting the text image to be detected into the text field detection model in step (2), filter by their scores; set different filtering thresholds for the text field s and connection N _s , are α and β respectively; take the filtered text field s' as a node, and the filtered connection N _s ' as an edge, and use them to construct a graph. 7. The method for detecting multi-directional text in natural pictures based on connected text fields according to claim 1 or 2, wherein the step (3.3) is specifically: performing a depth-first search on the graph for step (3.2) The obtained text field set S is combined into a complete word through the following steps, including: (3.3.1) Input: |S| is the number of text fields in the set S, where is the i-th text field, i is the superscript, are respectively the central abscissa and ordinate of the i-th text field bounding box s ⁽ⁱ⁾ , are the width and height of text field bounding box s ⁽ⁱ⁾ , respectively, is the angle between the text field bounding box s ⁽ⁱ⁾ and the horizontal direction; (3.3.2) where θ _b is the offset angle of the output bounding box, is the offset angle of the i-th text field bounding box s in the set, obtained from the average offset angle of all text fields in the set S; (3.3.3) Find the intercept b of the straight line tan(θ _b )x+b, so that all text fields in the set S reach the center point The sum of the distances is the smallest; (3.3.4) Find the two endpoints (x _p , y _p ) and (x _q , y _q ) of the straight line, p represents the first endpoint, q represents the second endpoint, x _p and y _p are the first The abscissa and ordinate of the first endpoint, x _q and y _q are respectively the abscissa and ordinate of the second endpoint; (3.3.5) b represents the output bounding box, x _b and y _b are the horizontal and vertical coordinates of the center of the output bounding box, respectively; (3.3.6) Where w _b is the width of the output bounding box, w _p and w _q are respectively the width of the bounding box centered on point p and the width of the bounding box centered on q; (3.3.7) h _b is the height of the output bounding box, is the height of the i-th text field bounding box s in the set, obtained from the average height of all text fields in the text field set S; (3.3.8)b:=(x _b , y _b , w _b , h _b , θ _b ), b is the output bounding box, represented by coordinate parameters, size parameters, and angle parameters; (3.3.9) Output the combined bounding box b.