CN114743201A - A multimeter reading recognition method and system based on rotating target detection - Google Patents

Abstract

The invention discloses a multimeter reading recognition method and system based on rotating target detection. The method includes: processing an image to be tested based on an improved YOLOv5 model, and outputting a reading area rotation frame, a reading area rotation angle and a switch rotation angle; The rotation angle of the reading area and the rotation angle of the transfer switch are used to calculate the actual rotation angle of the transfer switch; the actual rotation angle of the transfer switch is matched with the unit matching information to obtain the reading unit result; the rotation frame of the reading area is subjected to affine transformation and identification to obtain the reading number Result; Combine the reading digital result with the reading unit result to get the complete reading result. The system includes: a detection module, an actual rotation angle calculation module, a unit matching module, a digital identification module and a reading integration module. By using the present invention, the complete reading result under any rotation angle of the multimeter with a changeover switch can be detected and identified.

Description

A multimeter reading recognition method and system based on rotating target detection

technical field

The invention relates to the field of intelligent manufacturing in China, in particular to a multimeter reading recognition method and system based on rotating target detection.

Background technique

The automation and intelligence of industrial instrument reading recognition is the general trend. The instrument reading recognition technology based on computer vision can automatically identify the collected instrument and meter numerical information, and quickly enter it into the business system, effectively solve the problems of copying errors and omissions in the manual transcription process, improve transcription efficiency and reduce labor Entry workload, reduce enterprise labor costs, and automate instrument data entry.

The existing multimeter reading recognition methods mainly firstly detect the reading area and crop it, and then perform reading recognition on the cropped reading image, but most of them only focus on the detection and recognition of horizontal readings, and often ignore the recognition of reading units, which leads to application The value and scope of application are limited. The method based on traditional image processing requires a single environmental background and is greatly disturbed by environmental changes such as illumination, so the universality is poor. In contrast, YOLOv5 is a one-stage deep learning target detection algorithm without RPN structure, which can detect the read region well, and has a small backbone structure such as tiny-darknet, which can make the model lighter through a series of optimization measures. Quantization is suitable for deployment on mobile terminals as engineering algorithms, and has high practical application value. However, as a general target detection algorithm, YOLOv5 is not suitable for rotating target detection with a large inclination angle, and it is impossible to determine the target orientation and angle.

SUMMARY OF THE INVENTION

In order to solve the above technical problems, the purpose of the present invention is to provide a multimeter reading identification method and system based on rotating target detection, which can detect and identify the complete reading results of a multimeter with a switch at any rotation angle.

The first technical solution adopted by the present invention is: a multimeter reading identification method based on rotating target detection, comprising the following steps:

Based on the improved YOLOv5 model, the image to be tested is processed, and the rotation frame of the reading area, the rotation angle of the reading area and the rotation angle of the switch are output;

Calculate the actual rotation angle of the transfer switch according to the rotation angle of the reading area and the rotation angle of the transfer switch;

Match the actual rotation angle of the transfer switch with the unit matching information to get the reading unit result;

Perform affine transformation on the rotation frame of the reading area and identify it to obtain the digital result of the reading;

Integrate the reading numerical result with the reading unit result to obtain the complete reading result.

Further, the training steps of the improved YOLOv5 model specifically include:

Obtain the training image and label the rotating frame, encode the coordinates of the four corners of the rotating frame as geometric elements, and obtain the labeled image;

Data enhancement is performed on the labeled images to obtain a training set;

Input the training set to the YOLOv5 model;

Perform feature extraction, feature fusion, and frame regression in sequence to obtain rotation frame information;

The model parameters are updated by combining the ground truth label calculation loss to obtain an improved YOLOv5 model.

Further, the rotation frame information includes the center point coordinates, width and height, angle, confidence, bias information, probability of each category and probability of each orientation category of the rotation frame.

Further, the detection head output formula of the improved YOLOv5 model is expressed as follows:

In the above formula, n _h is the number of detection heads, _{b s} is the number of images to be tested in the input model, na is the number of preset frame anchor points of the grid unit, H _i is the output height of the ith detection head, W _i is the width of the output of the i-th detection head, (x, y) is the center coordinate of the prediction frame, (w, h) is the width and height of the prediction frame, conf is the confidence level of the prediction target, and C is the prediction frame belonging to each The probability of the category, n _c is the number of predicted target categories, r _j is the rotation box regression parameter, and O _k is the probability of each orientation category of the target.

Further, the step of matching the actual rotation angle of the transfer switch with the unit matching information to obtain the result of the reading unit specifically includes:

Obtain unit matching information, and obtain a unit information data set and an angle information data set;

Match the actual rotation angle of the transfer switch with the angle information data set to obtain the angle index value;

Match the angle index value to the unit info dataset to get the reading unit result.

Further, the step of performing affine transformation and identification on the rotating frame of the reading area, and obtaining the digital result of the reading, specifically includes:

Perform affine transformation on the rotating frame of the reading area to obtain a horizontal reading area map;

The affine transformation includes the combined transformation of translation, scaling, rotation, flipping and staggering of the rotation frame of the reading area;

Based on the CRNN read identification model, CNN feature extraction, RNN sequence modeling and CTC transcription were performed on the horizontal read region map in turn, and the read number results were obtained.

Further, the matrix formula of the affine transformation is as follows:

In the above formula, (x, y) is the coordinates of the rotation frame point in the reading area, (u, v) is the rotation frame point obtained by the affine transformation of the rotation frame point in the reading area, a ₀ , a ₁ , a ₂ , b ₀ , b ₁ and b ₂ are M parameters, and M is an affine transformation matrix.

The second technical solution adopted by the present invention is: a multimeter reading identification system based on rotating target detection, comprising:

The detection module, based on the improved YOLOv5 model, processes the image to be tested, and outputs the rotation frame of the reading area, the rotation angle of the reading area and the rotation angle of the switch;

The actual rotation angle calculation module is used to calculate the actual rotation angle of the transfer switch according to the rotation angle of the reading area and the rotation angle of the transfer switch;

The unit matching module is used to match the actual rotation angle of the transfer switch with the unit matching information to obtain the reading unit result;

The digital recognition module is used to perform affine transformation and recognition on the rotating frame of the reading area, and obtain the digital result of the reading;

The reading integration module is used to integrate the reading digital result and the reading unit result to obtain the complete reading result.

The beneficial effects of the method and system of the present invention are: by improving the YOLOv5 algorithm, the difficulty in identifying the reading of a multimeter under any rotation angle is solved, and a method for identifying the reading unit is proposed for a series of multimeters with a switch, so that the reading information is more complete, and The method is suitable for a class of digital multimeters with transfer switches, which expands the application scope and application value.

Description of drawings

Fig. 1 is the step flow chart of a kind of multimeter reading identification method based on rotating target detection of the present invention;

Fig. 2 is a kind of structural block diagram of the multimeter reading identification system based on rotating target detection of the present invention;

3 is a schematic diagram of a specific embodiment of the present invention to improve the YOLOv5 rotation area detection model;

4 is a schematic diagram of a specific embodiment of the present invention to improve the output rotation frame of YOLOv5;

5 is a schematic diagram of a transfer switch according to a specific embodiment of the present invention;

6 is a schematic diagram of a reading area correction according to a specific embodiment of the present invention;

FIG. 7 is a structural block diagram of a CRNN reading recognition model according to a specific embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The numbers of the steps in the following embodiments are only set for the convenience of description, and the sequence between the steps is not limited in any way, and the execution sequence of each step in the embodiments can be adapted according to the understanding of those skilled in the art Sexual adjustment.

1, the present invention provides a multimeter reading recognition method based on rotating target detection, the method includes the following steps:

S1. Referring to Figure 3, train the YOLOv5 model to obtain an improved YOLOv5 model;

S1.1. Obtain the training image and label the rotating frame, encode the coordinates of the four corners of the rotating frame as geometric elements, and obtain the labeled image;

Specifically, the coordinates of the four corners of the rotating frame are encoded as geometric elements with reference to the mode of FIG. 4 , and the specific operations are as follows:

||R _j ||=w×Sigmoid(r _j ), j∈{0,1};

In the above formula, R _j is the distance between the position of the predicted rotation frame j and the predicted horizontal frame j; w is the width of the horizontal frame predicted by the grid unit, and Sigmoid(r _j ) is the regression parameter of the frame. The normalized value, r _j is the border offset regression parameter.

S1.2, perform data enhancement on the labeled images to obtain a training set;

S1.3. Input the training set to the YOLOv5 model;

S1.4, perform feature extraction, feature fusion and frame regression in sequence to obtain the rotation frame information;

Specifically, the rotation frame information includes the center point coordinates, width and height, angle, confidence, bias information, probability of each category, and probability of each orientation category of the rotation frame.

Among them, the calculation operation of the rotation frame angle is as follows:

When r _j ≠ 0, that is, when there is a rotation frame, start from the vertical upward direction, and calculate the rotation angle in a clockwise direction

为标注后图像的四个顶点的坐标，

为旋转框角度。In the above formula,

are the coordinates of the four vertices of the labeled image,

is the rotation frame angle.

计算出目标的旋转角度

When r _j = 0, that is, there is only a horizontal frame, then the direction can be predicted by

Calculate the rotation angle of the target

为各朝向类别概率中最大概率的朝向，其具体计算操作如下：in,

is the orientation with the largest probability among the probabilities of each orientation category, and its specific calculation operations are as follows:

In the above formula, P(O _k ) is the probability of each orientation category, and its specific calculation operation is as follows:

P(O _k )=Sigmoid(O _k ),k∈{0,1,2,3}.

S1.5, update the model parameters by calculating the loss of the true value label, and obtain an improved YOLOv5 model.

Specifically, the loss function calculation operation of the model is as follows:

L=λ _box L _box +λ _obj L _obj +λ _cls L _cls +λ _rbox L _rbox +λ _rot L _rot ;

In the above formula, λ _box , λ _obj , λ _cls , λ _rbox and λ _rot are the weight coefficients of L _box , L _obj , L _cls , L _rbox and L _rot respectively; L _box is the loss function of the horizontal box bbox, using CIOU Loss loss function; L _obj is the target confidence loss function, using the Focal Loss loss function; L _cls is the prediction category loss function, using the Focal Loss loss function; L _rbox is the rotation frame offset regression loss function, using the Smooth _L1 loss function; L _rot is the classification loss function for each orientation, and the Focal Loss loss function is used.

Among them, the rotation box offset regression loss function L _rbox is expressed as follows:

为r_j的真实值，r_j为旋转框偏移回归参数，Smooth_L1为计算r_j′和

之间的回归误差损失的损失函数，其具体操作如下：In the above formula, r _j ′ is the normalized value of r _j ,

is the real value of r _j , r _j is the rotation frame offset regression parameter, Smooth _L1 is the calculation of r _j ′ and

The loss function between the regression error loss and the specific operation is as follows:

The classification loss function L _rot of each orientation is expressed as follows:

In the above formula, α _k is the weight coefficient used to balance positive and negative samples, and γ is used to balance simple and difficult samples.

S2. Process the image to be tested based on the improved YOLOv5 model, and output the rotation frame of the reading area, the rotation angle of the reading area and the rotation angle of the switch;

Specifically, the improved YOLOv5 model consists of an input end, a Backbone feature extraction backbone network, a Neck feature enhancement network, and a Head detection head.

Among them, the specific calculation operation of the output of the improved YOLOv5 model detection head is as follows:

In the above formula, n _h is the number of detection heads, _{b s} is the number of images to be tested in the input model, na is the number of preset frame anchor points of the grid unit, H _i is the output height of the ith detection head, W _i is the width of the output of the i-th detection head, (x, y) is the center coordinate of the prediction frame, (w, h) is the width and height of the prediction frame, conf is the confidence level of the prediction target, and C is the prediction frame belonging to each The probability of the category, n _c is the number of predicted target categories, r _j is the rotation box regression parameter, and O _k is the probability of each orientation category of the target.

S3. Calculate the actual rotation angle of the transfer switch according to the rotation angle of the reading area and the rotation angle of the transfer switch;

Specifically, the specific calculation operation of the actual rotation angle of the transfer switch is:

In the above formula, θ _d is the rotation angle of the reading area, θ _g is the rotation angle of the transfer switch, and θ _r is the actual rotation angle of the transfer switch.

where _θd and _θg are

S4. Referring to Figure 5, match the actual rotation angle of the transfer switch with the unit matching information to obtain the unit result of the reading;

S4.1. Obtain unit matching information, and obtain a unit information data set and an angle information data set;

Specifically, the unit information is stored in the array U, U={U _l l∈{1,2,...,n _u }};

In the above formula, U _l represents the lth unit, and n _u represents the number of types of transfer switch units;

The angle information corresponding to each U _l is stored in the array θ, θ={θ _l l∈{1,2,...,n _u }}:

In the above formula, θ _l represents the rotation angle corresponding to the unit U _l in the transfer switch.

S4.2. Match the actual rotation angle of the transfer switch with the angle information data set to obtain the angle index value;

Specifically, the matching operation formula between the actual rotation angle of the switch and the angle information data set is as follows:

m = argmin _l |θ _r -θ _l |;

In the above formula, m is the index value of the angle closest to θ _r in θ.

S4.3. Match the angle index value with the unit information data set to obtain the reading unit result.

Specifically, the matching operation formula between the angle index value and the unit information data set is as follows:

S5, referring to Fig. 6, carry out affine transformation and identification to the rotating frame of the reading area, and obtain the digital result of reading;

S5.1. Perform affine transformation on the rotating frame of the reading area to obtain a horizontal reading area map;

Specifically, the original rotation frame point coordinates (x, y) are transformed into coordinates (u, v) by affine transformation, and the specific operations are as follows:

Its matrix representation is as follows:

In the above formula, M is the affine transformation matrix, which can perform translation, scaling, rotation, flipping and staggered transformation on the rotation frame of the reading area, a ₀ , a ₁ , a ₂ , b ₀ , b ₁ and b ₂ are M parameter.

S5.2. Referring to Figure 7, based on the CRNN read identification model, CNN feature extraction, RNN sequence modeling and CTC transcription are sequentially performed on the horizontal read area map to obtain the read number result;

S5.2.1. Input the horizontal reading area map into the CRNN reading recognition model;

Specifically, the CRNN read recognition model consists of a CNN feature extraction module, an RNN sequence modeling module, and a CTC transcription module, and is a stable end-to-end text recognition framework.

S5.2.2. Perform feature extraction on the horizontal reading area map based on the CNN feature extraction module to obtain a convolution feature map;

Specifically, the horizontal reading area image is input into the MobileNetv3 lightweight convolutional neural network network to extract the overall feature information, and the convolutional feature map is output.

S5.2.3. Perform sequence modeling on the convolution feature map based on the RNN sequence modeling module to obtain a posterior probability matrix;

Specifically, in order to enhance the feature extraction capability of the RNN network before and after the sequence, this method uses the deep bidirectional LSTM network to extract the sequence features before and after the multimeter reading; the feature map size of the input Bi-LSTM is (1, T, D), and its height is 1, The maximum time length is T, each input vector has dimension D, and the Bi-LSTM output y is a posterior probability matrix.

S5.2.4, transcribe the posterior probability matrix based on the CTC transcription module, and obtain the digital result of reading.

Specifically, the CTC transcription module inserts each time slice of the posterior probability matrix y into softmax to obtain the predicted sequence, removes the repeated tags in the sequence, and then removes the invalid space tags in the sequence, so as to translate the predicted sequence containing spaces. It is text information; it is assumed that the prediction sequence output by the bidirectional LSTM network is "--11-2-22-3--", which becomes the final prediction result "1223" after being transcribed by CTC.

S6. Integrate the reading digital result and the reading unit result to obtain a complete reading result.

2, a multimeter reading recognition system based on rotating target detection, including:

The detection module, based on the improved YOLOv5 model, processes the image to be tested, and outputs the rotation frame of the reading area, the rotation angle of the reading area and the rotation angle of the switch;

The actual rotation angle calculation module is used to calculate the actual rotation angle of the transfer switch according to the rotation angle of the reading area and the rotation angle of the transfer switch;

The unit matching module is used to match the actual rotation angle of the transfer switch with the unit matching information to obtain the reading unit result;

The digital recognition module is used to perform affine transformation and recognition on the rotating frame of the reading area, and obtain the digital result of the reading;

The reading integration module is used to integrate the reading digital result and the reading unit result to obtain the complete reading result.

The contents in the above method embodiments are all applicable to the present system embodiments, the specific functions implemented by the present system embodiments are the same as the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

The beneficial effects of the present invention specifically include:

1) Able to detect and identify the reading unit result at any rotation angle of the multimeter: based on YOLOv5 expansion, the detection head adds the output of the target rotation frame offset regression parameter and the target orientation probability parameter, which can be completed with only one model. Detection of target rotation frame and angle; based on the multimeter reading area and the relationship between the position and attitude of the transfer switch, combined with the angle of the reading area and the angle of the transfer switch, the actual angle of the transfer switch can be calculated to obtain the reading unit.

2) The recognition accuracy of the multimeter reading digital result is improved and the reading recognition steps are simplified: the rotating frame of the detected reading area is affine transformed into a horizontal frame for reading recognition. The commonly used reading recognition method consists of two steps of reading character segmentation and recognition, while the CRNN algorithm does not need to perform reading character segmentation, and can complete the reading recognition end-to-end.

The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments, and those skilled in the art can make various equivalent deformations or replacements without departing from the spirit of the present invention. , these equivalent modifications or substitutions are all included within the scope defined by the claims of the present application.

Claims (8)

1. a multimeter reading identification method based on rotating target detection, is characterized in that, comprises the following steps: Based on the improved YOLOv5 model, the image to be tested is processed, and the rotation frame of the reading area, the rotation angle of the reading area and the rotation angle of the switch are output; Calculate the actual rotation angle of the transfer switch according to the rotation angle of the reading area and the rotation angle of the transfer switch; Match the actual rotation angle of the transfer switch with the unit matching information to get the reading unit result; Perform affine transformation on the rotation frame of the reading area and identify it to obtain the digital result of the reading; Integrate the reading numerical result with the reading unit result to obtain the complete reading result. 2. a kind of multimeter reading identification method based on rotating target detection according to claim 1, is characterized in that, the training step of described improved YOLOv5 model specifically comprises: Obtain the training image and label the rotating frame, encode the coordinates of the four corners of the rotating frame as geometric elements, and obtain the labeled image; Data enhancement is performed on the labeled images to obtain a training set; Input the training set to the YOLOv5 model; Perform feature extraction, feature fusion, and frame regression in sequence to obtain rotation frame information; The model parameters are updated by combining the ground truth label calculation loss to obtain an improved YOLOv5 model. 3. a kind of multimeter reading recognition method based on rotating target detection according to claim 2, is characterized in that, described rotating frame information comprises the center point coordinates, width and height, angle, confidence, offset information, each Class probabilities and class probabilities for each orientation. 4. a kind of multimeter reading identification method based on rotating target detection according to claim 1 is characterized in that, the detection head output formula of described improved YOLOv5 model is expressed as follows:

In the above formula, n _h is the number of detection heads, _{b s} is the number of images to be tested in the input model, na is the number of preset frame anchor points of the grid unit, H _i is the output height of the ith detection head, W _i is the width of the output of the i-th detection head, (x, y) is the center coordinate of the prediction frame, (w, h) is the width and height of the prediction frame, conf is the confidence level of the prediction target, and C is the prediction frame belonging to each The probability of the category, n _c is the number of predicted target categories, r _j is the rotation box regression parameter, and O _k is the probability of each orientation category of the target. 5. a kind of multimeter reading identification method based on rotating target detection according to claim 1, is characterized in that, described with actual rotation angle of transfer switch and unit matching information are matched, obtain this step of reading unit result, specifically comprises: Obtain unit matching information, and obtain a unit information data set and an angle information data set; Match the actual rotation angle of the transfer switch with the angle information data set to obtain the angle index value; Match the angle index value to the unit info dataset to get the reading unit result. 6. a kind of multimeter reading identification method based on rotating target detection according to claim 1, is characterized in that, described to reading area rotating frame is carried out affine transformation and identification, obtains this step of reading digital result, specifically comprises: Perform affine transformation on the rotation frame of the reading area to obtain a horizontal reading area map; The affine transformation includes the combined transformation of translation, scaling, rotation, flipping and staggering of the rotation frame of the reading area; Based on the CRNN read identification model, CNN feature extraction, RNN sequence modeling and CTC transcription were performed on the horizontal read region map in turn, and the read number results were obtained. 7. a kind of multimeter reading identification method based on rotating target detection according to claim 6 is characterized in that, the matrix formula of described affine transformation is as follows:

In the above formula, (x, y) is the coordinates of the rotation frame point in the reading area, (u, v) is the rotation frame point obtained by the affine transformation of the rotation frame point in the reading area, a ₀ , a ₁ , a ₂ , b ₀ , b ₁ and b ₂ are M parameters, and M is an affine transformation matrix. 8. A multimeter reading recognition system based on rotating target detection is characterized in that, comprising: The detection module, based on the improved YOLOv5 model, processes the image to be tested, and outputs the rotation frame of the reading area, the rotation angle of the reading area and the rotation angle of the switch; The actual rotation angle calculation module is used to calculate the actual rotation angle of the transfer switch according to the rotation angle of the reading area and the rotation angle of the transfer switch; The unit matching module is used to match the actual rotation angle of the transfer switch with the unit matching information to obtain the reading unit result; The digital recognition module is used to perform affine transformation and recognition on the rotation frame of the reading area, and obtain the digital result of the reading; The reading integration module is used to integrate the reading digital result and the reading unit result to obtain the complete reading result.

Images