CN113269067B - Periodic industrial video clip key frame two-stage extraction method based on deep learning - Google Patents

Abstract

The invention relates to a two-stage extraction method for key frames of periodic industrial video segments based on deep learning. The method includes: acquiring an industrial video image, extracting a region of interest, preprocessing, and obtaining a preprocessed image sequence; constructing a semantic segmentation network model based on deep learning, and extracting the target area of the preprocessed image; in the first stage, constructing The convolutional neural network classifies the preprocessed image, and divides its time series to obtain a set of candidate key frame sequences; in the second stage, constructing the similarity matrix of the target area, and performing a process on the candidate key frame sequence Clustering, screening and fusion, get keyframes. The present invention aims at the problem that the characteristics of industrial video are complex and the current method lacks globality and locality. It introduces deep learning technology and uses the two-stage idea of "first global and then local" to extract key frames of industrial video faster and more accurately. It is of guiding significance to optimize production and realize quality improvement and production increase.

Description

Two-stage key frame extraction method for periodic industrial video clips based on deep learning

Technical Field

The present invention relates to the fields of machine vision, image processing and pattern recognition, and in particular to a two-stage extraction method of key frames of periodic industrial video clips based on deep learning.

Background Art

The cyclical production process is a common industrial production process. In this type of process, a series of established procedures are executed repeatedly. For example, in the steel sintering process, there is a cyclical production process of "laying → ignition → trolley movement → unloading"; for another example, in the injection molding process, the series of procedures of "closing mold → filling → pressure holding → cooling → mold opening → demolding" are executed cyclically.

Industrial video is an intuitive representation and indirect reflection of the working condition information of the industrial production process. For a certain production process, the key frame is the image in the monitoring video clip that best reflects the current working condition characteristics of the industrial production process, and is one of the important characteristic parameters for evaluating the current production condition of the process. However, due to the complexity of the industrial process, the current extraction of key frames has the following problems.

(1) Dynamics of the production cycle

Theoretically, for a periodic production process, the time interval between each key frame can be determined when the production rate is constant. After the first key frame is manually determined, the key frames in the subsequent production process can be determined according to the production rate. However, due to the fluctuations of factors such as materials, fuel, operation, and environment, the production cycle often fluctuates to a certain extent, resulting in the inability to determine the time interval between key frames.

(2) Similarity between processes

In the actual production process, different processes are often carried out in the same place, which makes the monitoring videos of each process have many similar scenes. For example, in the tail section monitoring video of the sintering process, there is a common scene of "sintering material layer" between the two processes of "trolley operation" and "unloading", while the "burning zone" image unique to the "unloading" process only occupies a small part of this scene. From the perspective of image features, this leads to the similarity between the images of the two processes. However, the traditional manual feature extraction method cannot effectively overcome this similarity, which makes it difficult to segment the process video clips.

(3) Similarity within the process

In the actual production process, the movement of production equipment, as well as various physical and chemical changes of materials and products, are often continuous changes. The differences between the frames of the monitoring video are small and mainly reflected in the spatial position and texture. Traditional manual features cannot effectively express this difference. For example, in the tail section monitoring video of the sintering process, the main differences between the cross-sectional images of the "unloading" process are mainly manifested in the changes in the spatial distribution and texture of the combustion zone. Simple manual features such as brightness and histogram cannot accurately describe this change. This problem leads to the difficulty of extracting key frames of process video clips.

Therefore, how to overcome the above problems, accurately extract industrial video image features, and quickly realize key frame extraction of periodic industrial video clips is an urgent problem to be solved in industrial process condition assessment.

Summary of the invention

Based on this, the present invention proposes a key frame extraction method based on deep learning to address the above-mentioned technical problems. The purpose is to solve the technical problems that the time intervals of key frames in the existing key frame extraction process cannot be determined, the similarity of processes cannot be effectively overcome, and the features cannot be accurately described. This method provides a method for accurately extracting industrial video image features and quickly realizing key frame extraction of periodic industrial video clips.

The present invention provides a two-stage extraction method of key frames of periodic production segments of industrial videos based on deep learning, which specifically includes:

S1: Acquire industrial video images, extract interest area images, and perform preprocessing to obtain preprocessed image sequences;

S2: constructing a semantic segmentation network model based on deep learning to extract the image target area from the preprocessed image sequence;

S3: Obtain the output features of the middle layer of the semantic segmentation network model in step S2, and construct a convolutional neural network model to perform binary classification on the preprocessed image sequence to obtain image category features;

S4: Segmenting the preprocessed image sequence according to the image category features to obtain a set of candidate key frame sequences;

S5: calculating the similarity of each image target area in the candidate key frame sequence set, constructing a similarity matrix, and taking the similarity matrix as input, performing clustering processing on the candidate key frame sequence to obtain a multi-category image set;

S6: According to the actual needs of the industrial process, a key frame selection index and a weight matrix are constructed, the multi-category image set is screened according to the key frame selection index to obtain a key frame sequence, and the key frame sequence is weighted averaged according to the weight matrix to obtain a key frame.

Furthermore, the preprocessing in step S1 includes denoising, color correction and dehazing.

Furthermore, the step S2 specifically includes:

Randomly select a plurality of first typical images from the preprocessed image sequence, and screen out a first mask image to construct a first training set and a first test set;

Performing translation, scale, brightness and rotation transformation on the first training set and the first test set to obtain enhanced training set and test set;

Constructing a deep semantic segmentation network model, inputting the network model with the enhanced training set as input, and testing the network model with the enhanced test set to obtain a trained deep semantic segmentation network model;

The preprocessed image is subjected to category feature extraction using a trained deep semantic segmentation network model to obtain image category features.

Furthermore, the step S3 specifically includes:

Randomly selecting a plurality of second typical images from the preprocessed image sequence, and classifying the second typical images according to actual requirements of the industrial process to construct a second training set and a second test set;

Taking the second training set and the second test set as input, using the deep semantic segmentation model in step S2 for simulation, and obtaining the output of the middle layer of the model as the image depth feature;

Constructing a convolutional neural network model, taking the image depth features as input and classification as output, training and testing multiple networks to obtain a trained convolutional neural network model;

The preprocessed image is subjected to feature extraction using a trained convolutional neural network model to obtain image category features.

Furthermore, the step S4 specifically includes:

Constructing a temporary image sequence and setting a minimum image sequence length; traversing the preprocessed image sequence, extracting the category features of the current image, and determining whether the image belongs to the target image;

If the current image is the target image, the current image is added to the temporary image sequence, and the number of target images increases by 1. When the number of the temporary image sequence is greater than the minimum image sequence length, all images in the temporary image sequence except the last image are added to the current target image sequence; the current target image sequence set is the candidate key frame sequence set.

Furthermore, the constructing of the similarity matrix specifically includes:

中任意两张图像I_n和I_m，利用所述深度语义分割网络提取相应目标区域Mask_n和Mask_m，并计算Mask_n和Mask_m之间的相似度

Get candidate key frame sequence

For any two images I _n and I _m in the image, the deep semantic segmentation network is used to extract the corresponding target regions Mask _n and Mask _m , and the similarity between Mask _n and Mask _m is calculated.

表示Mask_n和Mask_m之间的匹配特征描述子数量，W和H分别为图像的宽度和长度，∑∑Mask_n和∑∑Mask_m分别表示目标区域Mask_n和Mask_m的面积，K_n和K_m分别表示Mask_n和Mask_m的特征描述子数量；in,

represents the number of matching feature descriptors between Mask _n and Mask _m , W and H are the width and length of the image respectively, ∑∑Mask _n and ∑∑Mask _m represent the areas of the target regions Mask _n and Mask _m respectively, K _n and K _m represent the number of feature descriptors of Mask _n and Mask _m respectively;

中所有图像之间的相似度，得相似度矩阵Calculate candidate key frame sequence

The similarity between all images in the , and the similarity matrix

Furthermore, the clustering process specifically includes:

According to actual industrial needs, the number of categories D is set, and the similarity matrix is used as input to perform clustering operations on the corresponding candidate key frame sequences to obtain a multi-category image set.

Furthermore, the step S6 specifically includes:

Taking the actual needs of the industrial process as the goal, constructing a key frame selection target according to the target area of the image, and selecting images from the category image set to obtain a key frame sequence;

Taking the actual needs of the industrial process as the goal, a weight matrix is constructed according to the target area of the image, and the images in the key frame sequence are weighted averaged to obtain the key frame.

Beneficial effects:

The two-stage method for extracting key frames of periodic production fragments of industrial videos based on deep learning described in the above-mentioned embodiment of the present invention makes up for the shortcomings of traditional manual feature methods in the ability to extract industrial image features by introducing deep learning technology, and can extract image features completely and accurately; by generating a candidate key frame sequence, a preliminary rough screening is performed on a large number of industrial images, which reduces the computational complexity of the clustering operation in the second stage and improves its accuracy; the candidate key frame sequence is segmented by clustering operation, which improves the similarity between images in the key frame sequence, reduces the computational complexity of key frame synthesis, and avoids the interference of noise images; the method of synthesizing key frames by weighted average of multiple images is adopted to minimize the feature loss during the image change process, and can more completely reflect the visual information in the industrial production process.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic flow chart of a two-stage method for extracting key frames of periodic industrial video clips based on deep learning according to the present invention;

FIG2 is a typical original ROI image and a preprocessed image thereof provided by an embodiment of the present invention;

FIG3 is a schematic diagram of a deep semantic segmentation network structure provided by an embodiment of the present invention;

FIG4 is a typical preprocessed image and its image target area provided by an embodiment of the present invention;

FIG5 is a schematic diagram of a clustering result provided by an embodiment of the present invention;

FIG6 is a typical key frame provided by an embodiment of the present invention;

FIG. 7 is a comparison diagram of the key frame extraction effects of various methods provided by the embodiments of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

As shown in FIG1 , in an embodiment of the present invention, a flowchart of a two-stage key frame extraction method for periodic industrial video clips based on deep learning is proposed, which specifically includes the following steps:

Step S1, acquiring industrial video images, extracting images of regions of interest, and performing preprocessing to obtain preprocessed image sequences.

In an embodiment of the present invention, the acquired industrial video image is cropped with a fixed width and height to remove useless background in the image, extract the region of interest (ROI) image, and then perform preprocessing operations such as denoising, color correction and defogging on the ROI image to reduce defects such as noise, uneven illumination and fogging caused by different lighting, high temperature and dust in the image, and obtain a preprocessed image sequence, such as the ROI image and its preprocessed image shown in Figure 2.

Step S2, constructing a semantic segmentation network model based on deep learning to extract the image target area from the preprocessed image sequence.

In an embodiment of the present invention, firstly, a plurality of first typical images are randomly selected from the preprocessed images, and a first mask image is screened out to construct a first training set and a first test set; the first training set and the first test set images are subjected to data enhancement operations such as random translation, scale, brightness and rotation transformation to obtain an enhanced training set and a test set; a deep semantic segmentation network is constructed, as shown in FIG3 , wherein the input sintering section size is 1024×128×3, and the overall structure includes four encoder layers and four corresponding decoder layers. Each encoder layer includes two 3×3 convolution layers, a batch normalization layer, and a maximum pooling layer; each decoder layer includes an upsampling layer, a 3×3 convolution layer, a concatenate layer, two 3×3 convolution layers, and a batch normalization layer. Finally, after two 3×3 convolution layers are activated by a Sigmoid activation function, a combustion zone morphology with a size of 1024×128×1 is output. FIG3 shows the deep semantic segmentation network structure designed in this paper. Then, the enhanced training set and the enhanced test set are selected to train and test the network. Cross Entropy is used as the loss function and Adam is used as the optimizer with a learning rate of 3×10 ^-4 during training. The trained deep semantic segmentation network is used to extract the target area of the preprocessed image. The extraction result is shown in FIG4 .

Step S3, obtaining the output features of the middle layer of the semantic segmentation network model in step S2, and constructing a convolutional neural network model to perform binary classification on the preprocessed image sequence to obtain image category features.

In the embodiment of the present invention, the idea of transfer learning is introduced, and a plurality of second typical images are randomly selected from the preprocessed image sequence, and the second typical images are classified according to the actual needs of the industrial process to construct a second training set and a second test set; the second training set and the second test set are used as input, and the deep semantic segmentation model in step S2 is used for simulation, and the output of the middle layer of the model is obtained as the image depth feature; a convolutional neural network model is constructed, which mainly includes a Flatten layer, a 128-dimensional fully connected layer, a batch normalization layer, a 2-dimensional fully connected layer and a Sigmoid activation layer, and the image depth feature is used as input and the manual classification result is used as output to train the network, and cross entropy (Cross Entropy) is used as the loss function during training, and Adam is used as the optimizer, and its learning rate is 3× ^10-4 ; the preprocessed image is subjected to feature extraction using the trained convolutional neural network model to obtain image category features.

Step S4: segment the preprocessed image sequence according to the image category features to obtain a set of candidate key frame sequences.

In the embodiment of the present invention, the segmentation process specifically includes:

Step S41, inputting a preprocessed image sequence S _input and a minimum image sequence length δ;

和临时图像序列T，以及目标图像数量C_g＝0和非目标图像数量Cn_g＝0；Step S42, defining the current target image sequence

and a temporary image sequence T, and a target image number C _g = 0 and a non-target image number C n _{g =} 0;

Step S43, traversing the image sequence S _input , extracting the category features of the current image I;

Step S44, determine whether the image I is the target image, if yes, jump to step S45; otherwise, jump to step S47;

Step S45, adding the image I to the temporary image sequence T, and increasing the number of target images _Cg by 1;

Step S46, determining whether the number of target images _Cg is greater than or equal to the minimum image sequence length δ, if so, setting the number of non-target images _Cng = 0;

Step S47, increment the number of non-target images C _ng by 1, and determine whether the number of target images C _g is greater than or equal to the minimum image sequence length δ. If yes, add the image I to the temporary image sequence T;

Step S48, determining whether the number of non-target images C _ng is greater than or equal to the minimum image sequence length δ, if yes, jump to step S49; otherwise, jump to step S412;

Step S49, determining whether the target image quantity _Cg is greater than or equal to the minimum image sequence length δ, if yes, jump to step S410; otherwise, jump to step S411;

Step S410: add all images except the last δ images in the temporary image sequence T to the current target image sequence

Step S411, clearing the number of target images _Cg and the number of non-target images _Cng , and clearing the temporary image sequence T;

Step S412, repeating steps S43 to S411 until the image sequence S _input is terminated;

Step S413: Obtain a candidate key frame sequence set

Step S5, calculating the similarity of each image target area in the candidate key frame sequence set, constructing a similarity matrix, and using the similarity matrix as input, performing clustering processing on the candidate key frame sequence to obtain a multi-category image set.

中任意两张图像I_n和I_m，利用所述深度语义分割网络提取相应目标区域Mask_n和Mask_m；首先使用STFI算法提取Mask_n和Mask_m的SIFT特征描述集合

和

其中

和

分别为128维的特征描述子；然后计算F_n中特征描述子

与F_m中各特征描述子

之间的欧式距离In the embodiment of the present invention, the candidate key frame sequence is taken

Take any two images I _n and I _m in the image, and use the deep semantic segmentation network to extract the corresponding target regions Mask _n and Mask _m ; first use the STFI algorithm to extract the SIFT feature description set of Mask _n and Mask _m

and

in

and

They are 128-dimensional feature descriptors respectively; then calculate the feature descriptors in F _n

and each feature descriptor in _Fm

The Euclidean distance between

在F_m的匹配特征描述子And select the feature descriptor with the smallest distance as

Matching feature descriptors in _Fm

在F_n中的匹配特征描述子为

如果Similarly, we can get the feature descriptor in _Fm

The matching feature descriptor in _Fn is

if

与

为M_i和M_j之间的匹配特征描述子；Then it is called

and

is the matching feature descriptor between _Mi and _Mj ;

Considering the temporal law of the industrial process and the similarity between the images in the candidate key frame sequence, the similarity between Mask _n and Mask _m is recorded as

表示Mask_n和Mask_m之间的匹配特征描述子数量，W和H分别为图像的宽度和长度，∑∑Mask_n和∑∑Mask_m分别表示Mask_n和Mask_m的面积，K_n和K_m分别表示Mask_n和Mask_m的特征描述子数量；in

represents the number of matching feature descriptors between Mask _n and Mask _m , W and H are the width and length of the image respectively, ∑∑Mask _n and ∑∑Mask _m represent the areas of Mask _n and Mask _m respectively, _Kn and _Km represent the number of feature descriptors of Mask _n and Mask _m respectively;

中所有图像之间的相似度，得相似度矩阵Calculate candidate key frame sequence

The similarity between all images in the , and the similarity matrix

进行聚类操作，得到多类别图像集合C＝(c₁,c₂,…,c_d,…,c_D)，其聚类结果示意图如图5所示。In the embodiment of the present invention, the clustering process specifically includes: combining the actual industrial production, dividing the production process into early stage, middle stage and late stage, selecting the number of categories D=3; using the spectral clustering algorithm, taking the similarity matrix _Ai as input, and clustering the corresponding candidate key frame sequences

A clustering operation is performed to obtain a multi-category image set C = (c ₁ , c ₂ , …, c _d , …, c _D ), and a schematic diagram of the clustering result is shown in FIG5 .

Step S6, constructing a key frame selection index and a weight matrix according to the actual needs of the industrial process, screening the multi-category image set according to the key frame selection index to obtain a key frame sequence, and performing weighted averaging on the key frame sequence according to the weight matrix to obtain a key frame.

In the embodiment of the present invention, the actual needs of the industrial process are taken as the target, and it is considered that the key frame sequence needs to meet the total area of the target area, which is the largest and located in the middle of the production cycle; based on the target area, a key frame selection index is constructed.

Where N is the number of images in the candidate key frame sequence.

The best image set is selected from the multi-category image set C = (c ₁ , c ₂ , ..., c _d , ..., c _D ) to obtain a key frame sequence

中所有图像，以所述权值矩阵W为权，计算其加权平均，得关键帧I_key，其结果如图6所示。Taking the actual needs of the industrial process as the goal, according to the image target area, the target area area is used as the weight to construct a weight matrix W = [w ₁ ,w ₂ ,…,w _K ]; for the key frame sequence

For all images in , the weight matrix W is used as the weight, and the weighted average is calculated to obtain the key frame I _key , and the result is shown in FIG6 .

In an embodiment of the present invention, FIG. 7 shows the result of extracting the key frames of industrial videos using the peak method of the image feature curve with the target area as the feature. FIG. A, B and C are the key frames extracted by the production expert, the key frames extracted by the peak method of the image feature curve and the key frames extracted by the method proposed in this article and their corresponding image target areas, respectively. In order to evaluate the excellence of the two methods, the present invention uses the mean hash distance, the difference hash distance, the perceptual hash distance, the cosine distance and the SIFT matching feature point matching rate (the percentage of SIFT feature points of the key frames extracted by the production expert being matched) to calculate the similarity between the two methods and the key frames proposed by the production expert. Among them, the smaller the mean hash distance, the difference hash distance and the perceptual hash distance, the higher the similarity between the two images; the larger the cosine distance and the SIFT matching feature point matching rate, the higher the similarity between the two images. Table 1 shows the evaluation results of the above method on the two algorithms, and it can be seen that the method proposed in this invention can extract key frames more accurately.

Table 1 Similarity between different methods and key frames proposed by production experts

According to production experts and the method proposed in this paper, the images in frame 1 in Figure 7 belong to the same production cycle, but the image feature curve peak method divides it into three cycles. It can be seen that the method proposed in this invention has a higher accuracy rate in extracting key frames.

The two-stage method for extracting key frames of periodic production fragments of industrial videos based on deep learning described in the above-mentioned embodiment of the present invention makes up for the shortcomings of traditional manual feature methods in the ability to extract industrial image features by introducing deep learning technology, and can extract image features completely and accurately; by generating a candidate key frame sequence, a preliminary rough screening is performed on a large number of industrial images, which reduces the amount of clustering calculations in the second stage and improves the accuracy of clustering; the candidate key frame sequence is secondary segmented using a clustering operation, which improves the similarity between images in the key frame sequence, reduces the amount of key frame calculation, and avoids the interference of noise images; the method of synthesizing key frames by weighted averaging of multiple images is used to minimize the feature loss during the image change process, and can more completely reflect the visual information in the industrial production process.

The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and/or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims (8)

1. A periodic industrial video clip key frame two-stage extraction method based on deep learning is characterized by specifically comprising the following steps:

s1: acquiring an industrial video image, extracting an interest area image, and preprocessing to obtain a preprocessed image sequence;

s2: constructing a semantic segmentation network model based on deep learning, and extracting an image target region from the preprocessed image sequence;

s3: acquiring the output characteristics of the middle layer of the semantic segmentation network model in the step S2, constructing a convolutional neural network model, and performing secondary classification on the preprocessed image sequence to obtain image classification characteristics;

s4: segmenting the preprocessed image sequence according to the image category characteristics to obtain a candidate key frame sequence set;

s5: calculating the similarity of each image target area in the candidate key frame sequence set, constructing a similarity matrix, and performing clustering processing on the candidate key frame sequence by taking the similarity matrix as input to obtain a multi-class image set;

s6: according to actual requirements of an industrial process, constructing a key frame selection index and a weight matrix, screening the multi-category image set according to the key frame selection index to obtain a key frame sequence, and performing weighted average on the key frame sequence according to the weight matrix to obtain a key frame;

the segmentation processing in step S4 specifically includes:

step S41, inputting a pre-processing image sequence S _input And a minimum image sequence length δ;

step S42, defining the current target image sequence

And a temporary image sequence T, and a number of target images C _g =0 and number of non-target images C _ng ＝0；

Step S43, traversing the image sequence S _input Extracting the category characteristics of the current image I;

step S44, judging whether the image I is a target image, if so, jumping to step S45; otherwise, jumping to step S47;

step S45, adding the image I to the temporary image sequence T and making the number of target images C _g Self-increment by 1;

step S46, judging the number C of target images _g Whether the image sequence length is larger than or equal to the minimum image sequence length delta or not, if so, the number of non-target images C is made _ng ＝0；

Step S47, make the number of non-target images C _ng Increasing by 1, and judging the number of target images C _g Whether the length is larger than or equal to the minimum image sequence length delta, if so, adding the image I to the temporary image sequence T;

step S48, judging the number C of non-target images _ng Whether the image sequence length is larger than or equal to the minimum image sequence length delta or not is judged, if yes, the step S49 is skipped; otherwise, jumping to step S412;

step S49, judging the number C of target images _g Whether the length is larger than or equal to the minimum image sequence length delta or not is judged, if yes, the step S410 is skipped to; otherwise, jumping to step S411;

in the step S410, the process is executed,adding all images except the last delta images in the temporary image sequence T to the current target image sequence

Step S411, counting the number of target images C _g And the number of non-target images C _ng Clearing and clearing the temporary image sequence T;

step S412, repeating steps S43 to S411 until the image sequence S _input Terminating;

step S413, obtaining a candidate key frame sequence set

2. The periodic industrial video segment key frame two-stage extraction method based on deep learning of claim 1, wherein the preprocessing in the step S1 comprises denoising, color correction and defogging.

3. The periodic industrial video segment key frame two-stage extraction method based on deep learning of claim 1, wherein the step S2 specifically comprises:

randomly selecting a plurality of first typical images from the preprocessed image sequence, screening out first mask images, and constructing a first training set and a first testing set;

carrying out translation, scale, brightness and rotation transformation processing on the first training set and the first test set to obtain an enhanced training set and a test set;

constructing a deep semantic segmentation network model, inputting the network model by taking the enhanced training set as input, and testing the network model by using the enhanced testing set to obtain a trained deep semantic segmentation network model;

and extracting a target region from the preprocessed image by adopting a trained deep semantic segmentation network model.

4. The periodic industrial video clip key frame two-stage extraction method based on deep learning according to claim 1, wherein the step S3 specifically comprises:

randomly selecting a plurality of second typical images from the preprocessed image sequence, classifying the second typical images according to the actual requirements of the industrial process, and constructing a second training set and a second testing set;

taking the second training set and the second testing set as input, simulating by adopting the depth semantic segmentation network model in the step S2, and obtaining the output of the middle layer of the model as the depth characteristic of the image;

constructing a convolutional neural network model, taking the image depth characteristics as input and classification as output, and training and testing multiple networks to obtain a trained convolutional neural network model;

and performing feature extraction on the preprocessed image by adopting a trained convolutional neural network model to obtain image category features.

5. The periodic industrial video segment key frame two-stage extraction method based on deep learning of claim 1, wherein the step S4 specifically comprises:

constructing a temporary image sequence and setting the length delta of the minimum image sequence; traversing the preprocessed image sequence, extracting the category characteristics of the current image, and judging whether the image belongs to a target image;

if the current image is the target image, adding the current image to the temporary image sequence, increasing the number of the target images by 1, and adding all images except the last image in the temporary image sequence to the current target image sequence when the number of the temporary image sequence is greater than the length delta of the minimum image sequence; and the current target image sequence set is a candidate key frame sequence set.

6. The periodic industrial video segment key frame two-stage extraction method based on deep learning of claim 1, wherein the constructing of the similarity matrix specifically comprises:

taking a sequence of candidate key frames

Any two images I in _n And I _m Extracting corresponding target area Mask by using the deep semantic segmentation network model _n And Mask _m And calculating Mask _n And Mask _m The degree of similarity therebetween->

Wherein,

represents Mask _n And Mask _m The number of matched feature descriptors between W and H are the width and length of the image, sigma Mask _n Sum sigma Mask _m Respectively represent target regions Mask _n And Mask _m Area of (C), K _n And K _m Respectively represent Mask _n And Mask _m The number of feature descriptors of (a);

computing a candidate sequence of key frames

In all images to obtain a similarity matrix>

7. The periodic industrial video segment key frame two-stage extraction method based on deep learning of claim 1, wherein the clustering process specifically comprises:

and setting the category quantity D according to the actual industrial requirements, and carrying out clustering operation on the corresponding candidate key frame sequence by taking the similarity matrix as input to obtain a multi-category image set.

8. The periodic industrial video segment key frame two-stage extraction method based on deep learning of claim 1, wherein the step S6 specifically comprises:

constructing a key frame selection index according to the image target region by taking the actual requirement of the industrial process as a target, and selecting an image from the category image set to obtain a key frame sequence;

and taking actual requirements of the industrial process as targets, constructing a weight matrix according to the image target region, and carrying out weighted average on the images in the key frame sequence to obtain key frames.

Images