CN114782417A - Real-time detection method for digital twin characteristics of fan based on edge enhanced image segmentation - Google Patents
Real-time detection method for digital twin characteristics of fan based on edge enhanced image segmentation Download PDFInfo
- Publication number
- CN114782417A CN114782417A CN202210680421.8A CN202210680421A CN114782417A CN 114782417 A CN114782417 A CN 114782417A CN 202210680421 A CN202210680421 A CN 202210680421A CN 114782417 A CN114782417 A CN 114782417A
- Authority
- CN
- China
- Prior art keywords
- fan
- edge
- real
- network
- digital twin
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000003709 image segmentation Methods 0.000 title claims abstract description 23
- 238000011897 real-time detection Methods 0.000 title claims abstract description 15
- 238000012549 training Methods 0.000 claims abstract description 17
- 238000005728 strengthening Methods 0.000 claims description 22
- 238000001514 detection method Methods 0.000 claims description 21
- 238000013528 artificial neural network Methods 0.000 claims description 10
- 238000013527 convolutional neural network Methods 0.000 claims description 10
- 239000013598 vector Substances 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 9
- 230000001965 increasing effect Effects 0.000 claims description 9
- 230000002708 enhancing effect Effects 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 4
- 238000004088 simulation Methods 0.000 claims description 4
- 238000013135 deep learning Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 230000002093 peripheral effect Effects 0.000 claims description 3
- 230000007797 corrosion Effects 0.000 claims description 2
- 238000005260 corrosion Methods 0.000 claims description 2
- 238000010276 construction Methods 0.000 claims 1
- 239000007787 solid Substances 0.000 claims 1
- 238000000605 extraction Methods 0.000 abstract description 3
- 230000007547 defect Effects 0.000 abstract description 2
- 238000013136 deep learning model Methods 0.000 abstract 1
- 230000011218 segmentation Effects 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000003708 edge detection Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- JEIPFZHSYJVQDO-UHFFFAOYSA-N iron(III) oxide Inorganic materials O=[Fe]O[Fe]=O JEIPFZHSYJVQDO-UHFFFAOYSA-N 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000007935 neutral effect Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
- G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/12—Edge-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20192—Edge enhancement; Edge preservation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30164—Workpiece; Machine component
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a real-time detection method for digital twin characteristics of a fan based on edge enhancement image segmentation. The method comprises the steps of training probability distribution in image data in a supervised mode from a large number of fan surface image samples with labels, improving the capacity of extracting image features from a model extraction image, simultaneously constructing a new fan model in a virtual environment by utilizing a digital twin method, adding defects at a reasonable position on a fan, carrying out simulated shooting in the virtual environment by using a camera, and marking generated pictures, thereby expanding data set samples during training and ensuring the generalization capacity of the deep learning model in classification and segmentation in wider complex images.
Description
Technical Field
The invention relates to the field of computer vision and the field of digital twinning, in particular to a real-time detection method for the digital twinning characteristics of a fan based on edge-enhanced image segmentation.
Background
The Digital Twin technology (Digital Twin) fully utilizes data such as a physical model, sensor updating, operation history and the like, integrates a multidisciplinary, multi-physical quantity, multi-scale and multi-probability simulation process, and finishes mapping of a physical world in a virtual world so as to reflect the whole life cycle process of corresponding entity equipment. A digital twin may be viewed as a digital mapping system of one or more important, interdependent equipment systems that are the ties of the physical world interacting and fusing with the virtual world.
The image feature classification method is to identify semantic information of image partition sub-regions into corresponding feature classes, wherein the semantic information comprises subdivision features such as colors, contours and textures. The traditional method is time-consuming and labor-consuming for the expression of features, and can only reliably operate on processing the same and well-made parts, but the traditional algorithm becomes more and more complex as the types and the number of defects increase. The development of the quality detection field is promoted by the appearance of a deep neural network model, and the deep neural network can quickly identify effective variables which are difficult to capture by a traditional machine vision method in an image through learning experience with a great number of images and having a potential of progress, such as: illumination, curved surfaces, or fields of view, etc., to accomplish reliable identification of complex features under more complex conditions.
The edge detection is an important problem in image processing and computer vision, the purpose of the edge detection is to reflect important events and changes of attributes in the form of irregular edges on the basis of image feature detection, and the image edge detection greatly reduces irrelevant information of images and only retains important structural attributes of the images.
Wind energy has been developed in recent years as a promising renewable energy source, and the frequency of accidents of wind turbines has also increased year by year. The method has the advantages that the primary damage characteristic detection is carried out on the surface of the wind turbine in time, and the method has important significance for reducing catastrophic accidents of the wind turbine and increasing the effective working time of the wind turbine.
Disclosure of Invention
The invention aims to provide a real-time detection method for the digital twin characteristics of a fan based on edge-enhanced image segmentation, which introduces a data set expansion mode based on digital twin and an image segmentation technology based on a deep neural network, enhances the edge recognition capability of the fan, and supervises probability distribution in image data from a large number of fan surface image samples with labels while generating a large amount of effective virtual data, thereby improving the capability of extracting the digital characteristics in the image by a model and ensuring the classification generalization capability of the model in wider complex images. And finally, carrying the trained model by using the unmanned aerial vehicle supporting remote high-precision control, and realizing high-automation, low-cost, quick and accurate fan surface target feature recognition in a field scene.
The purpose of the invention is realized by the following technical scheme:
a real-time detection method for a fan digital twin characteristic based on edge enhanced image segmentation comprises the following steps:
(1) selecting a plurality of fan pictures with surface characteristics of a certain specific fan, modeling the specific fan and the peripheral scene of the fan in a virtual environment on the basis of the selected fan pictures, and increasing the surface characteristics of the fan;
(2) carrying out simulation shooting on a virtual fan model with surface characteristics by using a camera in a virtual scene to obtain a digital twin virtual fan picture with the surface characteristics;
(3) firstly, expanding and enhancing a data set of a real fan picture and a digital twin virtual fan picture, and marking out areas of all surface characteristics displayed in the pictures and the category of each surface characteristic, wherein the categories comprise a vortex generator panel, vortex generator panel loss and corrosion;
(4) selecting Mask RCNN as a basic network for image segmentation, and enhancing the edge recognition effect of the Mask RCNN to obtain an edge-enhanced Mask RCNN; abstracting the areas of the fan surface features in all the fan pictures obtained in the step (3) and the category of the surface features into a digital vector and a scalar respectively, wherein the digital vector is used as the input of the edge strengthening Mask RCNN, and the scalar is used as a comparison label for classification of the edge strengthening Mask RCNN; the edge strengthening Mask RCNN is a deep learning image segmentation neural network after the weight occupied by the boundary of a prediction region in a loss function is increased during Mask RCNN training;
(5) inputting the fan pictures, the digital vectors and the comparison labels which are expanded and enhanced in the step (3) into the edge strengthening Mask RCNN for training to obtain edge strengthening Mask RCNN weight parameters with optimized parameters;
(6) and inputting the picture to be tested into the edge strengthening Mask RCNN after parameter optimization, and outputting the category to which all the surface features on the picture belong and the area of each surface feature in real time.
Further, the edge-enhanced Mask RCNN is formed by sequentially connecting four parts of a deep convolutional neural network, an RPN network, a RoIAlign network, a parallel full-connection layer and an improved FCN network in series; the deep convolutional neural network is used for extracting information of different scales of an input image, the RPN network is used for generating areas of surface features to be detected with different sizes, the RoIAlign network is used for generating the areas of the surface features to be detected with different sizes into digital features with fixed length, the full connection layer is used for class detection, and the improved FCN network is used for area detection of the surface features.
Further, the deep convolutional neural network selects Resnet 101.
Further, in the step (1), the virtual scene is constructed based on three-dimensional fusion of the entity photos, and the virtual fan and fan characteristics are generated based on the real fan model and the existing characteristic image.
Further, in the step (2), the virtual camera automatically rotates around the fan body through setting, and the virtual camera automatically shoots.
Further, in the step (3), firstly, the fan picture with the surface features is subjected to sample picture expansion by using two image enhancement methods of pyramid and patch enhancement, then all the expanded pictures are further expanded through left-right turning, contrast normalization and Gaussian blur, and then the area of each surface feature and the category of each surface feature on the picture are marked.
Further, in the step (3), the contour of each surface feature is labeled to represent the region where the surface feature is located, and the category to which each surface feature belongs is defined as an integer.
Further, the feature map output by the deep convolutional neural network is used as the input of the RPN network, a plurality of candidate frames with the length-to-width ratio of 1:2, 1:1, 2:1 and the sizes of 32, 64, 128, 256 and 512 are generated, and the candidate frames with the target object are reserved through detection of the RPN network.
The invention has the following beneficial effects:
(1) compared with the traditional data set expansion means such as image scaling, turning, rotation and the like, the method adopts a digital twin method to reconstruct the fan. In the reconstructed model, different types of characteristics with different shapes can be added to a proper position, and meanwhile, the environmental conditions such as illumination conditions, climate conditions and the like can be changed, so that the images of the fan under various conditions are enriched during training, and the adaptability of the neural network to the environment and the change of the characteristics of the fan during detection is improved.
(2) Compared with the existing target detection network, the method adopts the Mask RCNN with the edge enhancement, increases the weight of the edge during training, can predict more accurate pixel-level edges, and is more beneficial to the detection of the surface characteristics of the fan.
(3) The method can quickly identify the effective variable which is difficult to capture by the traditional machine vision method in the image, and can complete reliable identification of complex characteristics under complex working conditions. The method can accurately detect the edge of the surface characteristic of the wind turbine and the type of the characteristic, thereby detecting the damage characteristic of the surface of the wind turbine in time and maintaining the damage characteristic, reducing the catastrophic accidents of the wind turbine and increasing the effective working time of the wind turbine.
Drawings
FIG. 1 is a schematic diagram of a digital twinning expansion data set method.
Fig. 2 is a schematic diagram of an image enhancement method.
FIG. 3 is a schematic diagram of dataset tagging.
Fig. 4 is a schematic diagram of an image segmentation process implemented by the edge enhancement Mask RCNN.
FIG. 5 is a schematic diagram of the overall structural framework of the edge strengthening Mask RCNN.
Fig. 6 is a schematic diagram of RPN generation candidate sub-regions.
Fig. 7 is a schematic diagram of FCN network generation detection edge.
Detailed Description
The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments, and the objects and effects of the present invention will be more apparent, it being understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.
The invention discloses a real-time detection method for a fan digital twinning characteristic based on edge-enhanced image segmentation, which comprises the following steps of:
the method comprises the following steps: selecting a plurality of fan pictures with surface characteristics of a certain specific fan, modeling the specific fan and the peripheral scene of the fan in a virtual environment on the basis of the pictures, and increasing the surface characteristics of the fan;
the virtual scene is constructed based on three-dimensional fusion of the entity photos, and the virtual fan and fan characteristics are generated based on a real fan model and the existing characteristic images.
Step two: carrying out simulation shooting on a virtual fan model with surface characteristics by using a camera in a virtual scene to obtain a fan picture with the surface characteristics; as shown in fig. 1.
Step three: and selecting digital twin generated and real fan pictures with clear surface characteristics, and amplifying the selected pictures. Fig. 2 shows a schematic diagram of a pyramid + image enhancement method in the present invention. And enlarging or reducing the original image according to a certain proportion. Meanwhile, a sliding window with a fixed size is used for intercepting the magnified or reduced image, a target feature is ensured in the sliding window in the intercepting process, and finally the intercepted image is used as a new data sample.
In this embodiment, a 1.5-fold ratio is adopted for enlargement and reduction, 1161 pictures of the zoomed and original images are intercepted by using a 1000 pixel × 1000 pixel sliding window, 1969 images with 1000 pixel × 1000 pixel containing certain target features are obtained, and both are used as data sets of a training network.
As can be seen from fig. 2, in the original 4000 pixel × 3000 pixel image, the proportion of the target area to the whole image area is very small, which is not beneficial to extracting features from the target area by using a neural network. In the 1000 pixel by 1000 pixel image intercepted by the sliding window, the proportion of the target area to the whole image area is increased greatly, and the recognition of the characteristics of the target area by the neural network is facilitated. And then further expanding all the expanded pictures through left-right turning, contrast normalization and Gaussian blur.
The areas of all surface features shown in the picture are marked out, and each surface feature belongs to a category, wherein the categories comprise a vortex generator panel, a vortex generator panel missing category and a rust category.
As shown in fig. 3, the present embodiment outlines the features on the picture, and replaces the shapes of all the features with special-shaped polygons, and the categories of the features are expressed by integers (in the present invention, the 1-vortex generator panel VG, the 2-vortex generator panel lacks teeth VGMT, 3-RUST run). And only keeping each vertex and the category code number of the polygon in the generated label, and compressing the label into a Json file.
Step four: and abstracting the areas of the surface features of the fan in all the fan pictures obtained in the first step and the second step and the category of the surface features into a digital vector and a scalar respectively, wherein the digital vector is used as the input of the edge strengthening Mask RCNN, and the scalar is used as a comparison label for classifying the edge strengthening Mask RCNN.
Step five: inputting the fan picture, the digital vector and the comparison label which are expanded and enhanced in the third step into the edge strengthening Mask RCNN for training to obtain an edge strengthening Mask RCNN weight parameter with optimized parameters;
step six: and inputting the picture to be tested into the edge strengthening Mask RCNN after parameter optimization, and outputting the category to which all the surface features on the picture belong and the area of each surface feature in real time.
The edge-enhanced Mask RCNN is a deep learning image segmentation neural network which increases the weight occupied by the boundary of the prediction region in the loss function during the training of the Mask RCNN. The edge-enhanced Mask RCNN is formed by sequentially connecting a deep convolutional Neural Network, an RPN (Region probability Network candidate area generation Network), a RoIAlign (Region of Interest alignment) Network, a parallel Full-connection layer and an improved FCN (Full Connect Neural Network) Network in series. The deep convolutional neural network is used for extracting information of different scales of an input image, and Resnet101 is selected in the embodiment; the candidate area generation network is used for generating areas with different sizes of surface features to be detected; the RoIAlign network is used for generating the digital features with fixed length in the areas of the surface features to be detected with different sizes; the full connection layer is used for carrying out category detection; the improved FCN network is used for area detection of surface features.
The loss function of the edge strengthening Mask RCNN is improved, the weight occupied by the boundary of a prediction region during training is increased, so that the accurate prediction result near the region edge is ensured, and the loss function of the edge strengthening Mask RCNN is known as follows:
in the formula: w is aiFor the weight coefficient, for points inside the real label region, wi=1, true markPoints of the edge of the label area, wi=w(w>1). Firstly, convolving a real label picture by using a convolution kernel g of 3 x 3, filling the boundary with 0, setting the step length to be 1, obtaining the picture with the same size as the original picture after convolution, setting the corresponding element value of an edge point in the original picture to be a positive number, setting the corresponding element value of an internal point in a label area to be zero, and setting the corresponding element value of an external point in the label area to be a negative number. Then, each element is subjected to a step function, and a weight matrix of the loss function can be obtainedW=y(g*YTrue) Wherein y () is a step function,
，YTrueif the label is a real label, the pixel belonging to the target is 1, the pixel belonging to the background is 0, and then the loss function of the edge strengthening Mask RCNN branch is a weight matrixWAnd cross entropy on each pixelL P Average of the hadamard products (multiplied by element):
. The weight occupied by the boundary pixels in the training is enhanced, so that the generated mask is more accurate, and better positioning performance is provided.
Fig. 4 is a flowchart of using the edge enhanced Mask RCNN network of the present invention, and fig. 5 is a structural diagram of the edge enhanced Mask RCNN. The network training is mainly carried out on a GPU with more prominent repeated computing capability. In order to reduce the pressure of the data set on the GPU memory, preprocessing operation is firstly carried out on the data set image, and the image label is compressed. The method comprises the steps of firstly extracting a boundary frame in a label, and then compressing label color blocks in the boundary frame into 56 pixel by 56 pixel small pictures, thereby reducing the reading amount of data. When the compressed label color blocks are stored, data are not rounded, but stored as floating point numbers, and the compressed label color blocks can be accurately restored.
Then, the core Network structure part of the Network adopts a Resnet101 deep convolutional neural Network to extract multi-scale information of an input image, outputs a Feature map containing effective Feature information of the image, and matches with a Feature Pyramid Network (hereinafter referred to as FPN) to solve the problem of difficulty in detecting small-sized objects in a target detection scene. The convolution operation is carried out on the images with different sizes, so that the characteristics of objects with different scales can be obtained, simple targets can be distinguished by utilizing shallow characteristics, and complex targets can be distinguished by utilizing deep characteristics. Firstly, sending the picture into a constructed Resnet101 network, reducing the picture step by step to generate pictures with different scales, then establishing a corresponding top-down network on the basis, and overlapping the up-sampled picture and the feature picture with the corresponding size.
The RPN network generation candidate block diagram is shown in fig. 6. And after the feature extraction is completed, inputting the feature map into the RPN network for detecting the candidate frame. In practical experiments, the length-width ratio is 1:2, 1:1 and 2:1, and the size is 32, 64, 128, 256 and 512, so that each small area generates 15 candidate boxes, and in the candidate boxes, all rectangular boxes with target objects are left through detection of the RPN. These rectangle boxes are divided into three categories, and the Intersection ratio (IoU) between the rectangle box and gt (group truth) is defined as the following expression:
when IoU for the rectangle box with GT is greater than 0.7, it is positive, when IoU for the rectangle box with GT is less than 0.3, it is negative, when IoU for the rectangle box with GT is between 0.3 and 0.7, it is neutral, the positive and negative rectangle boxes participate in the training, the neutral rectangle box is discarded and does not participate in the training. And under the condition that a plurality of front rectangular frames are overlapped, carrying out non-maximum suppression optimization, and leaving a rectangular frame with the highest score.
The method comprises the steps of mapping an interest region to a position corresponding to an input feature map according to an input image by using a rectangular frame and the feature map with different scales finally obtained through processing, dividing the mapped region into a plurality of small regions, obtaining the size of a divided pixel value by using a bilinear interpolation method, and keeping a floating point number during feature extraction in the dividing process. RoIAlign ensures that the number of the divided parts is the same as the final output dimension, and finally performs maximum pooling operation on each divided part to obtain the digital feature vector with fixed length. And inputting the extracted digital features into an interest region classifier to identify and classify the candidate frame containing the target object, further optimizing the position and the size of the frame according to the identified type of the target object, and finally outputting the candidate frame with the highest confidence coefficient as an output boundary frame. The network structure of the edge strengthening Mask RCNN is shown in figure 5.
FIG. 7 is a diagram illustrating generation of target feature detection edges by a region-generating branch network. The FCN is adopted in the area generation branch network part, and the coiled layer is replaced by the full connection layer to output the spatial distribution information. The FCN uses the deconvolution layer to upsample the feature map of the last convolution layer to restore it to the same size as the input image, thereby producing a prediction on each pixel while preserving spatial information in the original input image, and finally performs pixel-by-pixel classification on the upsampled feature map to output the edges of local features. The image segmentation range is narrowed down to the range of the bounding box by the previous network identification. In a smaller area, the target object is ensured to be a main object of the identification area, so that the calculation amount required in training and identification is reduced, and the identification accuracy is greatly improved.
The following results are shown in table 1, in which the accuracy of each feature type and the total accuracy are used as indexes to compare with the conventional mainstream target detection networks RCNN, Faster RCNN, and Mask RCNN. The edge-enhanced Mask RCNN is superior to the Mask RCNN in detection of different features and overall accuracy.
TABLE 1 accuracy of multiple target detection networks
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the invention and is not intended to limit the invention to the particular forms disclosed, and that modifications may be made, or equivalents may be substituted for elements thereof, while remaining within the scope of the claims that follow. All modifications, equivalents and the like which come within the spirit and principle of the invention are intended to be included within the scope of the invention.
Claims (8)
1. A real-time detection method for a fan digital twin characteristic based on edge enhancement image segmentation is characterized by comprising the following steps:
(1) selecting a plurality of fan pictures with surface characteristics of a certain specific fan, modeling the specific fan and the peripheral scene of the fan in a virtual environment based on the selected fan pictures, and increasing the surface characteristics of the fan;
(2) carrying out simulation shooting on a virtual fan model with surface characteristics by using a camera in a virtual scene to obtain a digital twin virtual fan picture with the surface characteristics;
(3) firstly, expanding and enhancing a data set of a real fan picture and a digital twin virtual fan picture, and marking out areas of all surface features displayed in the pictures and the category of each surface feature, wherein the categories comprise a vortex generator panel, vortex generator panel loss and corrosion;
(4) selecting Mask RCNN as a basic network for image segmentation, and enhancing the edge recognition effect of the Mask RCNN to obtain an edge-enhanced Mask RCNN; abstracting the areas of the surface features of the fans in all the fan pictures obtained in the step (3) and the categories of the surface features into a digital vector and a scalar respectively, wherein the digital vector is used as the input of the edge strengthening Mask RCNN, and the scalar is used as a comparison label for classifying the edge strengthening Mask RCNN; the edge strengthening Mask RCNN is a deep learning image segmentation neural network which increases the weight occupied by the boundary of a prediction region in a loss function during the Mask RCNN training;
(5) inputting the fan picture, the digital vector and the comparison label which are expanded and enhanced in the step (3) into the edge strengthening Mask RCNN for training to obtain an edge strengthening Mask RCNN weight parameter with optimized parameters;
(6) and inputting the picture to be tested into the edge strengthening Mask RCNN after parameter optimization, and outputting the category to which all the surface features on the picture belong and the area of each surface feature in real time.
2. The real-time detection method for the digital twin characteristics of the fan based on the edge-enhanced image segmentation as claimed in claim 1, wherein the edge-enhanced Mask RCNN is formed by sequentially connecting four parts of a deep convolution neural network, an RPN network, a RoIAlign network, a parallel full connection layer and an improved FCN network in series; the deep convolutional neural network is used for extracting information of different scales of an input image, the RPN network is used for generating areas of surface features to be detected with different sizes, the RoIAlign network is used for generating the areas of the surface features to be detected with different sizes into digital features with fixed length, the full connection layer is used for class detection, and the improved FCN network is used for area detection of the surface features.
3. The real-time detection method for the fan digital twin characteristics based on the edge-enhanced image segmentation as claimed in claim 2, wherein the Resnet101 is selected as the deep convolutional neural network.
4. The real-time detection method for the digital twin characteristics of the fan based on the edge-enhanced image segmentation as claimed in claim 1, wherein in the step (1), the construction of the virtual scene is based on the three-dimensional fusion of the solid photos, and the characteristics of the virtual fan and the fan are generated based on the real fan model and the existing characteristic image.
5. The real-time detection method for the fan digital twin characteristics based on the edge-enhanced image segmentation as claimed in claim 1, wherein in the step (2), the virtual camera automatically shoots by setting to automatically rotate around the fan body.
6. The real-time detection method for the fan digital twin features based on the edge-enhanced image segmentation of claim 1, wherein in the step (3), a pyramid and a patch enhancement image enhancement method are firstly adopted to expand sample pictures of a fan picture with surface features, then all the expanded pictures are further expanded through left-right turning, contrast normalization and Gaussian blur, and then the area of each surface feature on the picture and the category of each surface feature are labeled.
7. The real-time detection method for the fan digital twin feature based on the edge-enhanced image segmentation as claimed in claim 1, wherein in the step (3), the contour of each surface feature is labeled to represent the region where the surface feature is located, and the category to which each surface feature belongs is defined as an integer.
8. The real-time detection method for the fan digital twin features based on the edge-enhanced image segmentation is characterized in that a feature map output by a deep convolutional neural network is used as an input of an RPN network, a plurality of candidate frames with the length-to-width ratio of 1:2, 1:1, 2:1 and the sizes of 32, 64, 128, 256 and 512 are generated, and the candidate frames with the target object are reserved through detection of the RPN network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210680421.8A CN114782417A (en) | 2022-06-16 | 2022-06-16 | Real-time detection method for digital twin characteristics of fan based on edge enhanced image segmentation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210680421.8A CN114782417A (en) | 2022-06-16 | 2022-06-16 | Real-time detection method for digital twin characteristics of fan based on edge enhanced image segmentation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114782417A true CN114782417A (en) | 2022-07-22 |
Family
ID=82421342
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210680421.8A Pending CN114782417A (en) | 2022-06-16 | 2022-06-16 | Real-time detection method for digital twin characteristics of fan based on edge enhanced image segmentation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114782417A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115780306A (en) * | 2022-12-20 | 2023-03-14 | 广东工业大学 | High-speed multi-scale MLCC detection device and method |
| CN115841557A (en) * | 2023-02-23 | 2023-03-24 | 河南核工旭东电气有限公司 | Intelligent crane operation environment construction method based on digital twinning technology |
| CN116030050A (en) * | 2023-03-27 | 2023-04-28 | 浙江大学 | On-line detection and segmentation method for surface defects of fan based on unmanned aerial vehicle and deep learning |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112298287A (en) * | 2020-11-13 | 2021-02-02 | 重庆交通大学 | Method for preventing collision of high-speed train groups in cooperative formation under communication interruption condition |
| WO2021067665A2 (en) * | 2019-10-03 | 2021-04-08 | Photon-X, Inc. | Enhancing artificial intelligence routines using 3d data |
| US20210110262A1 (en) * | 2019-10-14 | 2021-04-15 | Honda Research Institute Europe Gmbh | Method and system for semi-supervised deep anomaly detection for large-scale industrial monitoring systems based on time-series data utilizing digital twin simulation data |
| CN112712098A (en) * | 2019-10-25 | 2021-04-27 | 北京四维图新科技股份有限公司 | Image data processing method and device |
| CN113011288A (en) * | 2021-03-02 | 2021-06-22 | 中北大学 | Mask RCNN algorithm-based remote sensing building detection method |
| CN113505655A (en) * | 2021-06-17 | 2021-10-15 | 电子科技大学 | Bearing fault intelligent diagnosis method for digital twin system |
| CN114048600A (en) * | 2021-11-09 | 2022-02-15 | 苏州纽克斯电源技术股份有限公司 | Digital twin-driven multi-model fusion industrial system anomaly detection method |
| WO2022044297A1 (en) * | 2020-08-28 | 2022-03-03 | 日本電信電話株式会社 | Information processing method, information processing device, and information processing program |
| CN114332006A (en) * | 2021-12-28 | 2022-04-12 | 武汉大学 | Automatic quantitative assessment method for urban battlement loss |
| CN114359658A (en) * | 2021-12-15 | 2022-04-15 | 深圳市优必选科技股份有限公司 | Training data generation method and device, terminal equipment and storage medium |
| CN114492511A (en) * | 2021-12-31 | 2022-05-13 | 中云开源数据技术（上海）有限公司 | Fault diagnosis method based on digital twinning |
| CN114511582A (en) * | 2021-12-28 | 2022-05-17 | 武汉大学 | Automatic ancient city battlement extraction method |
-
2022
- 2022-06-16 CN CN202210680421.8A patent/CN114782417A/en active Pending
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021067665A2 (en) * | 2019-10-03 | 2021-04-08 | Photon-X, Inc. | Enhancing artificial intelligence routines using 3d data |
| US20210110262A1 (en) * | 2019-10-14 | 2021-04-15 | Honda Research Institute Europe Gmbh | Method and system for semi-supervised deep anomaly detection for large-scale industrial monitoring systems based on time-series data utilizing digital twin simulation data |
| CN112712098A (en) * | 2019-10-25 | 2021-04-27 | 北京四维图新科技股份有限公司 | Image data processing method and device |
| WO2022044297A1 (en) * | 2020-08-28 | 2022-03-03 | 日本電信電話株式会社 | Information processing method, information processing device, and information processing program |
| CN112298287A (en) * | 2020-11-13 | 2021-02-02 | 重庆交通大学 | Method for preventing collision of high-speed train groups in cooperative formation under communication interruption condition |
| CN113011288A (en) * | 2021-03-02 | 2021-06-22 | 中北大学 | Mask RCNN algorithm-based remote sensing building detection method |
| CN113505655A (en) * | 2021-06-17 | 2021-10-15 | 电子科技大学 | Bearing fault intelligent diagnosis method for digital twin system |
| CN114048600A (en) * | 2021-11-09 | 2022-02-15 | 苏州纽克斯电源技术股份有限公司 | Digital twin-driven multi-model fusion industrial system anomaly detection method |
| CN114359658A (en) * | 2021-12-15 | 2022-04-15 | 深圳市优必选科技股份有限公司 | Training data generation method and device, terminal equipment and storage medium |
| CN114332006A (en) * | 2021-12-28 | 2022-04-12 | 武汉大学 | Automatic quantitative assessment method for urban battlement loss |
| CN114511582A (en) * | 2021-12-28 | 2022-05-17 | 武汉大学 | Automatic ancient city battlement extraction method |
| CN114492511A (en) * | 2021-12-31 | 2022-05-13 | 中云开源数据技术（上海）有限公司 | Fault diagnosis method based on digital twinning |
Non-Patent Citations (1)
| Title |
|---|
| 金侠挺 等，: "基于贝叶斯CNN和注意力网络的钢轨表面缺陷检测系统", 《自动化学报》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115780306A (en) * | 2022-12-20 | 2023-03-14 | 广东工业大学 | High-speed multi-scale MLCC detection device and method |
| CN115841557A (en) * | 2023-02-23 | 2023-03-24 | 河南核工旭东电气有限公司 | Intelligent crane operation environment construction method based on digital twinning technology |
| CN116030050A (en) * | 2023-03-27 | 2023-04-28 | 浙江大学 | On-line detection and segmentation method for surface defects of fan based on unmanned aerial vehicle and deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| He et al. | Towards fast and accurate real-world depth super-resolution: Benchmark dataset and baseline | |
| Zhang et al. | A review of deep learning-based semantic segmentation for point cloud | |
| Su et al. | Splatnet: Sparse lattice networks for point cloud processing | |
| CN108416266B (en) | Method for rapidly identifying video behaviors by extracting moving object through optical flow | |
| Han et al. | Livestock detection in aerial images using a fully convolutional network | |
| CN114782417A (en) | Real-time detection method for digital twin characteristics of fan based on edge enhanced image segmentation | |
| CN111179217A (en) | Attention mechanism-based remote sensing image multi-scale target detection method | |
| CN111898507A (en) | Deep learning method for predicting earth surface coverage category of label-free remote sensing image | |
| CN113298936B (en) | Multi-RGB-D full-face material recovery method based on deep learning | |
| Geng et al. | Using deep learning in infrared images to enable human gesture recognition for autonomous vehicles | |
| CN112232134B (en) | Human body posture estimation method based on hourglass network and attention mechanism | |
| CN113408584B (en) | RGB-D multi-modal feature fusion 3D target detection method | |
| Ouyang et al. | Vehicle target detection in complex scenes based on YOLOv3 algorithm | |
| CN113205502A (en) | Insulator defect detection method and system based on deep learning | |
| CN112561926A (en) | Three-dimensional image segmentation method, system, storage medium and electronic device | |
| CN112329771B (en) | Deep learning-based building material sample identification method | |
| CN117636131A (en) | Yolo-I model-based small target identification method and related device | |
| WO2021096190A1 (en) | Method for synthesizing 2d image of scene as viewed from desired viewpoint and electronic computing device implementing the same | |
| Lu et al. | Edge-reinforced convolutional neural network for road detection in very-high-resolution remote sensing imagery | |
| CN118410724A (en) | Transmission line foreign matter identification method, system, computer equipment and medium | |
| CN117746130A (en) | Weak supervision deep learning classification method based on remote sensing image punctiform semantic tags | |
| CN115630660B (en) | Barcode positioning method and device based on convolutional neural network | |
| CN113657375B (en) | Bottled object text detection method based on 3D point cloud | |
| CN117011648A (en) | Haptic image dataset expansion method and device based on single real sample | |
| CN116758219A (en) | Region-aware multi-view stereo matching three-dimensional reconstruction method based on neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220722 |