CN117132914B - General power equipment identification large model method and system - Google Patents

Abstract

The present invention proposes a general electric equipment identification large model method and system, taking the oblique images collected by the drone inspection system as the research object. According to the characteristics of its data, a single-stage object detector is trained to identify the object bounding box in the image as the initial prompt information, and the intermediate layer features of the image encoder are used to generate prompts containing semantic category information. By fusing two kinds of prompt information, a category-related universal power equipment segmentation model is formed. This method better solves the problem of large data and few identification types required for training models in power scenarios, and provides basic data for subsequent power equipment defect diagnosis and three-dimensional modeling.

Description

General power equipment identification large model method and system

Technical field

The invention belongs to the application of automatic identification of power equipment in inspection images during UAV power inspection, and proposes a general large model for identification of power equipment.

Background technique

As the mileage of transmission lines increases on a large scale year after year, it poses a major challenge to ensuring the safe and stable operation of transmission lines. With the explosive development of drones, computer vision, and embedded technologies, the power inspection operation mode is gradually changing from traditional manual methods to precise drone inspections. During the inspection process of the drone, the sensor pod mounted on the aircraft collects oblique images along the power corridor, locates and identifies power equipment through the target detection algorithm deployed on the aircraft or in the background, and diagnoses hidden faults. The new power inspection method of "drone + visual recognition system" is gradually becoming the mainstream inspection operation mode because of its low cost and high efficiency.

In the field of computer vision, SAM technology achieves zero-sample high-performance detection and achieves excellent segmentation performance in general scenarios. In the field of power inspection, due to the complexity of UAV images and the large differences in target sizes, the existing detection algorithms have low versatility, limited parameters, and poor generalization capabilities, making it difficult to realize a general large model of power equipment and implement it in actual power transmission. Performance on the line still lacks robustness. Based on this, in the field of drone power inspection, the use of SAM technology can effectively improve detection performance. However, SAM is a category-independent instance segmentation method that relies heavily on a priori manual cues, including points, boxes, and rough masks. These limitations make SAM unsuitable for fully automatic interpretation of power inspection images.

Contents of the invention

Aiming at the problems of poor versatility and few recognition categories of existing algorithms in power scenarios, the present invention takes inspection data processing in power scenarios as the research object and designs a general large model method and system for power equipment identification with wide application scenarios and complete recognition categories.

The special feature of the general power equipment identification large model method designed by this invention is that it includes the following steps:

Step 1: Collect power inspection image data and construct a transmission line data set;

Step 2, train a single-stage object detector to detect object bounding boxes from the dataset images as explicit cues;

Step 3: Use the image encoder in the large model to process the image data in step 1, and then use the prompter of the large model to generate an implicit prompt containing semantic category information; fuse the two forms of prompts in steps 2 and 3, and the fused The semantic category information is passed into the large model to obtain general electric power recognition results;

Among them, the fusion method is to align the explicit prompts generated by the single-stage detector with the implicit prompt features generated by the middle layer of the large model, and fuse the positions of the two prompts by calculating the mapping relationship between the explicit prompts and the implicit prompt feature maps. Information and category information.

A preferred way is to implement step 1 as follows:

Step 1.1: First, screen and clean the collected inspection images of various scenes;

Step 1.2, use labelImg to label inspection images. The transmission line scenes include cities, greenhouses, farmland, bushes, wasteland, lakes, etc. The types of power equipment and external intruders cover transmission towers, insulators, voltage equalizing rings, anti-vibration hammers, and intervals. Rods, insulator bursting discs, floating objects;

Step 1.3, input the processed oblique image to the single-stage object detector.

Further, the single-stage object detector uses YOLOv8.

A preferred way is that the specific process of step 2 is as follows:

Step 2.1, the original image undergoes scale change and filling;

Step 2.2: After data enhancement and preprocessing, the image processed in step 2.1 is input into the single-stage detector backbone network;

Step 2.3: Perform multi-scale feature fusion on the features extracted by the backbone network;

Step 2.4, the fused features are input to the single-stage target detector to obtain the target category and rough detection frame contained in the image.

Further, data enhancement and preprocessing in step 2.2 include: horizontal and vertical flipping, contrast adjustment, rotation, mosaic enhancement, adaptive anchor frame calculation and adaptive grayscale filling.

A preferred way is to use the SAM large model in step 3. The specific process is as follows:

Step 3.1, input the original image, and generate an intermediate feature map after passing through the pre-trained VIT backbone network;

Step 3.2: Input the intermediate features obtained through the ViT backbone network in the previous step into the lightweight feature aggregation module to obtain the fused semantic features;

Step 3.3, after obtaining the fused semantic features, use the hinter to generate implicit hint embeddings for the SAM mask decoder;

Step 3.4, align the display cues generated by the single-stage detector with the implicit cues features generated by the SAM intermediate layer, and then perform cue fusion to extract rich semantic information.

Based on the same inventive concept, this solution also designs a system that implements a large model method for universal power equipment identification:

It includes a data acquisition module to collect and obtain power inspection image data and build a transmission line data set;

A single-stage object detector module that uses detected object bounding boxes as explicit cues;

The general electric power recognition module uses the middle layer features of the image encoder in the large model to form the input of the prompter, and generates an implicit prompt containing semantic category information; it fuses the display prompt and the implicit prompt, and passes in the fused semantic category information. In large models, general power identification results are obtained;

Among them, the fusion method is to align the explicit prompts generated by the single-stage detector with the implicit prompt features generated by the SAM intermediate layer, and fuse the position information of the two prompts by calculating the mapping relationship between the explicit prompts and the implicit prompt feature maps. and category information.

Based on the same inventive concept, this solution also designs an electronic device, including:

one or more processors;

A storage device for storing one or more programs;

When one or more programs are executed by the one or more processors, the one or more processors implement a general power equipment identification large model method.

Based on the same inventive concept, this solution also designs a computer-readable medium on which a computer program is stored, which is characterized in that: when the program is executed by the processor, a universal power equipment identification large model method is implemented.

Compared with the existing technology, the present invention has the following advantages and beneficial effects:

The identified types of power equipment and application scenarios have been greatly expanded, which can meet the automatic processing of inspection data in multiple time periods and multiple types of scenarios, such as cities, farmland, lakes, forests, grasslands, wastelands, etc.; the overall detection accuracy has been significantly improved, ensuring higher Recall rate; strong application robustness and high generalization ability in actual transmission line scenarios.

The oblique images collected by the UAV inspection system are used as the research object. In view of its data characteristics, a single-stage object detector is used to identify the target bounding box in the image as the initial prompt information, and the intermediate layer features of the image encoder are used to generate prompts containing semantic category information. By fusing two kinds of prompt information, a category-related universal power equipment segmentation model is completed. This method can better solve the problem of low identification categories of massive inspection data in power scenarios, and provide basic data for subsequent power equipment defect diagnosis and three-dimensional modeling.

Description of the drawings

Figure 1 is a flow chart in an embodiment of the present invention.

Figure 2 is a structural diagram of a single-stage detector in an embodiment of the present invention.

Figure 3 is a specific structural diagram of a single-stage detector in an embodiment of the present invention.

Figure 4 is a structural diagram of the decoder in the embodiment of the present invention.

Detailed ways

The technical solution of the present invention will be described below with reference to the drawings and examples.

Embodiment 1

This invention designs a method for identifying large models of general electric equipment. The inspection images collected by the UAV inspection system are selected to illustrate the method proposed by this invention in detail. The process of this method is shown in Figure 1 and includes the following steps:

Step 1: Collect power inspection image data and construct a transmission line data set;

Step 2: Train a single-stage object detector and use the detected image data object bounding box as an explicit prompt.

Step 3: The intermediate layer features of the image encoder are used to form the input of the prompter, and a prompt containing semantic category information is generated. The two forms of cues are fused to pass semantic category information into SAM. Thus, universal power identification results are obtained. Preferably, the large model uses SAM, and other image segmentation models are also possible. In this embodiment, the SAM model is optimal.

Further, the specific implementation of step 1 includes the following sub-steps:

Step 1.1: First, screen and clean the collected inspection images of various scenes.

Step 1.2, use labelImg to label inspection images. The transmission line scenarios include cities, greenhouses, farmland, bushes, wasteland, lakes, etc., and the types of power equipment and external intruders cover transmission poles and towers, insulators, voltage equalizing rings, anti-vibration hammers, spacer rods, insulator burst discs, floating objects, etc. .

Step 1.3, input the processed oblique image to the single-stage object detector YOLOv8.

Further, the specific implementation of step 2 includes the following sub-steps:

Step 2.1, the original image is scaled and filled to a scale of 640×640.

Step 2.2: Perform data enhancement and preprocessing on the scaled image. Enhancement methods include horizontal and vertical flipping, contrast adjustment, rotation, mosaic enhancement, etc. Preprocessing includes adaptive anchor box calculation and adaptive grayscale filling.

Step 2.3, the data-enhanced image is input into the single-stage detector backbone network. First, the original image passes through a convolutional CBS module with a convolution kernel size of 6×6 and a step size of 2, and then passes through a joint module composed of CBS and C2f with a convolution kernel size of 3×3 and a step size of 2. The CBS module consists of 1 2D convolution, 1 2D batch normalization and scaled exponential linear unit activation function. The C2f module learns residual features and adds more skip connections and additional separation operations to obtain richer gradient flow information while ensuring lightweight.

Step 2.4, the features extracted through the backbone network are integrated into the multi-scale feature pyramid PAN-FPN of C2f for multi-scale feature fusion, and the output feature map scales are three feature maps of 80×80, 40×40 and 20×20.

Step 2.5, the fused features are input to the YOLO detection head. YOLOv8 uses a decoupled detection head to separate the classification and detection heads. The loss calculation process mainly includes the positive and negative sample allocation strategy and loss calculation. YOLOv8 adopts the principle of task alignment allocation and selects positive samples based on the score-weighted results of classification and regression. The loss value calculation includes two branches: classification and regression. The classification branch still uses the two branches. Value cross entropy loss, and the regression branch uses Distribution Focal Loss and CIoU loss functions. Output the object categories and rough detection boxes contained in the image.

Further, step 3 uses the SAM large model, which includes an encoder, prompter, fusion module and decoder. Its specific implementation includes the following sub-steps:

Step 3.1: The original drone inspection image data is input to the encoder in SAM. After passing through the pre-trained VIT encoder backbone network, an intermediate feature map is generated. The pre-trained masked autoencoder of the VIT backbone network processes the original image into intermediate features. The original image is scaled to 1024 scale, and a convolution with a convolution kernel size of 16 and a step size of 16 is used to discretize the image into a 64×64×768 vector. The vector is sequentially flattened in the feature map width and channel dimensions before entering Multi-layer VIT backbone network, the output vector of the VIT backbone network is compressed through two layers of convolution. The feature dimension is 256, and the convolution kernel sizes of the two layers are 1 and 3 respectively.

Step 3.2: Input the intermediate features obtained through the ViT backbone network in the previous step into the lightweight feature aggregation module of the large model prompter to generate fused semantic features, and then use the mask decoder to generate implicit prompts. This module learns to represent semantic features from various intermediate feature layers of ViT without increasing the computational complexity of the prompter. The process is expressed by the following formula:

and/> Represents the characteristics and structure of the SAM backbone/> Generated downsampled features. First, use a 1×1 convolutional layer to reduce the channel from c to 1/16 of the original dimension, and then use a 3×3 convolutional layer with a stride of 2 to reduce the spatial dimension. /> Represents a size of 3×3 convolutional layer, /> Represents the final fused convolutional layer, including two 3×3 convolutional layers and one 1×1 convolutional layer to restore the original channel dimensions of the SAM mask decoder. Input the fused semantic features obtained in the previous step and use the hinter to generate hints for the SAM mask decoder. Candidate object boxes are first generated using an anchor-based region proposal network. Obtain the visual feature representation of a single object from the position-encoded feature map through RoI pooling. Three perceptual heads are extracted from visual features: semantic head/> , positioning head/> and prompt header/> . The semantic header determines a specific target category, and the localization head establishes matching criteria between the generated hints and target instance masks, i.e., localization-based greedy matching. The hint header generates the hint embedding required by the SAM mask decoder. Among them/> Represents lightweight RPN. /> The operation may cause subsequent cue generation to lose position information relative to the entire image, merging the position encoding (PE) into the original fused feature (Fagg). The process is represented by the following formula:

The losses of this model include the binary classification loss and localization loss of the RPN network, the classification loss of the semantic head, the regression loss of the localization head, and the segmentation loss of the frozen SAM mask decoder. The total loss can be expressed as:

Step 3.3: Align the explicit prompts generated by the single-stage detector with the implicit prompt features generated by the SAM intermediate layer. By calculating the mapping relationship between explicit prompts and implicit prompt feature maps, the location information and category information of the two prompts are integrated. , thereby providing more precise positioning and classification accuracy.

Step 3.4, the mask decoder integrates the two embeddings output by the image encoder and the cue encoder respectively, decodes the final segmentation mask, and uses the Transformer to learn the image embedding layer aligned with the cue and the embedding of the additional 4 Tokens. The four Token embeddings are the IoU Token embedding and the three segmentation result Token embeddings. The Token embedding is obtained through Transformer learning and passes through the final task header to obtain the target result.

The technical solutions and beneficial effects of the present invention will be further described below with reference to specific examples.

Using multiple oblique image data sets collected by drones and processed by the method of the present invention, more than 12 types of power scene components and more than 20 types of defects can be identified, the power equipment segmentation mIoU is 0.65, and the operating speed reaches 60 FPS. It shows that the present invention can achieve more types and higher precision general electric equipment segmentation results while ensuring processing efficiency.

Embodiment 2

Based on the same inventive concept, this plan also designs a general power equipment identification large model system, including a data acquisition module to collect and obtain power inspection image data and build a transmission line data set;

A single-stage object detector module that uses detected object bounding boxes as explicit cues;

The general electric power recognition module uses the image encoder in the large model to process the data to obtain the image data in the module, and then generates implicit prompts containing semantic category information through the prompter; fuses the display prompts and implicit prompts, and transmits the fused semantic category information. into a large model to obtain general power identification results;

Among them, the fusion method is to align the explicit prompts generated by the single-stage detector with the implicit prompt features generated by the SAM intermediate layer, and fuse the position information of the two prompts by calculating the mapping relationship between the explicit prompts and the implicit prompt feature maps. and category information.

Since the equipment introduced in Embodiment 2 of the present invention is a system used to implement a universal power equipment identification large model method in Embodiment 1 of the present invention, based on the method introduced in Embodiment 1 of the present invention, those skilled in the art can understand the electronic The specific structure and deformation of the equipment will not be described again here.

Embodiment 3

Based on the same inventive concept, the present invention also provides an electronic device, including one or more processors; a storage device for storing one or more programs; when one or more programs are processed by the one or more processors Execution causes the one or more processors to implement the method described in Embodiment 1.

Since the equipment introduced in Embodiment 3 of the present invention is an electronic device used to implement a method for identifying large models of general power equipment according to Embodiment 1 of the present invention, those skilled in the art can understand based on the method introduced in Embodiment 1 of the present invention. The specific structure and deformation of the electronic device will not be described again here. All electronic devices used in a method of the embodiments of the present invention fall within the scope of protection of the present invention.

Embodiment 4

Based on the same inventive concept, the present invention also provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method described in Embodiment 1 is implemented.

Since the equipment introduced in Embodiment 4 of the present invention is a computer-readable medium used to implement a method for identifying large models of general electric equipment in Embodiment 1 of the present invention, based on the method introduced in Embodiment 1 of the present invention, those skilled in the art can Understand the specific structure and deformation of the electronic device, so I will not go into details here. All electronic devices used in a method of the embodiments of the present invention fall within the scope of protection of the present invention.

The above content is a further detailed description of the present invention in combination with specific implementation modes. It cannot be concluded that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all of them should be regarded as belonging to the protection scope of the present invention.

Claims (10)

1. A method for identifying a large model of a universal power device, comprising the steps of:

step 1, acquiring power inspection image data and constructing a power transmission line data set;

step 2, training a single-stage target detector, and taking the detected image data target bounding box as an explicit prompt;

step 3, adopting a SAM large model, wherein the large model comprises a VIT encoder, a prompter, a fusion module and a decoder, processing the image data in the step 1 by using the VIT encoder in the large model, and generating an implicit prompt containing semantic category information by using the prompter of the large model; the fusion module fuses the prompts in the two forms in the steps 2 and 3, and transmits the fused semantic category information into the large model to obtain a universal power recognition result;

the fusion mode is that an explicit prompt generated by a single-stage target detector is aligned with an implicit prompt feature generated by a large model middle layer, and the position information and the category information of the two prompts are fused by calculating the mapping relation between the explicit prompt and the implicit prompt feature map.

2. The universal power device identification large model method of claim 1, wherein: the step 1 is specifically implemented as follows:

step 1.1, screening and cleaning acquired inspection images of various scenes;

step 1.2, labeling inspection images by using labelImg, wherein a transmission line scene comprises cities, greenhouses, farmlands, bushes, barren lands and lakes, and power equipment and external invaded object types cover transmission towers, insulators, equalizing rings, damper blocks, spacers, insulator burst sheets and hanging objects;

step 1.3, inputting the processed oblique image to a single-stage object detector.

3. The universal power device identification large model method of claim 2, wherein: the single-stage object detector employs YOLOv8.

4. The universal power device identification large model method of claim 1, wherein: the specific process of the step 2 is as follows:

step 2.1, performing scale change and filling on the original image;

step 2.2, the image processed in the step 2.1 is subjected to data enhancement and pretreatment and then is input into a backbone network of the single-stage target detector;

step 2.3, carrying out multi-scale feature fusion on the features extracted by the backbone network;

and 2.4, inputting the fused characteristics to a single-stage target detector, and acquiring the target category and the rough detection frame contained in the image.

5. The universal power device identification large model method of claim 4, wherein:

the data enhancement and preprocessing in step 2.2 comprises: horizontal and vertical overturn, contrast adjustment, rotation, mosaic enhancement, adaptive anchor frame calculation and adaptive gray filling.

6. The universal power device identification large model method of claim 1, wherein:

the specific implementation process of the SAM big model in the step 3 is as follows:

step 3.1, inputting original image data, and generating an intermediate feature map through a pre-training VIT encoder;

step 3.2, the lightweight feature aggregation module of the prompter generates fusion semantic features from the acquired intermediate feature map, and then the prompter is utilized to generate implicit prompt for the SAM mask decoder;

step 3.3, aligning the display prompt generated by the single-stage target detector with the implicit prompt feature generated by the SAM middle layer by the fusion module, and then carrying out prompt fusion to extract semantic information;

and 3.4, integrating two embeddings output by the VIT encoder and the prompter by the decoder, and decoding a final segmentation mask.

7. The universal power device identification large model method of claim 6, wherein: the original image input in step 3.1 is scaled to 1024 scales.

8. A system for implementing the universal power device identification large model method of claim 1, characterized in that:

the power transmission line data acquisition module acquires power inspection image data and constructs a power transmission line data set;

the single-stage target detector module takes the detected target boundary box as an explicit hint;

the universal power recognition module processes the image data in the data acquisition module by utilizing an image encoder in the large model, and generates an implicit prompt containing semantic category information by a large model prompter; fusing the display prompt and the implicit prompt, and transmitting the fused semantic category information into a large model to obtain a general electric power recognition result;

the fusion mode is that an explicit prompt generated by a single-stage target detector is aligned with an implicit prompt feature generated by a SAM middle layer, and the position information and the category information of the two prompts are fused by calculating the mapping relation between the explicit prompt and the implicit prompt feature map.

9. An electronic device, comprising:

one or more processors;

a storage means for storing one or more programs;

when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-7.

10. A computer readable medium having a computer program stored thereon, characterized by: the program, when executed by a processor, implements the method of any of claims 1-7.