CN117456356A - A video recognition and early warning method for urban waterlogging based on deep learning - Google Patents

Abstract

The invention discloses a deep learning-based urban waterlogging video identification and early warning method, which includes: collecting urban waterlogging picture data and generating label data in json format; processing the json format data into a binary image format; and processing the water logging image data. Including pre-processing work such as rotation, scaling, color gamut transformation, Gaussian blur, etc.; input the pre-processed image data and corresponding label data into the DeepLabV3+ deep learning model for training to obtain the best model training weight file; input urban waterlogging video data The model is used to identify frame by frame, and the video data after model identification is used to calculate the proportion of water pixels in the entire image pixels frame by frame, which is used to represent the dynamic range change process of urban waterlogging in a long-term sequence. The invention can use monitoring facilities widely distributed in the city as monitoring media for urban waterlogging to identify water accumulation conditions in real time, and can reflect dynamic changes in urban waterlogging.

Description

A video recognition and early warning method for urban waterlogging based on deep learning

Technical field

The invention relates to a deep learning-based urban waterlogging video recognition and early warning method, and belongs to the technical field of urban waterlogging monitoring and early warning.

Background technique

At present, the traditional manual monitoring method is inefficient and dangerous; the automatic station monitoring method is high cost, difficult to maintain, and the automatic station distribution is small and difficult to take into account the overall situation; the remote sensing monitoring method is affected by the satellite return cycle and it is difficult to monitor waterlogging in real time, and Optical satellites have difficulty penetrating clouds, and SAR satellites are affected by the complex urban environment and cannot effectively extract waterlogged areas. Therefore, efficient and safe monitoring of urban waterlogging is of great significance to reduce the risks caused by waterlogging and reduce the loss of life, property and economic losses caused by waterlogging.

Contents of the invention

The technical problem to be solved by this invention is to provide a deep learning-based urban waterlogging video recognition and early warning method, using an image segmentation model to predict water accumulation areas and perform early warning based on the dynamic changes of water accumulation areas, thus solving the existing high cost of waterlogging monitoring. , there are too few monitoring sites and it is difficult to take into account the overall situation.

The present invention adopts the following technical solutions to solve the above technical problems:

A deep learning-based urban waterlogging video recognition and early warning method includes the following steps:

Step 1: Obtain the urban waterlogging image data set, label the water accumulation area in each original water accumulation image in the data set, and save the labeled water accumulation image in json format;

Step 2: Binarize the annotated water accumulation image in json format to obtain a binary image that corresponds one-to-one to the original water accumulation image;

Step 3: Perform data enhancement on the original waterlogging images in the urban waterlogging and waterlogging image data set obtained in step 1 to obtain an enhanced data set. The enhanced data set and corresponding binary images are divided into training sets and verification sets. set and test set;

Step 4: Construct the DeepLabV3+ image segmentation model, use the training set and verification set to train and verify the DeepLabV3+ image segmentation model, and obtain the trained DeepLabV3+ image segmentation model;

Step 5: Use the trained DeepLabV3+ image segmentation model to identify the water area in the test set image, and calculate the ratio of the number of pixels in the water area in the image to the number of pixels in the entire image, that is, the water pixel ratio. According to the water pixel ratio Characterize the dynamic changes of the water accumulation area and provide an early warning when the water pixel ratio exceeds the preset threshold.

As a preferred solution of the present invention, in step 1, the water accumulation area in the original water accumulation image is framed using irregular polygons using a visual interpretation method to obtain an annotated water accumulation image.

As a preferred solution of the present invention, in step 2, the pixel value of the water accumulation area in the marked water accumulation image is assigned a value of 1, and the pixel values of the remaining parts are assigned a value of 0, thereby obtaining a binary image.

As a preferred solution of the present invention, in step 3, the specific operations of data enhancement are as follows:

1) Randomly select a number a between 0 and 1. If a is between 0 and 0.5, perform data enhancement operation, otherwise no data enhancement operation will be performed;

2) Randomly select a number b between 0 and 1. If b is between 0 and 0.25, randomly scale the length and width of the original water image. The scaling factor is randomly selected between 0.25 and 2, and the original product is The binary image corresponding to the water image is scaled at the same magnification;

3) Randomly select a number c between 0 and 1. If c is between 0.25 and 0.5, randomly flip the original water image. The flip angle is randomly selected between 0 and 360 degrees, and the original water image corresponding to The binary image is flipped at the same angle;

4) Randomly select a number d between 0 and 1. If d is between 0.5 and 0.75, perform Gaussian blur on the original water image, and set the blur kernel size to 5×5;

5) Randomly select a number e between 0 and 1. If e is between 0.75 and 1, perform color gamut transformation on the original water image, convert the original water image into HSV color space, and perform hue, saturation, Random transformation of brightness.

As a preferred solution of the present invention, in step 4, the DeepLabV3+ image segmentation model includes two parts: encoding and decoding; wherein, the encoding part includes a four-times downsampling backbone feature extraction network and a dilated spatial convolution module and a 1 An enhanced feature extraction network composed of ×1 convolutional layers; the backbone feature extraction network includes the first to fourth downsampling modules connected in sequence and a 1×1 convolutional layer. The first downsampling module includes a 3x1 convolutional layer connected in sequence. ×3 convolutional layer and two 3×3 bottleneck structures. The second downsampling module includes two 3×3 bottleneck structures connected in sequence. The third downsampling module includes three 3×3 bottlenecks connected in sequence. structure, the fourth downsampling module includes seven 3×3 bottleneck structures connected in sequence; the atrous spatial convolution module includes five parallel layers, which are one 1×1 convolution layer and three 3×3 expansion layers. Parallel atrous convolutional layers with rates of 6, 12 and 18 and a full pooling layer; the decoding part includes a 1×1 convolutional layer, a 3×3 convolutional layer and the first and second quadruple sampling module;

The enhanced water accumulation image and the corresponding binary image are used as the input of the DeepLabV3+ image segmentation model, and the backbone feature extraction network is used to generate a feature tensor with a size of 1/16 of the original image and a convolution with a size of 1/4 of the original image. Feature layer; the feature tensor is passed into the hole space convolution module, and the feature tensor is processed in parallel by five parallel layers and then spliced. The spliced layer is subjected to 1×1 convolution processing to obtain enhanced features. The output of the extraction network is quadrupled by the first quadruple upsampling module of the decoding part to quadruple the output of the enhanced feature extraction network; the convolutional feature layer generated by the backbone feature extraction network is processed by 1×1 convolution and The layers output by the first quadruple upsampling module are channel spliced, and the splicing results are sequentially subjected to 3×3 convolution processing and the second quadruple upsampling module to obtain the output of the DeepLabV3+ image segmentation model.

A computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the deep learning-based method as described above is implemented. Steps of urban waterlogging video identification and early warning method.

A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the deep learning-based urban flooding video recognition and early warning method are implemented as described above.

Compared with the existing technology, the present invention adopts the above technical solution and has the following technical effects:

1. The method proposed by the present invention can monitor waterlogging without traditional monitoring sites. Monitoring facilities widely distributed in the city can be used as media for urban waterlogging monitoring, and can provide massive training data for the model; the model can be continuously updated and optimized. , to improve the recognition accuracy of waterlogging; it can characterize and calculate the changes in the dynamic range of waterlogging in long-term video sequences by calculating the water pixel ratio, and use this as a reference to issue urban waterlogging early warning information.

2. The present invention breaks through the limitations of traditional water accumulation monitoring sites such as low distribution and high cost. It can use monitoring facilities widely distributed in the city as a monitoring medium for urban waterlogging to identify water accumulation conditions in real time, and can reflect the occurrence of urban waterlogging. Dynamic changes provide technical support for more efficient, safe and timely monitoring and early warning of waterlogging.

Description of the drawings

Figure 1 is a flow chart of a deep learning-based urban waterlogging video recognition and early warning method of the present invention;

Figure 2 is a schematic diagram of expected binarization of water accumulation, in which (a) is the original image and (b) is the binarized image;

Figure 3 is a structural diagram of the backbone feature extraction network in the image segmentation model;

Figure 4 is a structural diagram of the image segmentation model;

Figure 5 is a rendering of video waterlogging at different stages identified using the image segmentation model;

Figure 6 is a graph showing the change in water pixel ratio in the waterlogging video.

Detailed ways

Embodiments of the invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and cannot be construed as limitations of the present invention.

As shown in Figure 1, it is a flow chart of a deep learning-based urban waterlogging video identification and early warning method proposed by the present invention, which includes the following steps:

Step 1: Create an urban waterlogging image data set and generate a json format file.

Collect image data that can reflect urban waterlogging conditions in complex scenarios, and use image annotation tools to draw polygons. The coordinates of the vertices in the lower left corner of the image are (0,0), and the coordinates of each vertex of the drawn polygon are {(x1,y1), (x2,y2)...(xk,yk)}. k means that a certain polygon has k vertices. The area surrounded by the lines connecting these k vertices is the water accumulation area, and a json file is used to save the vertex coordinate information of each polygon.

Step 2: Convert the json file into a binary image; the image pixels in the water area surrounded by polygons in the json file are assigned a value of 1, and the remaining image pixels are assigned a value of 0. Each binary image corresponds to an original image. The schematic diagram of the expected binarization of water accumulation is shown in (a) and (b) of Figure 2.

Step 3: Preprocess the original urban waterlogging image data and split the data into a training set, a verification set and a test set.

Step 3.1: Perform data enhancement on the original image data, including random scaling, random flipping, Gaussian blur, and color gamut transformation. Random scaling and random flipping change the position and range of waterlogging on the image, so the same scaling and flipping operations need to be performed on the binary image corresponding to the scaled and flipped original image. Gaussian blur and color gamut transformation do not change the location and range information of waterlogging in the original image, and the binary image corresponding to this part of the original image does not require any transformation operation. The enhanced image has a one-to-one correspondence with the binary image. The specific data enhancement operations are as follows:

(1) Randomly select a number between 0 and 1. If it is between 0 and 0.5, the enhancement operation will be performed. Otherwise, the enhancement operation will not be performed.

(2) Randomly select a number between 0 and 1. If it is between 0 and 0.25, the length and width of the image will be scaled. The scaling factor will be randomly selected between 0.25 and 2, and the corresponding binary image will be Perform zoom operations at the same magnification.

(3) Randomly select a number between 0 and 1. If it is between 0.25 and 0.5, perform image rotation. The rotation angle is randomly selected between 0 and 360 degrees, and the corresponding binary image is rotated at the same angle. Rotation operation.

(4) Randomly select a number between 0 and 1. If it is between 0.5 and 0.75, perform Gaussian blur on the image. Set the blur kernel size to 5×5, which is the pixel value of the central area of a square composed of 25 pixels. Set to the average of the surrounding 24 pixel values, and the corresponding binary image is not operated.

(5) Randomly select a number between 0 and 1. If it is between 0.75 and 1, perform image color gamut transformation, convert the image to HSV color space, and perform random transformations on hue, saturation, and brightness. The corresponding No operation is performed on the binarized image.

Step 3.2: Randomly divide the above-mentioned enhanced images and their corresponding binary images into training sets, verification sets and test sets in a ratio of 8.5:1:0.5.

Step 4: Train the image data of the training set based on the DeeplabV3+ image segmentation algorithm, and debug the model parameters using the image data of the verification set. Specifically follow the following steps to implement:

Step 4.1: Build the DeepLabV3+ image segmentation model. The overall model consists of two parts: encoding and decoding. The encoding part consists of four times down-sampled backbone feature extraction network (DCNN) and enhanced feature extraction composed of atrous spatial convolution module (ASPP). Network composition; the decoding part is the splicing and upsampling processing of shallow features extracted by the backbone feature extraction network and features extracted by the enhanced feature extraction network. The structure of the backbone feature extraction network is shown in Figure 3. The structure of the model is shown in Figure 4.

Download the weight file of DeepLabV3+ for the VOC data set from the GitHub website as the pre-training weight for training water images.

Step 4.2: Adjust the downsampling multiple, batch size (batch_size), number of iterations (epoch) and other parameters required for model training to train the model. The training steps are as follows:

(1) Input the images of the training set into the model and perform encoding processing. After a series of convolution operations, a feature tensor with a size of 1/16 of the original image is generated.

(2) Pass these feature tensors into the ASPP structure, which consists of a 1×1 convolution layer, three 3×3 parallel atrous convolution layers with expansion rates of 6, 12 and 18 respectively, and a full The pooling layer processes the above feature tensors. These layers are channel spliced, and then the spliced layers are subjected to 1×1 convolution.

(3) Extract the convolutional feature layer generated by the backbone network with a size of 1/4 of the original image, upsample the ASPP result layer four times to form a layer of the same size for channel splicing, and then stitch the spliced image The layer performs 3×3 convolution processing and four times upsampling processing to form an urban flooding prediction effect map with the same size as the original image.

Step 5: Input the test set into the model trained in step 4, so that the model can identify the water areas in the image frame by frame. Construct the water pixel ratio parameter and calculate the ratio of the number of identified water pixels to the number of pixels in the entire image to represent the dynamic changes in the water range in the monitored area. If the ratio is greater than a certain threshold, an early warning will be issued. Specifically follow the following steps to implement:

Step 5.1: Test the test set based on the best model weight file generated after the training in step 4.2, superimpose the identified waterlogged area and the original image, and present it. Figure 5 is a rendering of video waterlogging at different stages.

Step 5.2: Calculate the proportion of the water pixels _Flooded in the image identified in step 5.1 to the total image pixels _Total , and construct the water pixel ratio parameter (Water Pixel Ratio, WPR), whose expression is:

This parameter can be understood as the range of water accumulation in the visible area captured by the stationary monitoring facility, and its value varies between 0% and 100%. When the view of the waterlogged area is blocked by objects or people, the WPR values recorded at consecutive times will fluctuate significantly, but this fluctuation will not affect the changing trend of WPR, that is, when the WPR is continuously calculated in a long-term series time, it can characterize the changing trend of urban waterlogging extent. When this parameter is greater than a certain threshold, it indicates that waterlogging is serious and early warning should be carried out.

Taking a video of water accumulation as an example, the present invention identifies the waterlogging area in the video frame by frame, and selects 263 frames of the video through the frame-taking method at equal time intervals, and draws the real water pixel ratio change curve and the The change curve of the water pixel ratio predicted by the actual model was fitted to verify the feasibility of the model, as shown in Figure 6.

Based on the same inventive concept, embodiments of the present application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned deep learning-based Steps of urban waterlogging video identification and early warning method.

Based on the same inventive concept, embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the aforementioned deep learning-based urban waterlogging video recognition and early warning method is implemented. A step of.

Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device for realizing the functions specified in one process or multiple processes of the flowchart and/or one block or multiple blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in a process or processes of the flowchart and/or a block or blocks of the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.

The above embodiments are only for illustrating the technical ideas of the present invention and cannot limit the protection scope of the present invention. Any changes made based on the technical solutions based on the technical ideas proposed by the present invention will fall within the protection scope of the present invention. Inside.

Claims (7)

1. A city waterlogging video recognition early warning method based on deep learning is characterized by comprising the following steps:

step 1, acquiring a city waterlogged ponding image dataset, marking ponding areas in each original ponding image in the dataset, and storing the marked ponding images in json format;

step 2, binarizing the water accumulation image marked in json format to obtain a binarized image corresponding to the original water accumulation image one by one;

step 3, carrying out data enhancement on the original ponding image in the urban waterlogging ponding image data set obtained in the step 1 to obtain an enhanced data set, and dividing the enhanced data set and the corresponding binarized image into a training set, a verification set and a test set;

step 4, constructing a deep LabV3+ image segmentation model, and training and verifying the deep LabV3+ image segmentation model by using a training set and a verification set to obtain a trained deep LabV3+ image segmentation model;

and 5, identifying the ponding area of the image in the test set by using the trained deep LabV3+ image segmentation model, calculating the proportion of the number of pixels of the ponding area in the image to the number of pixels of the whole image, namely the ponding pixel ratio, representing the dynamic change condition of the ponding area according to the ponding pixel ratio, and carrying out early warning when the ponding pixel ratio exceeds a preset threshold value.

2. The method for identifying and pre-warning urban waterlogging video based on deep learning according to claim 1, wherein in the step 1, a method of visual interpretation is utilized to frame and select a ponding area in an original ponding image by using an irregular polygon, so as to obtain a marked ponding image.

3. The urban waterlogging video recognition and early warning method based on deep learning according to claim 1, wherein in the step 2, the pixel value of the ponding area in the marked ponding image is assigned 1, and the pixel value of the rest part is assigned 0, so as to obtain a binarized image.

4. The urban waterlogging video recognition early warning method based on deep learning according to claim 1, wherein in the step 3, the specific operation of data enhancement is as follows:

1) Randomly selecting a number a from 0 to 1, if a is between 0 and 0.5, performing data enhancement operation, otherwise, not performing data enhancement operation;

2) Randomly selecting a number b from 0 to 1, randomly scaling the length and the width of the original ponding image if b is between 0 and 0.25, randomly selecting the scaling factor from 0.25 to 2, and performing scaling operation of the same scaling factor on the binarized image corresponding to the original ponding image;

3) Randomly selecting a number c from 0 to 1, if c is between 0.25 and 0.5, randomly overturning the original ponding image, randomly selecting an overturning angle from 0 to 360 degrees, and carrying out overturning operation of the same angle on the binarized image corresponding to the original ponding image;

4) Randomly selecting a number d from 0 to 1, and if d is between 0.5 and 0.75, carrying out Gaussian blur on an original ponding image, wherein the size of a blur kernel is set to be 5 multiplied by 5;

5) And randomly selecting a number e from 0 to 1, if the number e is between 0.75 and 1, performing color gamut conversion on the original ponding image, converting the original ponding image into an HSV color space, and performing random conversion on hue, saturation and brightness.

5. The urban waterlogging video recognition early warning method based on deep learning according to claim 1, wherein in the step 4, a deep labv3+ image segmentation model comprises an encoding part and a decoding part; the coding part comprises a trunk feature extraction network for four times of downsampling and a reinforced feature extraction network formed by a cavity space convolution module and a 1 multiplied by 1 convolution layer; the trunk feature extraction network comprises first to fourth downsampling modules and a 1×1 convolution layer which are sequentially connected, wherein the first downsampling module comprises a 3×3 convolution layer and two 3×3 bottleneck structures which are sequentially connected, the second downsampling module comprises two 3×3 bottleneck structures which are sequentially connected, the third downsampling module comprises three 3×3 bottleneck structures which are sequentially connected, and the fourth downsampling module comprises seven 3×3 bottleneck structures which are sequentially connected; the cavity space convolution module comprises five parallel layers, namely a 1 multiplied by 1 convolution layer, three 3 multiplied by 3 parallel cavity convolution layers with expansion rates of 6, 12 and 18 respectively and a full pooling layer; the decoding part comprises a 1 x1 convolution layer, a 3 x 3 convolution layer and first and second quadruple up-sampling modules;

taking the enhanced ponding image and the corresponding binarized image as input of a deep LabV < 3+ > image segmentation model, and generating a characteristic tensor with the size of 1/16 of the original image and a convolution characteristic image layer with the size of 1/4 of the original image through a trunk characteristic extraction network; in the feature tensor input hole space convolution module, five parallel layers are used for carrying out parallel processing on the feature tensor and then splicing, a 1 multiplied by 1 convolution processing is carried out on the spliced image layer, so that the output of the reinforced feature extraction network is obtained, and the first four-time up-sampling module of the decoding part is used for carrying out four-time up-sampling on the output of the reinforced feature extraction network; and carrying out channel splicing on the convolution characteristic layer generated by the trunk characteristic extraction network and the layer output by the first four-time up-sampling module after carrying out convolution processing on the convolution characteristic layer by 1 multiplied by 1, and obtaining the output of the deep LabV3+ image segmentation model after the splicing result sequentially passes through convolution processing of 3 multiplied by 3 and the second four-time up-sampling module.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor, when executing the computer program, implements the steps of the deep learning based urban waterlogging video recognition pre-warning method of any one of claims 1 to 5.

7. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the steps of the deep learning based urban waterlogging video recognition pre-warning method of any one of claims 1 to 5.