CN117708625A - Dam monitoring historical data filling method under spent data background - Google Patents

Abstract

The invention provides a dam monitoring historical data filling method under a spent data background, which comprises the following steps: acquiring monitoring data, preprocessing the monitoring data, and associating the preprocessed data with a time sequence to form time sequence data with the same time unit; identifying a measuring point with a missing data after pretreatment as a target missing measuring point, and selecting a homologous measuring point by using a cross-line method; decomposing the time series data of the target missing measuring point and the homologous measuring point by using an empirical mode decomposition algorithm; calculating similar distances by taking each missing component as a distance starting point and the decomposed component as a distance end point, sequencing the obtained similar distances, and carrying out aggregation integration on the target measuring point component and the calculated distance component; inputting the cluster function set into a prediction model; the predicted component is superimposed on the eigenmode function component of the target missing station. The method is more suitable for filling the missing value of the dam of the medium and small-sized reservoir, has the characteristics of less front data, high prediction precision, wider applicability and the like.

Description

Dam monitoring historical data filling method under spent data background

Technical Field

The invention relates to the field of dam monitoring, in particular to a dam monitoring historical data filling method under a data-starvation background.

Background

The safety evaluation guideline of the reservoir dam indicates that the monitoring data should be timely integrated and analyzed, so that the integrity of the monitoring data is ensured, the shape of the dam is timely known through the monitoring data, and basic data is provided for the overall safety evaluation of the dam. The traditional dam missing data complement method depends on complete prepositive data and empirical functions, and has poor effect on small and medium-sized earth-rock dams with data missing. The deformation value is used as a comprehensive variable reflecting the safety state of the reservoir dam, is an important index for evaluating structural performance, and is also an important research part in the current dam monitoring field.

The operation state of the dam can be better known through monitoring and analyzing the deformation value of the dam body, but the premise of taking the deformation as a main object of prediction is that the deformation is taken as a monitoring dependent variable, and the preposed data and the monitoring data of the acknowledged interpretation components (such as water pressure, temperature, aging and the like) are required to be complete and effective, so that better prediction precision can be obtained.

However, at present, some old projects are not automatically monitored, and due to lack of standardization of manual recording and large subjectivity and uncertainty of manual reading counting, data are not recorded or wrongly recorded. At present, a method for filling the self-variable data loss is not known to be effective, and the missing part in the monitoring data cannot be accurately filled, so that effective data cannot be provided for subsequent monitoring of dam deformation.

Disclosure of Invention

In order to overcome the defects of the prior art, the invention aims to provide a dam monitoring historical data filling method under the background of spent data.

In order to achieve the above object, the present invention provides the following solutions:

a dam monitoring historical data filling method under a spent data background comprises the following steps:

acquiring monitoring data from the dam embedded monitoring equipment and available data sources;

preprocessing the collected monitoring data, and correlating the preprocessed data with a time sequence to form time sequence data with the same time unit;

identifying a measuring point with a missing data after pretreatment as a target missing measuring point, and selecting a homologous measuring point by using a cross-line method;

decomposing the time sequence data of the target missing measuring point and the homologous measuring point by using an empirical mode decomposition algorithm to respectively obtain a missing component and a decomposition component;

calculating similar distances by taking each missing component as a distance starting point and the decomposition component as a distance end point, sequencing the obtained similar distances, and aggregating and integrating the target measuring point component and the calculated distance component to obtain each clustering function set;

inputting the cluster function set into a prediction model to obtain each prediction component;

and superposing the predicted component on the eigenmode function component of the target missing measuring point.

Preferably, the monitoring data includes: osmometer water level data, GNSS displacement meter displacement data in all directions and deep displacement data.

Preferably, the prediction model is constructed based on a gating cycle unit GRU.

Preferably, after identifying a point where the preprocessed data has a defect as a target missing point and selecting a homologous point by using a cross-hair method, the method further comprises:

and drawing a data line graph to determine whether the homologous measuring points and the target missing measuring points have the same trend, if the difference value of the trends is smaller than a preset threshold value, determining the same group of data, and if the difference value of the trends is larger than the preset threshold value, replacing the homologous measuring points.

Preferably, the process of preprocessing the collected monitoring data includes: data cleansing, data flattening, data format conversion and data integration.

According to the specific embodiment provided by the invention, the invention discloses the following technical effects:

the invention provides a dam monitoring historical data filling method under a spent data background, which comprises the following steps: acquiring monitoring data from the dam embedded monitoring equipment and available data sources; preprocessing the collected monitoring data, and correlating the preprocessed data with a time sequence to form time sequence data with the same time unit; identifying a measuring point with a missing data after pretreatment as a target missing measuring point, and selecting a homologous measuring point by using a cross-line method; decomposing the time sequence data of the target missing measuring point and the homologous measuring point by using an empirical mode decomposition algorithm to respectively obtain a missing component and a decomposition component; calculating similar distances by taking each missing component as a distance starting point and the decomposition component as a distance end point, sequencing the obtained similar distances, and aggregating and integrating the target measuring point component and the calculated distance component to obtain each clustering function set; inputting the cluster function set into a prediction model to obtain each prediction component; and superposing the predicted component on the eigenmode function component of the target missing measuring point. The method is more suitable for filling the missing value of the dam of the medium and small-sized reservoir, has the characteristics of less front data, high prediction precision, wider applicability and the like.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings that are needed in the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a technical route provided by an embodiment of the present invention;

FIG. 3 is a cross-hair plot provided in an embodiment of the present invention;

FIG. 4 is an exploded data diagram of sample measurement points according to an embodiment of the present invention;

FIG. 5 is a graph of a distance of 100 time steps DTW before a sample measurement point according to an embodiment of the present invention;

FIG. 6 is a diagram of a pre-processed data line graph provided by an embodiment of the present invention;

fig. 7 is a comparison chart of prediction results and a prediction residual chart provided by an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The invention aims to provide a dam monitoring historical data filling method under a spent data background, which has the characteristics of being more suitable for filling of missing values of a dam of a medium-small reservoir, less front data, high prediction precision, wider applicability and the like.

In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.

Fig. 1 is a flowchart of a method provided by an embodiment of the present invention, and as shown in fig. 1, the present invention provides a method for filling dam monitoring historical data in a spent data background, including:

step 100: acquiring monitoring data from the dam embedded monitoring equipment and available data sources;

step 200: preprocessing the collected monitoring data, and correlating the preprocessed data with a time sequence to form time sequence data with the same time unit;

step 300: identifying a measuring point with a missing data after pretreatment as a target missing measuring point, and selecting a homologous measuring point by using a cross-line method;

step 500: decomposing the time sequence data of the target missing measuring point and the homologous measuring point by using an empirical mode decomposition algorithm to respectively obtain a missing component and a decomposition component;

step 600: calculating similar distances by taking each missing component as a distance starting point and the decomposition component as a distance end point, sequencing the obtained similar distances, and aggregating and integrating the target measuring point component and the calculated distance component to obtain each clustering function set;

step 700: inputting the cluster function set into a prediction model to obtain each prediction component;

step 800: and superposing the predicted component on the eigenmode function component of the target missing measuring point.

As shown in fig. 2, the technical flow of the present embodiment includes:

1) Collecting data: monitoring data is obtained from dam-embedded monitoring equipment and other available data sources. Such data may include osmometer water level data, GNSS displacement meter directional displacement data, deep displacement data, and other relevant monitoring information. And downloading and transferring the data into readable monitoring data. The data of the selected partial pore water pressure monitoring sample are shown in table 1, wherein the first behavior is the measuring point position.

TABLE 1

2) Data preprocessing: preprocessing the collected monitoring data and correlating the data with a time sequence to form time sequence data with the same time unit, wherein the data preprocessing comprises data cleaning, data flattening, data format conversion and data integration.

Data cleaning refers to performing repeated value deletion, recognizing and removing abnormal values, eliminating isolated points in data and the like on readable data.

Data flattening refers to time stamping format normalization of data acquired by different sensors and time stamping alignment of monitored data for different time units.

Data format conversion refers to converting the type of data field into a format suitable for time series analysis to accommodate subsequent model reading and writing.

Data integration refers to integrating data from different sensors into one complete time series data set and aligning it in the same time units for time series analysis.

3) Screening the deletion measuring point and the homologous measuring point: and identifying the measurement points with the missing data after pretreatment, and selecting the homologous measurement points by using a cross-hair method.

The cross line method is to use a dam to monitor that the same datum line of the buried point and the homologous measuring point data under the elevation have similar trend or dependence, take the missing data measuring point as a reference point, use a range finder or a measuring tool to pull out the datum line along a straight line, pull out two perpendicular lines from two ends of the datum line vertically and intersect the datum line to form a cross-shaped structure, and take the passing measuring point as the homologous measuring point. The sample selection is shown in fig. 3.

4) Measuring point positioning: and (3) establishing a data line graph to determine whether the homologous measuring points and the missing target measuring points have the same trend, if the trends are close, determining the homologous measuring points as the same group of data, and if the trends are different greatly, replacing the homologous measuring points.

5) Data decomposition: and decomposing the time series data of the target missing measuring point and the homologous measuring point by using an empirical mode decomposition algorithm to obtain different eigenmode functions IMF.

The empirical mode decomposition algorithm is a signal processing technique that aims to decompose complex nonlinear and non-stationary signals into a set of IMF components. These IMFs satisfy the condition that the number of local extremum points and zero crossings must be equal, or differ by a maximum of 1, over the entire data range; at any instant in time, the average of the envelope of the local maxima (upper envelope) and the envelope of the local minima (lower envelope) must be zero. The method comprises the steps of finding the maximum value and minimum value points of an original signal x (t), and fitting the maximum value points by a curve interpolation method to obtain an upper envelope line x of the signal _max (t) and lower envelope x _min (t) and averaging the routes of the upper and lower packets using the formula:

for the original signal x (t) and the average envelope m ₁ (t) subtracting to obtain the remaining signal d ₁ (t). Typically, for a stationary signal, it is the first IMF of the original signal x (t). But for non-stationary signalsInstead of monotonically increasing in a certain region, an inflection point may occur. If these inflection points reflecting the specific characteristics of the original signal x (t) are not selected, the first-order mode function obtained is inaccurate, i.e. d is usually obtained ₁ (t) the two conditions of IMF are not satisfied, so that it is necessary to continue screening.

For the remaining signal d ₁ (t) repeating the steps until SD (sieving threshold value, generally value (0.2-0.3) is smaller than threshold value), in this embodiment 0.2, obtaining the final suitable first-order modal component c ₁ (t), i.e., the first IMF. Wherein the SD formula is:

for signals x (t) and c ₁ (t) obtaining a first-order residual quantity r by taking a difference ₁ (t) r is to ₁ (t) repeating the above steps in place of the original signal x (t) until the nth order modal function c can be obtained after n times _n (t) and the final standard-compliant residual amount r _n (t). The formula of the original signal x (t) decomposed by EMD is:

the obtained measurement point data of the sample C1-3 is shown in FIG. 4 after being decomposed. The partial time steps are shown in table 2, wherein the first behavior raw data value is the superposition value of the decomposition amounts:

TABLE 2

6) Clustering data: and calculating the DTW distance by taking each missing component as a distance starting point and taking the decomposition component as a distance ending point. And aggregating and integrating the target measuring point component and the calculated distance component according to the obtained similar distance to obtain each clustering function set. The DTW distance is calculated as follows:

note that the two timing sequences are q= (Q1, Q2, …, qn) and c= (C1, C2, …, cm), respectively, with lengths n and m, respectively. Constructing a matrix D of n x m, and recording D [ i, j ] to represent Euclidean distance between Qi and Pj. The goal of DTW is to find a path from [1,1] to [ n, m ] such that the cumulative euclidean distance of the path is minimized. The calculation formula is as follows:

γ(i,j)＝d(q _i ,c _j )+min(γ(i-1,j-1),γ(i-1,j),γ(i,j-1)

wherein: w (w) ^* For DTW distance, γ (i, j) is the distance from [1,1] in matrix D]Arrive at [ i, j]Is the cumulative distance omega _k The square of the Euclidean distance between some two points (qi, cj) between q and c stored by some lattice point where the path passes through is represented, and d (i, j) is the current lattice point distance.

And calculating the DTW distance by taking each missing component as a distance starting point and the IMF component as a distance ending point, so as to obtain the nearest first distance D1, setting a threshold value as mu, obtaining the nearest DTW distance component of each missing component, and carrying out aggregation and integration to obtain each cluster function set D1, D2, … and Dn, wherein the clustering rule is that the first distance D1 to the final distance d1+mu are a function set. The distance of the DTW 100 time steps before taking the sample C1-3 is shown in FIG. 5. The partial calculation results are shown in Table 3

TABLE 3 Table 3

7) Model prediction: and establishing a prediction model based on the gate control circulation unit GRU for each function set based on the clustered homologous measuring points and the decomposed IMF data. The present embodiment uses a Python-based language to build the predictive model. Third party libraries used PyTorch, pandas and scikit-learn. Based on the foregoing samples, in the data processing section, as shown in fig. 6, 80% was used as the training set and 20% was used as the verification set.

The training set is input into the GRU model for training, and in this embodiment, the optimizer selects Adam and the loss function to use the relative error MSE. The super parameter is provided with a hidden layer of 64 layers and a GRU prediction layer of 1 layer. The super-parameters set the learning rate to 0.001, and the training times and batches were 100 and 64, respectively. After training is completed, the data of the verification set is input into the model, and the predicted component values of the corresponding function set can be obtained.

8) Superposition components: and superposing the predicted missing data value into the original data, and filling the missing part in the historical data. This step may be accomplished by adding the predicted value to the original IMF component. The partial results of the above samples obtained by model prediction are shown in table 4:

TABLE 4 Table 4

After model training is completed, the prediction effect of the model needs to be evaluated. The embodiment mainly adopts a main stream evaluation index mean square error MSE and an average absolute error MAE in machine learning. The formulas are as follows:

wherein y is _i As an actual value of the pore water pressure,for the predicted value, n is the number of samples。

FIG. 7 is a graph of the prediction results of the GRU model. For better evaluation of the model effect, the prediction result of the model is compared with the existing method, as shown in fig. 6. The index comparison results are shown in Table 5

TABLE 5

Model class	MSE	MAE
			Linear regression	2.04	1.43
Random forest	9.28	3.05
			ES-LSTM	1.87	1.37
GRU	0.52	0.72

As can be seen from Table 5, for the small and medium reservoir dam monitoring missing values lacking the pre-data and the experience component in the low data background, the mean square error and the mean absolute error of the algorithm proposed by the model are all due to the traditional linear regression, random forest and long and short term memory network LSTM. The invention combines the EMD algorithm, the DTW distance and the GRU model to solve the filling problem of dam monitoring history missing data under the background of the dead data. The comprehensive method fully utilizes the data decomposition and component analysis capability of an EMD algorithm, combines the similarity measurement of the DTW distance and the time sequence prediction capability of the GRU model, adapts to the problem of data loss of monitoring of small and medium-sized reservoir dams in China, ensures the reliability of data, and provides effective data for the overall safety evaluation of the follow-up dams.

In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other.

The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.

Claims (5)

1. The dam monitoring historical data filling method under the background of the spent data is characterized by comprising the following steps of:

acquiring monitoring data from the dam embedded monitoring equipment and available data sources;

preprocessing the collected monitoring data, and correlating the preprocessed data with a time sequence to form time sequence data with the same time unit;

identifying a measuring point with a missing data after pretreatment as a target missing measuring point, and selecting a homologous measuring point by using a cross-line method;

decomposing the time sequence data of the target missing measuring point and the homologous measuring point by using an empirical mode decomposition algorithm to respectively obtain a missing component and a decomposition component;

calculating similar distances by taking each missing component as a distance starting point and the decomposition component as a distance end point, sequencing the obtained similar distances, and aggregating and integrating the target measuring point component and the calculated distance component to obtain each clustering function set;

inputting the cluster function set into a prediction model to obtain each prediction component;

and superposing the predicted component on the eigenmode function component of the target missing measuring point.

2. The method for filling dam monitoring history data in the background of spent data according to claim 1, wherein the monitoring data comprises: osmometer water level data, GNSS displacement meter displacement data in all directions and deep displacement data.

3. The method for filling dam monitoring historical data in the spent data background according to claim 1, wherein the prediction model is constructed based on a gate-controlled circulation unit (GRU).

4. The filling method of dam monitoring historical data in a spent data background according to claim 1, wherein after identifying a missing measuring point of the preprocessed data as a target missing measuring point and selecting a homologous measuring point by using a cross-line method, further comprising:

and drawing a data line graph to determine whether the homologous measuring points and the target missing measuring points have the same trend, if the difference value of the trends is smaller than a preset threshold value, determining the same group of data, and if the difference value of the trends is larger than the preset threshold value, replacing the homologous measuring points.

5. The method for filling dam monitoring historical data in the context of spent data according to claim 1, wherein the preprocessing of the collected monitoring data comprises: data cleansing, data flattening, data format conversion and data integration.