CN103676649A - Local self-adaptive WNN (Wavelet Neural Network) training system, device and method - Google Patents

Abstract

The invention relates to a local self-adaptive WNN training system, device and method. The method comprises the steps of online local self-adaptive WNN structure adjustment; online WNN weight update; and WNN update strategy selection. According to the invention, a WNN model is updated online, it is ensured that the model can be popularized, the problems, which are caused by different uncertain factors and condition change of the system in practical, in WNN model adaption are overcome, and the operational stability of the system is improved, the fluctuation of production quality is lowered and service life of equipment is prolonged when the system, device and method are applied to industrial process control.

Description

Local Adaptive Wavelet Neural Network Training System, Equipment and Method

technical field

The invention relates to a local adaptive wavelet neural network training system, equipment and method, in particular to a local adaptive wavelet neural network training system, equipment and method with high system stability.

Background technique

              （1） There are sample points, and its input-output relationship is represented by a wavelet neural network (WNN) model

(1)

Here, is the radial basis wavelet function, its form is, and are respectively the translation parameter and the scale parameter, and is the hidden node number of WNN. When training WNN, first establish a wavelet neuron candidate set, the wavelet neuron parameters in the candidate set, and the initial value of the wavelet function parameters are determined by the results of data set clustering. For details, please refer to [Stephen A. Billings, Hua-Liang Wei. A new class of wavelet networks for nonlinear system identification. IEEE Transaction on Neural Networks, 16(4):862-870, 2005].

Obviously, WNN is a 3-layer neural network, and its model parameters include the number of hidden layer nodes, hidden layer-output layer connection weights and WNN hidden node parameters. WNN has good multi-scale approximation characteristics and generalization performance, and wavelet node parameters have clear physical meanings, wavelet functions have excellent local support, and are widely used in nonlinear dynamic system modeling, nonlinear classifier modeling, etc. field. Generalization performance (that is, the prediction accuracy of new samples) is an important index to measure the performance of WNN, and the generalization performance directly depends on the structural complexity of WNN (the number of nodes in the hidden layer of the wavelet network) and the selection of hidden node parameters of the wavelet network.

At present, WNN training methods are mainly divided into the following categories:

(1) WNN training method based on model criteria such as AIC and BIC.

There are mainly two methods of this type of method:

a. First select a large number of redundant wavelet nodes, and use the above model selection criteria to determine the WNN structure while keeping the wavelet node parameters unchanged. Since it is a very difficult problem to select the appropriate wavelet node parameters, this method is difficult to find the optimal WNN model;

b. Use the genetic algorithm (GA) to select the optimal wavelet node parameters and WNN structure based on the above model selection criteria. This method is computationally expensive and difficult to apply in practice.

(2) Empirical risk minimization WNN training method based on gradient descent method.

Since the optimization problem is a multi-variable, nonlinear, non-convex optimization problem, this method has the same defects as the conventional neural network training method based on gradient descent, such as slow training speed, easy convergence to local minimum points, and difficulty in determining the appropriate WNN structure and other defects. Although some researchers use the conjugate ladder descent method to improve the WNN training speed to a certain extent, the above drawbacks are still not resolved.

(3) Use the support vector machine (SVM) theoretical method to improve the generalization performance of WNN.

It is a landmark achievement in the development of statistical learning theory. It not only has a solid theoretical foundation, but also has good generalization performance, and has attracted great attention from academia and industry. Since SVM and WNN have the same structure, the applicant proved that the radial basis kernel function is a kind of kernel function satisfying the Mercer condition, and proposed a multi-scale wavelet SVM (WSVM) modeling method similar to WNN. The relevant results of the author were published in the 4th issue of "Journal of Circuits and Systems" in 2008. Due to the limitation of calculation amount, this method only gives the WNN modeling method on two scales.

(4) WNN improvement method based on multi-core learning theory.

In recent years, some scholars have proposed a multi-core SVM method. This method uses a linear combination of multiple kernel functions to represent the kernel function of SVM, and obtains the weight of the optimal kernel function and the parameters of the SVM model by solving an optimization problem. When the kernel function is radial basis wavelet function, multi-scale SVM and multi-kernel SVM essentially have the multi-scale approximation characteristics of WNN. Although this method inherits the advantages of SVM and multi-scale approximation, it cannot adjust the parameters of the kernel function, which seriously affects the performance of the model.

It should be pointed out that none of the above methods can adjust the structure and parameters of the WNN model online. In practical applications, there will be uneven distribution of samples (the modeling training data of some modalities is not sufficient or complete), changes in equipment working conditions, input Uncertain factors such as interference, external environment, and equipment aging lead to a decline in the prediction accuracy of the trained WNN model, so it is necessary to train the WNN model online.

Although the process characteristics of modeling objects in each working mode (working area) are different, but in general, all process characteristics have similar potential characteristics, namely the "common part" part, and the description work Modality-specific latent properties of the "variation pattern (Specific part)" part. Therefore, the present invention adopts different model update strategies according to the mode change of the process system, which can greatly reduce the time and computational complexity of model learning, thereby realizing online learning of models.

Appendix: Relevant background knowledge.

Optimum criteria for optimal experimental design.

For a given data set, assume that they have the following linear relationship;

Among them, is Gaussian noise (note that it is a random variable with a mean of 0 and a variance of ). The optimal experimental design is to choose the experimental data that contains the most information to learn the prediction function, so that the prediction error is the smallest. order, then. The optimal estimation method is to use the minimum mean square error as the cost function, and its optimal solution is .           

In order to ensure the generalization of the regression model, we also expect the variance of the model parameters to be minimal. because

It can be seen that the model parameter estimation shown in the above formula is an unbiased estimation, and its covariance can be expressed as

Therefore, the forecast variance of the predictive model is

Optimum优化准则引起人们广泛的关注。

Optimum优化准则就是最小化参数协方差矩阵的迹，其等价平均方差。

Optimum优化准则方法就是使得参数协方差矩阵的行列式（determinant）最小。 It can be seen from the above formula that the minimum prediction variance is equivalent to the minimum variance of the model estimated parameters, ie. At present, there have been multiple optimization criteria to measure the variance of model parameters, among which and

Optimum optimization criterion has aroused widespread concern.

The Optimum optimization criterion is to minimize the trace of the parameter covariance matrix, and its equivalent mean variance.

The Optimum optimization criterion method is to minimize the determinant of the parameter covariance matrix.

Since the output of WNN has a linear relationship with the output of hidden nodes, if the hidden nodes of WNN can be seen as data characteristics, it is obvious that the problem of controlling the complexity of the WNN model is transformed into a feature selection problem. Therefore, using the Optimum method to eliminate WNN redundant hidden nodes can guarantee the generalization performance of WNN.

Manifold regularization.

Manifold learning algorithms can reveal the intrinsic geometric structure of low-dimensional data from high-dimensional data, and each intrinsic dimension corresponds to an explanatory variable, so that high-dimensional data can be explained according to a small number of hidden variables. According to Pareto's law, most of the important characteristics of nonlinear systems are described by the locality of nonlinear systems. Therefore, considering the local geometric properties (such as distance, angle) of the dataset is an effective way to improve the performance of WNN. Like the linear projection method, the manifold learning algorithm relies on the calculation of the similarity matrix, but its computational complexity does not increase greatly.

.相似度矩阵使用图矩阵定义，图边权重矩阵定义如下： For regression datasets, define

.The similarity matrix is defined using a graph matrix, and the graph edge weight matrix is defined as follows:

Here, is the class label of ; is the K-nearest neighbor set of . hypothetical sample

The representation on the embedded low-dimensional manifold is

According to the spectrogram theory, the spectrogram regularization factor can use a matrix to measure the smoothness of the low-dimensional representation. The Laplacian manifold regularization factor can be expressed as

where, the diagonal matrix, , . By minimizing the Laplacian manifold regularization factor, the data set in the low-dimensional space maintains the local geometric structure of the high-dimensional original data set, effectively improving the performance of the learning machine.

Contents of the invention

In order to make the system more stable, the invention provides a local adaptive wavelet neural network training device, system and method with high system stability.

To achieve this purpose, the present invention provides a local adaptive wavelet neural network training system,

The local adaptive wavelet neural network training system is composed of offline WNN training module connected by signal and online updating WNN module.

Preferably, the offline WNN training module establishes a WNN initial model;

The online update WNN module uses different WNN model update strategies to predict the data according to the distribution characteristics of the new data.

The present invention also provides a local self-adaptive wavelet neural network training device, the local self-adaptive wavelet neural network training device is composed of a signal-connected data preprocessing module, an online satisfactory G-K fuzzy clustering module, a wavelet function parameter setting module, and a WNN updater Strategy selection module, hidden node selection module, extended Kalman filter (EKF) training module, Laplacian manifold regularization LSSVM module, optimal experimental design Optimum module, sample increase WNN weight update module, sample removal WNN weight update module, WNN prediction module composition.

Preferably, the function and function of the data preprocessing module: the function input parameter is a data set, and the output parameter is a normalized data set;

The functions and functions of the online satisfaction G-K fuzzy clustering module:

Input parameters: dataset, initial membership matrix, number of clusters;

Output parameters: number of clusters, membership matrix;

The functions and functions of the wavelet function parameter setting module:

Input parameters: membership matrix, data set, number of clusters, node function generation strategy;

Output parameter: wavelet function parameter matrix;

The functions and functions of the WNN update strategy selection module:

Input parameters: membership degree matrix at past time, number of clusters at past time, current membership degree matrix, current number of clusters;

Output parameters: membership matrix, number of clusters;

The functions and functions of the implicit node selection module:

Input parameters: candidate node set, WNN weight vector, fitting data set;

Output parameters: wavelet node parameters, corresponding weights;

The functions and functions of the Extended Kalman Filter (EKF) training module:

Input parameters: wavelet node parameters, algorithm termination threshold, training data set;

Output parameters: wavelet node parameters, corresponding weights;

The functions and functions of the Laplacian manifold regularization LSSVM module:

Input parameters: training data set, model parameters, matrix L;

Output parameter: weight vector;

Optimum模块的功能与作用： The optimal experimental design

The functions and functions of the Optimum module:

Input parameters: training data set, weight vector, parameter matrix composed of WNN hidden nodes, candidate node label vector;

Output parameters: node selection flag vector;

This sample increases the functions and functions of the WNN weight update module:

Sliding wide-mouth data set, newly added data, Q matrix, R matrix WNN implicit node parameter matrix;

Output parameters: update Q matrix, update R matrix;

This sample removes the functions and functions of the WNN weight update module:

Input parameters: Q matrix, R matrix, remove data number;

Output parameters: update Q matrix, update R matrix;

The functions and functions of the WNN prediction module:

Input data, WNN hidden node parameter matrix, weight matrix, input data vector;

Output parameter: Predicted output data.

The present invention provides a kind of local adaptive wavelet neural network training method again, this local adaptive wavelet neural network training method comprises:

S31. Online local adaptive WNN structure adjustment;

S32. Updating the WNN weight online;

S33. The WNN updates the selection strategy.

Preferably, S31, online local adaptive WNN structure adjustment, specifically includes:

S311. Select WNN hidden nodes;

S312. Control the complexity of the WNN model.

Preferably, the S312, controlling the WNN model complexity, specifically includes:

S3121, WNN weight estimation based on Laplacian manifold regularization LSSVM;

Optimum的WNN隐含节点序贯选择。 S3122, based on

Optimum's WNN implies sequential selection of nodes.

Preferably, the S32, updating the WNN weight online, specifically includes:

S321, sample adding and updating stage;

S322. Sample removal and update phase.

Preferably, the S33, WNN update selection strategy specifically includes:

S331, initialization;

S332. Perform clustering according to the membership degree matrix;

S333, judging whether to end;

S334. Find the sample least similar to the sample center as the new cluster center

S335. Calculate a corresponding new initial membership degree matrix;

S336, order, transfer to S332.

The beneficial effects brought by the technical solution provided by the embodiments of the present invention are:

1). Based on the above ideas, in the WNN modeling process, the WNN nodes describing the "common mode" part remain unchanged, and only need to adjust the local WNN structure to fit the "changing mode", which greatly reduces the amount of calculation required for training WNN, suitable for Online WNN modeling;

2). Iteratively select wavelet neurons from the WNN hidden node candidate set to add to WNN, and use the Extended Kalman Filter (EKF) method to adjust the parameters and related weights of the newly added wavelet nodes;

3). The online update WNN weight algorithm based on the sliding window QR decomposition, and the model weight is corrected online through the recursive algorithm of sample increase and decrease;

Optimum优化准则结合的WNN复杂度控制方法，较好地考虑了训练数据集的几何结构，保证了WNN的推广性能。 4). Introduce the idea of manifold learning, and propose Laplacian regularization and

The WNN complexity control method combined with the Optimum optimization criterion better considers the geometric structure of the training data set and ensures the generalization performance of WNN.

In this way, in practical applications, combining prediction error and prior knowledge, by adjusting the WNN structure online and controlling the model complexity or WNN weight update, the prediction accuracy of WNN is guaranteed, which effectively overcomes the difficulty of online learning and generalization performance of existing WNN algorithms. The problem.

Description of drawings

Through the following description of a preferred embodiment of the present invention in conjunction with the accompanying drawings, the technical solution and technical effects of the present invention will become clearer and easier to understand. in:

Fig. 1 shows the structural representation of the local adaptive wavelet neural network training system of the embodiment of the present invention;

Fig. 2 shows the structural representation of the local adaptive wavelet neural network training equipment of the embodiment of the present invention;

Fig. 3 shows the method flowchart of the local adaptive wavelet neural network training method of the embodiment of the present invention;

Fig. 4 shows the flow chart of the method for controlling the complexity of the WNN model in Fig. 3;

Fig. 5 shows the flow chart of the method for updating WNN weight online in Fig. 3;

FIG. 6 shows a flow chart of the method for WNN updating and selecting a strategy in FIG. 3 .

Detailed ways

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be pointed out that the "left" and "right" are only for the purpose of illustration and description in conjunction with the accompanying drawings, and are not limited thereto.

first embodiment

Fig. 1 shows a schematic structural diagram of a local adaptive wavelet neural network training system according to the first embodiment of the present invention.

The local adaptive wavelet neural network training system is composed of offline WNN training module connected by signal and online updating WNN module.

The offline WNN training module mainly establishes the WNN initial model.

The steps are as follows: firstly cluster the training data set by using the online satisfactory G-K fuzzy clustering method, and determine the parameters of multiple wavelet functions according to the clustering results. The wavelet function scale parameters and translation parameters are randomly generated according to the cluster center, radius and variance results, and these wavelet node functions form the WNN hidden node candidate set; then use the existing WNN training algorithm to establish the initial WNN model as the current WNN.

The online update WNN module uses different WNN model update strategies to update WNN and predict new data according to the clustering results of new data.

The steps are to use the online satisfactory G-K fuzzy clustering method to cluster the new data, and use the following three strategies to predict the data according to the clustering results;

S11. If the new data and the previous time data belong to the same cluster and the membership degree is >0.5, the current WNN is still used to predict the new data.

, If the new data clustering result is a newly added cluster or the new data is transferred to other clusters (membership degree>0.5) or the original cluster membership degree<0.2 (deviating from the current working condition model), use the online local automatic Adapt to the WNN update algorithm, that is, first select the optimal wavelet function from the candidate set and recursively add it to the hidden layer of the current WNN model, and use EKF to train the parameters of the newly added hidden nodes until the error threshold is met; then Laplacian regularization and Optimum optimization Criteria combine to select WNN hidden nodes, and add redundant nodes deleted from the WNN model to the candidate set of WNN hidden nodes.

. If the membership degree of the new data to the original cluster is ≥0.2 and ≤0.5, it indicates that the object is affected by system dynamic uncertainties, and only the weight parameters of the WNN model need to be updated. This method realizes the WNN weight update by adding samples in the sliding window to update the WNN weight stage and the WNN weight update stage after the old samples are removed. If using S11 or S12 to update the WNN model, it is necessary to replace the existing WNN model with the updated WNN model as the current WNN prediction data.

The technical solution provided by the embodiment of the present invention can make full use of historical knowledge (candidate WNN hidden node set) to quickly update the WNN model, so as to facilitate online training of WNN.

second embodiment

Fig. 2 shows a schematic structural diagram of a locally adaptive wavelet neural network training device according to a second embodiment of the present invention.

The local adaptive wavelet neural network training equipment consists of a signal-connected data preprocessing module, an online satisfactory G-K fuzzy clustering module, a wavelet function parameter setting module, a WNN update strategy selection module, a hidden node selection module, and an extended Kalman (EKF) It is composed of training module, Laplacian regularization LSSVM module, experimental design Optimum module, sample addition WNN weight update module, sample removal WNN weight update module, and WNN prediction module.

The function and role of the data preprocessing module: the input parameter of the function is a data set, and the output parameter is a normalized data set.

The functions and functions of the online satisfaction G-K fuzzy clustering module:

Input parameters: dataset, initial membership matrix, number of clusters;

Output parameters: number of clusters, membership matrix.

The functions and functions of the wavelet function parameter setting module:

Input parameters: membership matrix, data set, number of clusters, node function generation strategy;

Output parameter: wavelet function parameter matrix.

The functions and functions of the update strategy selection module:

Input parameters: membership degree matrix at past time, number of clusters at past time, current membership degree matrix, current number of clusters;

Output parameters: membership matrix, number of clusters.

The functions and functions of the implicit node selection module:

Input parameters: Candidate WNN hidden node set, WNN weight vector, fitting data set;

Output parameters: wavelet node parameters, corresponding weights.

Functions and functions of the Extended Kalman (EKF) training module:

Input parameters: wavelet node parameters, algorithm termination threshold, training data set;

Output parameters: wavelet node parameters, corresponding weights.

The functions and functions of the regularized LSSVM module:

Input parameters: training data set, model parameters (selected by cross-validation method), (recommended selection range is [2,20]), matrix (the number of neighbors is selected within [3,5]);

Output parameter: weight vector.

optimal experimental design The functions and functions of the Optimum module:

Input parameters: training data set, weight vector, parameter matrix composed of WNN hidden nodes, candidate node label vector (0-the node is removed, 1-the node is to be selected);

Output Parameters: Vector of node selection flags.

The functions and functions of the sample increase WNN weight update module:

Sliding wide-mouth data set, newly added data, Q matrix, R matrix WNN implicit node parameter matrix;

Output parameters: update Q matrix, update R matrix.

The functions and functions of the sample removal WNN weight update module:

Input parameters: Q matrix, R matrix, remove data number;

Output parameters: update Q matrix, update R matrix.

The functions and functions of the prediction module:

Input parameters: WNN hidden node parameter matrix, weight matrix, input data vector;

Output parameter: Predicted output data.

According to the relationship of the modules in Fig. 2, combined with the input and output of the above modules, it is easy to analyze the information exchange and processing relationship between each module, and the technical personnel can realize the method of the present invention without creative labor.

The device is completed by the above program modules, and does not require users to set system parameters. It is very convenient to use and has good maintainability. It does not need to add hardware devices in practical applications, and it is convenient to modify the existing system.

third embodiment

Fig. 3 shows a method flow chart of a locally adaptive wavelet neural network training method according to a third embodiment of the present invention.

The local adaptive wavelet neural network training method includes:

S31. Online local adaptive WNN structure adjustment;

S32. Updating the WNN weight online;

S33. The WNN updates the selection strategy.

The S31, online local adaptive WNN structure adjustment, specifically includes:

S311. Select WNN hidden nodes;

S312. Control the complexity of the WNN model. the

S311. Select WNN hidden nodes, that is, gradually increase the hidden nodes of WNN and adjust node parameters until the fitting error meets the preset threshold; specifically include:

The online local adaptive WNN adjustment method is used to overcome the mismatch between the WNN model and the actual system caused by the nonlinear structure change of the system due to the transformation of the system working mode. The method is to gradually select wavelet nodes that can reduce the WNN approximation error from the wavelet node candidate set to join WNN, and then use the EKF method to adjust the parameters and weights of the newly added hidden nodes. Its detailed description is as follows:

Assuming that the current WNN already has a wavelet neuron, for the training sample set, let , be the output of the WNN, and be the WNN approximation error, , denote. Firstly, the importance of each node in the candidate node set is estimated, and its importance is estimated by the degree of error reduction generated by the upper projection. The projection on

(2)

where is a normalized vector. Note that the above formula (2) can be regarded as an approximation on the vector, so its corresponding projection (approximation) error is

                (3)

(3)

Obviously, choose the node with the largest error reduction, that is,

            (4)

(4)

Here, is a very small positive number, which can be regarded as a regularization factor, and the purpose is to prevent overfitting problems caused by being too small. Obviously, adding wavelet neurons will reduce the approximation error of WNN to samples. It can be proved that the algorithm is convergent (the analysis process is omitted), and its convergence rate is given by Theorem 1 below (the proof is omitted).

Theorem 1. Suppose the sample set satisfies some bounded and unknown bounded function relationship. If there is a positive number, so that it is true for all, then for any approximation error threshold, the algorithm in this paper passes at most

                    (5)

(5)

After increasing the wavelet neurons for 2 iterations, the fitting error does not exceed, ie. Among them, ; is the rounding operator. Among them, (the intention of the modification is to indicate the output of the node on the data set, which is consistent with the previous one, so it is changed to) is the degree of correlation between the node and the sample set, and the larger the value, the greater the degree of correlation between the function and the sample set.

Theorem 1 shows that the error can reach the set value by increasing the hidden nodes of WNN. However, in practice, it is difficult to pre-set the parameters of the wavelet function in the candidate set, which inevitably leads to too many hidden nodes. According to the principle of Occam's razor, redundant hidden nodes will reduce the generalization performance of the WNN model. Let be the fitting error of WNN with hidden nodes, and the fitting error of WNN decreases at a rate after adding hidden wavelet nodes, namely

(6)

Since the degree of correlation determines the speed of error reduction, it affects the number of hidden nodes in WNN. Therefore, it is necessary to locally adjust the parameters of the currently selected wavelet neurons to maximize the correlation of local wavelet nodes, thereby reducing the sample fitting error. The invention adopts the EKF method to search for the optimal parameters and related weights of the newly added wavelet nodes. The translation, scale and weight of the local wavelet neuron, remember. In order to speed up the convergence speed of the training algorithm, this paper increases the parameter learning rate factor. The parameter training method based on the EKF algorithm is as follows: 

                                      (7)

(7)

(8)

             (9)

(9)

                            (10)

(10)

Among them, and are respectively the local wavelet neuron output and the expected output; is the training error; is the learning rate; is the Kalman gain vector; the estimated noise covariance can be obtained by recursion; is the state estimation error covariance matrix; is The number of iterations of the EKF algorithm; . In order to adaptively adjust the learning rate to speed up the convergence of the algorithm, Theorem 2 gives the conditions that the learning rate must satisfy, which better solves the problem of adaptive selection of the rate.

Theorem 2. Let be the parameter of the hidden node, the predicted output of the hidden node for a given new sample, is the expected output of the hidden node, is the learning rate of the corresponding parameter, and is the EKF gain vector. If the learning rate satisfies

                                      (11)

(11)

Then the WNN training algorithm based on EKF is consistent convergence, among them.

It is easy to prove Theorem 2 by choosing the Lyapunov function, where is the predicted output of the sample by WNN.

Because the method gradually adds new hidden nodes and uses the variable step size EKF algorithm to adjust the local wavelet node parameters, the convergence speed is very fast, and the algorithm can obtain a satisfactory modeling error in an average of 40 seconds.

Optimum结合的WNN模型复杂度控制方法。 In practical applications, although gradually increasing the number of hidden nodes in WNN can effectively reduce the modeling error, redundant hidden nodes will inevitably appear. According to the principle of Omka's razor, too many redundant nodes in WNN will reduce the generalization performance of the model. Utilizing the local geometric structure information of data is an important means to improve the performance of WNN. Drawing on manifold learning theory, the present invention proposes Laplacian regularization and

A WNN model complexity control method combined with Optimum.

Optimum优化准则结合的WNN复杂度控制方法。 , Control the complexity of the WNN model, that is, based on Laplacian regularization and

WNN complexity control method combined with Optimum optimization criterion.

optimum准则，考虑到数据集的局部几何结构，本发明提出两步WNN隐含节点选择方法：首先基于Laplacian流形正则化的最小二乘支持向量机(LSSVM)估计回归模型WNN权重参数，然后根据估计参数方差最小的

Optimum优化准则序贯地选择WNN隐含节点。 According to the optimal experimental design method, under the premise of keeping the hidden node parameters of WNN unchanged, minimizing the variance of the expected model parameters is equivalent to selecting important features of the data set. Based on optimal experimental design

Optimum criteria, considering the local geometric structure of the data set, the present invention proposes a two-step WNN hidden node selection method: firstly estimate the weight parameters of the regression model WNN based on the least squares support vector machine (LSSVM) of Laplacian manifold regularization, and then according to Estimated parameters with the smallest variance

The Optimum optimization criterion sequentially selects WNN hidden nodes.

As shown in Figure 4, the S312, controlling the complexity of the WNN model, specifically includes:

S3121, WNN weight estimation based on Laplacian regularization LSSVM;

For the above data set, it is assumed that there are hidden nodes in WNN, and the input samples,,, WNN output and its hidden node output are recorded as sum, . Note, Let be the vector composed of the first feature of the data set, feature set. Obviously, the th feature of the data set corresponds to the th hidden node of WNN. Therefore, selecting an important WNN node is equivalent to selecting an important feature from the data set. Suppose you choose a feature of . Let be the sample set formed after selecting features, where . Order,,, its definition is as follows:

                               (12)

(12)

but. Then the regression model in the selected feature space is

        

where is the unknown error with a mean of 0. It is assumed that the errors of different data samples are independent of each other, but have the same variance. In order to ensure the generalization performance of the selected nodes and take into account the local geometric structure of the data, we use the optimal value obtained by the LSSVM based on Laplacian regularization, and the optimization problem is as follows:

     (13)

(13)

Wherein, for regularization factor, matrix and weight definition, please refer to the relevant part of the background technology. The optimal solution obtained by deriving the objective function and being equal to 0:

    

where is the identity matrix of . order, then

                                                                 (14)

(14)

Optimum的WNN隐含节点序贯选择。 S3122, based on

Optimum's WNN implies sequential selection of nodes.

Note that the positive definite symmetry and, let

The bias and variance of the estimated parameters are then

                                   (15)

(15)

                       (16)

(16)

According to formula (14), it can be seen that the predicted value is, then the variance of the forecast error is

                 (17)

(17)

Note that and , according to formula (16), the variance of the parameter estimation error is

     (18)

(18)

Put the above formula into formula (17) to get

 

Generally speaking, the regularization coefficient is set to be relatively small, while the error penalty coefficient is set to be large. Note that and are positive definite matrices, and we have

                        (19)

(19)

Similarly

                       (20)

(20)

Optimim优化准则：。 Based on the principle of optimal experimental design, we expect the selected feature subset to minimize the covariance matrix of the estimated parameters. And minimization can also minimize the prediction error of new samples, which can be equivalent to

Optimim optimization guidelines: .

For matrices, L is a positive semidefinite matrix, which is a positive definite, invertible matrix. Let, according to the Woodbury formula

                    (21)

(twenty one)

Note that we can get

                (22)

(twenty two)

Considering it is a constant, the problem of selecting WNN hidden nodes is transformed into the following optimization problem:

                       (23)

(twenty three)

Note that there are only selection features: , so it can be written as . Since, then, the optimization problem shown in formula (23) is transformed into

            (24)            

(twenty four)

We use a sequential optimization method to solve the above optimization problem. First, assuming that a node has been selected, the first node can be selected through the following optimization problem:

                (25)

(25)

Order, according to Woodbury and Sherman-Morrison formula can be obtained

             (26)

(26)

Since the sum is a constant matrix, the sequential optimization problem shown in formula (25) is transformed into

              （27）

(27)

In this way, important hidden nodes of WNN can be selected sequentially by solving the above optimization problem.

This method is proposed by the applicant for the first time. Compared with the existing technology, this method has advanced ideas and technical methods. It can adjust the complexity of WNN online and obtain the optimal parameter value of WNN. It is especially suitable for systems with nonlinear structural changes and uncertainties. Modeling dynamic uncertainty, system identification situations with manifold-structured datasets.

, Updating WNN weights online.

In order to overcome the influence of equipment dynamics and uncertain factors during the operation of industrial objects, avoid using a large amount of computer memory, and reduce the influence of old samples on the model, the present invention adopts an online update weight algorithm based on a fixed-length sliding window. The sliding window needs to remove one of the least important samples from the training samples after adding a new sample. In the online update WNN weight algorithm, it is assumed that the WNN model first obtains a WNN model containing wavelet neurons according to the training sample set.

As shown in Figure 5, S32, online update WNN weights, specifically include:

S321, sample increment update stage;

S322. Sample removal and update phase.

, Sample incremental update stage, specifically including:

Let the new sample be, the WNN hidden node output be, . Assuming that QR is decomposed into, then the upper triangular matrix of QR decomposed into can be obtained row by row through the following formula:

 

  

                     (28) 

(28)

where is the first column of the matrix. Assuming that the first sample is to be removed, since the final prediction model has nothing to do with the order of the samples, the sample is first swapped with , and then the sample recursion form after the sample is removed is given. Suppose the matrix A is decomposed by QR. If the QR decomposition of the matrix A after exchanging the first row and the first row is, where is an orthogonal matrix and is an upper triangular matrix, then. Here is the matrix after the row and row of matrix Q have been exchanged. In this way, the least important sample is moved to the first row through the above method, so that only the first row of samples is removed.

, The sample removal update stage, specifically including:

，假设的QR分解为，那么矩阵的QR分解的上三角矩阵可逐行由下式得到： Recursive QR decomposition of the system matrix after removing the first row of samples. The form of the output vector of the known sample corresponding to the hidden node of WNN is, and the QR decomposition of it is

, the assumed QR decomposition is, then the upper triangular matrix of the QR decomposition of the matrix can be obtained row by row as follows:

 

  

             

  

        

where is the first column of the matrix. Through the above method, it is easy to obtain the recursive QR decomposition form of the system matrix after removing the samples, and then obtain the updated weights. If only the oldest sample is removed from the training samples, the value update steps of the above sample increment and sample elimination can be combined into one step.

After the calculation of the above two stages, the WNN weight update is calculated by the following formula

 

This algorithm step eliminates the calculation of the matrix and inversion, and only needs to recursively calculate their QR decomposition, and notice that the calculation of the upper triangular matrix inversion is very small, which fully meets the online learning requirements.

, WNN update selection strategy.

For new data samples, it is necessary to determine whether the WNN model is updated and the above-mentioned update strategy adopted according to a certain strategy. The existing update criterion is to determine whether to update and the update strategy according to the judgment result that the relative prediction error is greater than the threshold value defined in advance within the set time period.

Conventional methods need to determine the length of the discriminant period and two threshold parameters in advance, and how to select these parameters is still an open problem. Considering that there are a large number of cross-overlaps among complex category data, the physical meaning of the data clustering results is obvious and reliable, and can well explain the data distribution characteristics. Fuzzy satisfactory clustering to establish PH neutralization process model. Control and Decision, 2002] to determine WNN weight parameters or WNN structure adjustment learning strategy. Compared with the existing update strategy selection criteria, the present invention uses a method for accurately selecting the WNN update strategy for data clustering results, which is suitable for the online realization of complex data distribution clustering, without the need to select cluster number parameters, and overcomes the limitations of the existing methods. shortcoming. This method also has another significant advantage. The scale parameters and translation parameters of the wavelet function are selected according to the clustering results, which can greatly reduce the time required for parameter adjustment.

Assume that its sample is expressed as , where is the input of the system and is the output of the system. If the input and output of the system are regarded as a sample, ie, the sample set is expressed as .

As shown in Figure 6, S33 and WNN update the selection strategy, which is based on online satisfaction G-K fuzzy clustering, and its implementation steps are as follows:

S331. Initialization: set the number of initial clusters, the algorithm end threshold, and the initial membership degree matrix.

, Clustering according to the membership degree matrix: according to the initial membership degree matrix, solve the satisfactory G-K fuzzy clustering optimization problem

;

。计算得到隶属度矩阵，然后根据样本所属各聚类的隶属度选取最大值进行分类, 将样本集分为c个子集。 Among them, the membership value,; is the sample; is the cluster center; m is the ambiguity, where the covariance ,

. Calculate the membership degree matrix, and then select the maximum value for classification according to the membership degree of each cluster to which the sample belongs, and divide the sample set into c subsets.

为预设阈值)时算法结束，否则算法转到下一步。一般取作为性能指标，为爹代次数，一般取 , Judging whether to end: calculate the current value of the given system performance index, when (

is the preset threshold), the algorithm ends, otherwise the algorithm goes to the next step. It is generally taken as a performance index, and it is the number of generations, which is generally taken as

S334. Find a new cluster; find a sample that is not similar to each cluster according to the membership degree matrix. In order to avoid noise, it is generally necessary to find several similar samples and calculate their average value as the new cluster center.

, Let be the new clustering initial center, and calculate the corresponding new initial membership degree matrix.

, 转S332。 ,make

, turn to S332.

For newly arrived samples, the updating strategy selection method of the present invention is as follows:

(1) Keep WNN unchanged: the membership degree of the calculated sample belonging to the current cluster is greater than 0.5;

(2) The local structure of WNN needs to be adjusted: when a new cluster needs to be added or it belongs to another cluster;

(3) WNN weight update: when the sample is at the intersection of multiple clusters (generally, whether the membership degree of the current cluster is in the interval (0.2, 0.5)).

The invention does not require high CPU performance, can be implemented and applied in embedded systems, and greatly improves the scope of application of the method, such as classifiers in pattern recognition, interpolation and fitting of complex nonlinear systems, and the like.

The invention can update the WNN model on-line, and can ensure the generalization performance of the model, and can well overcome the problem of WNN model adaptation caused by various uncertain factors and changes in working conditions in the actual system, and is used in industrial process control It can increase the stability of system operation, reduce the fluctuation of product quality, improve the life of equipment, and obtain good economic benefits.

For those skilled in the art, with the development of technology, the inventive concept can be implemented in different ways. The embodiments of the invention are not limited to the examples described above but may vary within the scope of the claims.

Claims (9)

1. a local auto-adaptive wavelet neural network training system, is characterized in that,

Off-line WNN training module and online updating WNN module that this local auto-adaptive wavelet neural network training system is connected by signal form.

2. local auto-adaptive wavelet neural network training system as claimed in claim 1, is characterized in that,

This off-line WNN training module is set up WNN initial model;

This online updating WNN module, according to the distribution character of the data of newly arriving, adopts different WNN model modification strategies to predict data.

3. a local auto-adaptive wavelet neural network exercise equipment, is characterized in that,

The data preprocessing module that this local auto-adaptive wavelet neural network exercise equipment is connected by signal, be satisfied with G-K fuzzy clustering module, wavelet function parameter online and module, WNN update strategy are set select module, implicit node to select module, spreading kalman (EKF) training module, Laplacian regularization LSSVM module, experimental design Optimum module, sample to increase that WNN weight update module, sample remove WNN weight update module, WNN prediction module forms.

4. local auto-adaptive wavelet neural network exercise equipment as claimed in claim 3, is characterized in that,

The Function and operation of this data preprocessing module: function input parameter is data set, output parameter is normalization data set;

This is satisfied with the Function and operation of G-K fuzzy clustering module online:

Input parameter: data set, initial degree of membership matrix, number of clusters;

Output parameter: number of clusters, degree of membership matrix;

This wavelet function parameter arranges the Function and operation of module:

Input parameter: degree of membership matrix, data set, number of clusters, node function generation strategy;

Output parameter: wavelet function parameter matrix;

This WNN update strategy is selected the Function and operation of module:

Input parameter: the past is degree of membership matrix constantly, the past is number of clusters constantly, current degree of membership matrix, current number of clusters;

Output parameter: degree of membership matrix, number of clusters;

This implicit node is selected the Function and operation of module:

Input parameter: both candidate nodes set, WNN weight vectors, fitting data collection;

Output parameter: small echo node parameter, respective weights;

The Function and operation of this spreading kalman (EKF) training module:

Input parameter: small echo node parameter, algorithm stops threshold value, training dataset;

Output parameter: small echo node parameter, respective weights;

The Function and operation of this Laplacian regularization LSSVM module:

Input parameter: training dataset, model parameter

,

, matrix

;

Output parameter: weight vectors;

The Function and operation of this experimental design Optimum module:

Input parameter: training dataset, weight vectors, the parameter matrix that the implicit node of WNN forms, vertex ticks vector to be selected;

Output parameter: node selected marker vector;

This sample increases the Function and operation of WNN weight update module:

The wide mouthful data set that slides, newly adds data, Q matrix, and R matrix W NN implies node parameter matrix;

Output parameter: upgrade Q matrix, upgrade R matrix;

This sample removes the Function and operation of WNN weight update module:

Input parameter: Q matrix, R matrix, removes data number;

Output parameter: upgrade Q matrix, upgrade R matrix;

The Function and operation of this WNN prediction module:

Input data, WNN implies node parameter matrix, weight matrix, input data vector;

Output parameter: prediction output data.

5. a local auto-adaptive wavelet neural network training method, is characterized in that,

This local auto-adaptive wavelet neural network training method comprises:

S31, online local auto-adaptive WNN structural adjustment;

S32, online updating WNN weight;

S33, WNN upgrade selection strategy.

6. local auto-adaptive wavelet neural network training method as claimed in claim 5, is characterized in that,

S31, online local auto-adaptive WNN structural adjustment, specifically comprise:

S311, choose the implicit node of WNN;

S312, control WNN model complexity.

7. local auto-adaptive wavelet neural network training method as claimed in claim 6, is characterized in that,

This S312, control WNN model complexity, specifically comprise:

S3121, the WNN weight based on Laplacian regularization LSSVM are estimated;

S3122, the WNN based on Optimum imply the sequential selection of node.

8. local auto-adaptive wavelet neural network training method as claimed in claim 5, is characterized in that,

This S32, online updating WNN weight, specifically comprise:

S321, sample increase progressively the more new stage;

S322, sample remove the more new stage.

9. local auto-adaptive wavelet neural network training method as claimed in claim 5, is characterized in that,

This S33, WNN upgrade selection strategy, specifically comprise:

S331, initialization;

S332, according to degree of membership Matrix Cluster;

S333, judge whether to finish;

S334, find new cluster;

S335, the corresponding new initial degree of membership matrix of calculating;

S336, order, turn S332.

Images