CN103018177A - Spectrogram abnormal sample point detection method based on random sampling agree set - Google Patents

Abstract

The invention discloses a spectrogram abnormal sample point detection method based on a random sampling agree set. The spectrogram abnormal sample point detection method comprises the following steps of: eliminating part of an abnormal sample in advance so as to obtain a correcting sample set through principal component analysis on the base of a maximum posteriori probability random sampling agree set and starting from a given spectroscopic data, carrying out random sampling, establishing multivariated correcting model and evaluating a model property, and selecting an appropriate sample subset as an inner point set through random sampling for many times. The designed spectrogram abnormal sample point detection method based on the random sampling agree set, provided by the invention, has the advantages of being rapid, effective, high in accuracy and wide in application range.

Description

Spectrogram exceptional sample point detecting method based on the consistent collection of stochastic sampling

Technical field

The present invention relates to Chemical Measurement Multivariate Correction model data processing technology field, particularly a kind of spectrogram exceptional sample point detecting method based on the consistent collection of stochastic sampling.

Background technology

Along with the development of Modern Analytical Instrument, detection signal has been complete spectrogram by traditional single numerical value change, or even image.For spectroscopic data, dimension is normally very high with respect to the number of samples that gathers, and proofread and correct the regression problem Very Ill-conditioned this moment, and traditional monobasic single argument bearing calibration is difficult to these data analysis be the substitute is multivariate calibration methods ^[1]Chemical Measurement Multivariate Correction technology is directly utilized measuring-signal, sets up quantitative model between spectral signal and the sample concentration by dimensionality reduction, feature extraction, eigentransformation and multiple regression technology, to realize quantitative test.

Yet classical multivariate calibration methods is such as multiple linear regression, principal component regression, partial least squares regression ^[2-3]Usually be subject to especially the impact of exceptional sample point.Usually, compare with the most of sample of data centralization, exceptional sample be exactly have nothing to do or be wrong and abnormal sample in a way.Exceptional sample is generally caused by reasons such as instrument failure, acquisition condition factor, manual operation error or data self-defects.The existence of exceptional sample can affect the quality of model, and the model that causes setting up can't reflect the true relation of data, can't be predicted the outcome accurately.Therefore, need the impact of rejecting abnormalities sample point to set up sane model ^[4]

For principal component regression, generally adopt sane covariance to estimate to substitute traditional data covariance matrix, thereby realize sane principal component regression.Return for offset minimum binary (PLS), different sane PLS models are suggested, as with least square regression method involved in the PLS method, partly or entirely replace to certain robustness regression method, such as iteration heavy weighted least-squares (IRLS), minimum median quadratic method (LMS) and truncation least square method (LTS) etc.; Iteration is weighting offset minimum binary (IRPLS) method heavily ^[5]Inclined to one side robust M homing method ^[6]The RSIMPLS method ^[7]

Also have class methods to detect exceptional sample by cross validation, as based on staying a cross validation to obtain spectrum residual error corresponding to each sample or concentration residual error, judge that then the sample that residual error exceeds certain threshold value is exceptional sample ^[8]Similarly, model's Caro cross validation also is used to exceptional sample and detects, the method model model Caro cross validation model, then sort according to Prediction sum squares, and add up the frequency of occurrence of each sample in different models, final anomaly-based sample judges with the frequency of occurrence difference of normal sample whether sample is unusual.

Yet, based on the exceptional sample detection method of cross validation, may produce " covering " phenomenon, cause detecting or the wrong identification exceptional sample.It is relatively poor to detect effect when sane principal component regression or partial least squares regression are more for the data centralization exceptional sample.Unanimously collect based on the maximum a posteriori probability stochastic sampling ^[9], carry out the Multivariate Correction exceptional sample and detect, be a kind of new method, it can pass through constantly stochastic sampling, rejects the exceptional sample in the data, yet there are no proven technique and document.Various complicated cases in the real world applications such as observation condition, operation factors etc., all can cause the appearance of exceptional sample point.Various dissimilar exceptional sample points are different to the influence degree of calibration model, and impact how effectively to eliminate these exceptional samples is a difficult problem of Chemical Measurement Multivariate Correction technology.

[1]Martens?H,Nas?T.Multivariate?calibration.Wiley,1992

[2]Wold?H.Soft?modelling?by?latent?variables:the?nonlineariterative?partial?least?squares?approach.Perspectives?in?Probability?andStatistics.London:Academic?Press,1975

[3]de?Jong?S.SIMPLS:an?alternative?approach?squares?regression?topartial?least?squares?regression.Chemometrics?and?IntelligentLaboratory?Systems，1993,18(3),251-263

[4]Liang?Y?Z,Kvalheim?O?M.Robust?methods?for?multivariate?analysis-a?tutorial?review.Chemometrics?and?Intelligent?Laboratory?Systems，1996,32(1),1-10

[5]Cummins?D?J,Andrews?C?W.Iteratively?reweighted?partial?leastsquares:A?performance?analysis?by?monte?carlo?simulation.Journal?ofChemometrics，1995,9(6),489-507

[6]Serneels?S,Croux?C,Filzmoser?P,et?al.Partial?robust?Mregression.Chemometrics?and?Intelligent?Laboratory?Systems，2005,79(1-2),55-64

[7]Hubert?M,Branden?K.Robust?methods?for?partial?least?squaresregression.Journal?of?Chemometrics，2003,17,537-549

[8]Koshoubu?J,Iwata?T,Minami?S.Elimination?of?the?uninformativecalibration?sample?subset?in?the?modified?UVE-PLS?method.AnalyticalSciences，2001,17(2),319-322

[9]Torr?P.Bayesian?Model?Estimation?and?Selection?for?EpipolarGeometry?and?Generic?Manifold?Fitting.International?Journal?of?ComputerVision，2002,50(1),35-61

Summary of the invention

Technical matters to be solved by this invention provides a kind of fast effective, accuracy spectrogram exceptional sample point detecting method based on the consistent collection of stochastic sampling high with applied widely.

The present invention has designed following technical scheme in order to solve the problems of the technologies described above: the present invention has designed a kind of spectrogram exceptional sample point detecting method based on the consistent collection of stochastic sampling, comprises following concrete steps:

Step (1): given spectroscopic data X is carried out sane principal component analysis (PCA), detect and elimination exceptional spectrum sample point, obtain calibration samples collection X _c, note calibration samples collection X _cMiddle number of samples is m _c

Step (2): the calibration samples collection X in described step (1) _cOn carry out stochastic sampling, obtain current training set X _s

Step (3): based on the training set X in the described step (2) _sSet up the Multivariate Correction model, and computation model prediction residual error E _s

Step (4): utilize Multivariate Correction model and model prediction residual error E in the step (3) _s, the performance of evaluation model also draws the evaluation score, and with the calibration samples collection X in the step (1) _cBe defined as interior point set u _c

Step (5): repeating step (2) is to step (4) N time, and wherein N is defined as natural number, estimates score thereby obtain N, and selecting wherein to estimate the corresponding calibration samples collection of the most much higher first calibration model of score is final interior point set u _m

As a kind of optimization method of the present invention: described step (1) comprises following concrete steps:

Step (11): set up model X=TP ^T, T[t wherein ₁, t ₂..., t _a] ^TBe defined as score matrix, P[p ₁, p ₂..., p _a] ^TBe defined as loading matrix, a is defined as the major component number;

Step (12): utilize formula t _Median=median (t ₁, t ₂... t _a) calculating principal component scores vector t ₁, t ₂..., t _aIntermediate value t _Median

Step (13): based on the intermediate value t in the step (12) _MedianAnd following formula

s _mad=1.4826median(|t ₁-t _median|,|t ₂-t _median|,…|t _a-t _median|)

Calculate the intermediate value absolute deviation s of data _Mad

Step (14): utilize formula d _i=| t _i-t _Median| calculate the error amount d between each principal component scores data and the intermediate value _i, i=1 wherein ..., m _c, reject d _i〉=3 * s _MadSample point, the data set that obtains is calibration samples collection X _c

As a kind of optimization method of the present invention: described step (2) comprises following concrete processing:

At calibration set X _cOn carry out stochastic sampling, pick out randomly m=m _c/ 2 samples, wherein, m is defined as positive even numbers, forms sample set

As current training set.

As a kind of optimization method of the present invention: described step (3) comprises following concrete processing:

Set up concentration value Multivariate Correction model Y _s=X _sB, and utilize formula

Calculate the prediction residual error E of all samples _s, wherein, i=1 ..., m _c,

Be defined as the actual concentration value,

Be defined as model predication value, B is defined as Optimal Parameters.

As a kind of optimization method of the present invention: described step (4) comprises following concrete processing:

Step (41): utilize formula ${\mathrm{\θ}}^{*}=\arg \max P\left(\mathrm{\θ}\right|e,I)=\arg \max \frac{P\left(e\right|\mathrm{\θ},I)P\left(\mathrm{\θ}\right|I)}{P\left(e\right|I)}$ Calculate the maximum a posteriori probability of Multivariate Correction model parameter θ, wherein, I is defined as the probability distribution of data, and the posterior probability of θ is expressed as: $P\left(\mathrm{\θ}\right|e,I)\∝{\mathrm{\Π}}_{i=1}^{m_c}({\mathrm{\γ}}_i\frac{1}{\sqrt{2\mathrm{\π\σ}}}\exp (-\frac{e_i^2}{2{\mathrm{\σ}}^2})+(1-{\mathrm{\γ}}_i)\frac{1}{v})P\left(\mathrm{\θ}\right|I),$ Wherein γ is defined as the prior probability of interior point, and v is defined as the size of the error space;

Step (42): maximization $P\left(\mathrm{\θ}\right|e,I)\∝{\mathrm{\Π}}_{i=1}^{m_c}({\mathrm{\γ}}_i\frac{1}{\sqrt{2\mathrm{\π\σ}}}\exp (-\frac{e_i^2}{2{\mathrm{\σ}}^2})+(1-{\mathrm{\γ}}_i)\frac{1}{v})P\left(\mathrm{\θ}\right|I)$ Be equivalent to and minimize following objective function:

Step (43): minimize above-mentioned objective function $c={\mathrm{\Σ}}_i\mathrm{\ρ}\left(e_i^2\right),\mathrm{\ρ}\left(e_i^2\right)=\left\{\begin{array}{c}{e}_i\mathrm{\σ},e_i/{\mathrm{\σ}}^2\≤T\\ \sqrt{T}{\mathrm{\σ}}^2,e_i/{\mathrm{\σ}}^2>T\end{array}\right.,$ Thereby obtain current optimum model parameter θ ^*, and the likelihood function value of interior point, exterior point if an exterior point likelihood function value corresponding to sample is put the likelihood function value greatly in the inner, judges that then this sample is the exceptional sample point, eliminate these exceptional sample points after, determine corresponding interior point set u _c

The present invention compared with prior art has following advantage:

1. the designed spectrogram exceptional sample point detecting method based on the consistent collection of stochastic sampling of the present invention is simple to operate, convenient and swift, need not to set in advance parameter;

2. the designed spectrogram exceptional sample point detecting method based on the consistent collection of stochastic sampling of the present invention passes through repeatedly stochastic sampling, carries out model evaluation and determines consistent sample set, and no matter whether data centralization has exceptional sample, and algorithm all can be obtained satisfied performance; Especially, when the data centralization exceptional sample was more, algorithm can be rejected these exceptional samples effectively by repeatedly screening, improves the robustness of follow-up Multivariate Correction model;

3. the designed spectrogram exceptional sample point detecting method based on the consistent collection of stochastic sampling of the present invention can be processed the exceptional sample point of number of different types, has stronger practicality.

Description of drawings

Fig. 1 is the designed process flow diagram based on the consistent spectrogram exceptional sample point detecting method that collects of stochastic sampling of the present invention.

Embodiment

The present invention is described in further detail below in conjunction with accompanying drawing:

As shown in Figure 1, the present invention has designed a kind of spectrogram exceptional sample point detecting method based on the consistent collection of stochastic sampling, comprises following concrete steps:

Step (1): given spectroscopic data X is carried out sane principal component analysis (PCA), detect and elimination exceptional spectrum sample point, obtain calibration samples collection X _c, note calibration samples collection X _cMiddle number of samples is m _c

Step (2): the calibration samples collection X in described step (1) _cOn carry out stochastic sampling, obtain current training set X _s

Step (3): based on the training set X in the described step (2) _sSet up the Multivariate Correction model, and computation model prediction residual error E _s

Step (4): utilize Multivariate Correction model and model prediction residual error E in the step (3) _s, the performance of evaluation model also draws the evaluation score, and with the calibration samples collection X in the step (1) _cBe defined as interior point set u _c

Step (5): repeating step (2) is to step (4) N time, and wherein N is defined as natural number, estimates score thereby obtain N, and selecting wherein to estimate the corresponding calibration samples collection of the most much higher first calibration model of score is final interior point set u _m

As a kind of optimization method of the present invention: described step (1) comprises following concrete steps:

Step (11): set up model X=TP ^T, T[t wherein ₁, t ₂..., t _a] ^TBe defined as score matrix, P[p ₁, p ₂..., p _a] ^TBe defined as loading matrix, a is defined as the major component number;

Step (12): utilize formula t _Median=median (t ₁, t ₂... t _a) calculating principal component scores vector t ₁, t ₂..., t _aIntermediate value t _Median

Step (13): based on the intermediate value t in the step (12) _MedianAnd following formula

s _mad=1.4826median(|t ₁-t _median|,|t ₂-t _median|,…|t _a-t _median|)

Calculate the intermediate value absolute deviation s of data _Mad

Step (14): utilize formula d _i=| t _i-t _Median| calculate the error amount d between each principal component scores data and the intermediate value _i, i=1 wherein ..., m _c, reject d _i〉=3 * s _MadSample point, the data set that obtains is calibration samples collection X _c

As a kind of optimization method of the present invention: described step (2) comprises following concrete processing:

At calibration set X _cOn carry out stochastic sampling, pick out randomly m=m _c/ 2 samples, wherein, m is defined as positive even numbers, forms sample set

As current training set.

As a kind of optimization method of the present invention: described step (3) comprises following concrete processing:

Set up concentration value Multivariate Correction model Y _s=X _sB, and utilize formula

Calculate the prediction residual error E of all samples _s, wherein, i=1 ..., m _c, Be defined as the actual concentration value, Be defined as model predication value, B is defined as Optimal Parameters.

As a kind of optimization method of the present invention: described step (4) comprises following concrete processing:

Step (41): utilize formula ${\mathrm{\θ}}^{*}=\arg \max P\left(\mathrm{\θ}\right|e,I)=\arg \max \frac{P\left(e\right|\mathrm{\θ},I)P\left(\mathrm{\θ}\right|I)}{P\left(e\right|I)}$ Calculate the maximum a posteriori probability of Multivariate Correction model parameter θ, wherein, I is defined as the probability distribution of data, and the posterior probability of θ is expressed as: $P\left(\mathrm{\θ}\right|e,I)\∝{\mathrm{\Π}}_{i=1}^{m_c}({\mathrm{\γ}}_i\frac{1}{\sqrt{2\mathrm{\π\σ}}}\exp (-\frac{e_i^2}{2{\mathrm{\σ}}^2})+(1-{\mathrm{\γ}}_i)\frac{1}{v})P\left(\mathrm{\θ}\right|I),$ Wherein γ is defined as the prior probability of interior point, and v is defined as the size of the error space;

Step (42): maximization $P\left(\mathrm{\θ}\right|e,I)\∝{\mathrm{\Π}}_{i=1}^{m_c}({\mathrm{\γ}}_i\frac{1}{\sqrt{2\mathrm{\π\σ}}}\exp (-\frac{e_i^2}{2{\mathrm{\σ}}^2})+(1-{\mathrm{\γ}}_i)\frac{1}{v})P\left(\mathrm{\θ}\right|I)$ Be equivalent to and minimize following objective function:

Step (43): minimize above-mentioned objective function $c={\mathrm{\Σ}}_i\mathrm{\ρ}\left(e_i^2\right),\mathrm{\ρ}\left(e_i^2\right)=\left\{\begin{array}{c}{e}_i\mathrm{\σ},e_i/{\mathrm{\σ}}^2\≤T\\ \sqrt{T}{\mathrm{\σ}}^2,e_i/{\mathrm{\σ}}^2>T\end{array}\right.,$ Thereby obtain current optimum model parameter θ ^*, and the likelihood function value of interior point, exterior point if an exterior point likelihood function value corresponding to sample is put the likelihood function value greatly in the inner, judges that then this sample is the exceptional sample point, eliminate these exceptional sample points after, determine corresponding interior point set u _c

Described sane principal component analysis (PCA) detects the exceptional sample point and refers to given spectroscopic data, carries out principal component analysis (PCA) and obtains the principal component scores matrix; For this principal component scores data matrix, use sane 3 ^σPrinciple detects the exceptional sample point.Described principal component scores matrix refers to choose the data matrix that score vector corresponding to front several major component forms.Describedly choose front several major component, the determining of major component number generally determined according to cumulative proportion in ANOVA, also can specify in advance the major component number, as choose 10 or 20 major components.

Described sane 3 ^σPrinciple refers at first calculate the intermediate value of principal component scores vector, and based on the intermediate value absolute deviation of these median calculation data; Then calculate the error between each principal component scores vector and the intermediate value, be the exceptional sample point if this error amount, is then judged this sample greater than 3 times of the intermediate value absolute deviation.

Described stochastic sampling refers on the calibration samples collection that ascertains the number, and half the sample of picking out randomly the calibration set number forms sample set, as current training set.

Described Multivariate Correction model can be multiple linear regression model, principal component regression model, partial least square model or other nonlinear model.The performance of described evaluation model, refer to according to model prediction residual error, based on Bayes principle, posterior probability corresponding to maximization model parameter, can be converted into and minimize an objective function, thereby obtain current optimum model parameter, and the likelihood function value of interior point, exterior point, if an exterior point likelihood function value corresponding to sample is put the likelihood function value greatly in the inner, judge that then this sample is the exceptional sample point.After eliminating these exceptional sample points, determine the corresponding consistent sample set of interior point.

N stochastic sampling of described repetition and model evaluation process select to estimate the highest model of score, and determine that corresponding sample set is final interior point set, refer to stochastic sampling, set up Multivariate Correction model, these three steps repetitions of model evaluation N time; In this N time model evaluation, selecting the solution of posterior probability values maximum (value of objective function C is minimum) is final solution, and determines that interior corresponding sample set is final interior point set.

Above-described specific embodiment; purpose of the present invention, technical scheme and beneficial effect are further described; institute is understood that; the above only is specific embodiments of the invention; be not limited to the present invention; within the spirit and principles in the present invention all, any modification of making, be equal to replacement, improvement etc., all should be included within protection scope of the present invention.

Claims (5)

1. the spectrogram exceptional sample point detecting method based on the consistent collection of stochastic sampling is characterized in that, comprises following concrete steps:

Step (1): given spectroscopic data X is carried out sane principal component analysis (PCA), detect and elimination exceptional spectrum sample point, obtain calibration samples collection X _c, note calibration samples collection X _cMiddle number of samples is m _c

Step (2): the calibration samples collection X in described step (1) _cOn carry out stochastic sampling, obtain current training set X _s

Step (3): based on the training set X in the described step (2) _sSet up the Multivariate Correction model, and computation model prediction residual error E _s

Step (4): utilize Multivariate Correction model and model prediction residual error E in the step (3) _s, the performance of evaluation model also draws the evaluation score, and with the calibration samples collection X in the step (1) _cBe defined as interior point set u _c

Step (5): repeating step (2) is to step (4) N time, and wherein N is defined as natural number, estimates score thereby obtain N, and selecting wherein to estimate the corresponding calibration samples collection of the most much higher first calibration model of score is final interior point set u _m

2. according to claim 1 based on the consistent spectrogram exceptional sample point detecting method that collects of stochastic sampling, it is characterized in that described step (1) comprises following concrete steps:

Step (11): set up model X=TP ^T, T[t wherein ₁, t ₂..., t _a] ^TBe defined as score matrix, P[p ₁, p ₂..., p _a] ^TBe defined as loading matrix, a is defined as the major component number;

Step (12): utilize formula t _Median=median (t ₁, t ₂... t _a) calculating principal component scores vector t ₁, t ₂..., t _aIntermediate value t _Median

Step (13): based on the intermediate value t in the step (12) _MedianAnd following formula

s _mad=1.4826median(|t ₁-t _median|,|t ₂-t _median|,…|t _a-t _median|)

Calculate the intermediate value absolute deviation s of data _Mad

Step (14): utilize formula d _i=| t _i-t _Median| calculate the error amount d between each principal component scores data and the intermediate value _i, i=1 wherein ..., m _c, reject d _i〉=3 * s _MadSample point, the data set that obtains is calibration samples collection X _c

3. according to claim 2 based on the consistent spectrogram exceptional sample point detecting method that collects of stochastic sampling, it is characterized in that described step (2) comprises following concrete processing:

At calibration set X _cOn carry out stochastic sampling, pick out randomly m=m _c/ 2 samples, wherein, m is defined as positive even numbers, forms sample set

As current training set.

4. according to claim 3 based on the consistent spectrogram exceptional sample point detecting method that collects of stochastic sampling, it is characterized in that described step (3) comprises following concrete processing:

Set up concentration value Multivariate Correction model Y _s=X _sB, and utilize formula

Calculate the prediction residual error E of all samples _s, wherein, i=1 ..., m _c,

Be defined as the actual concentration value,

Be defined as model predication value, B is defined as Optimal Parameters.

5. according to claim 4 based on the consistent spectrogram exceptional sample point detecting method that collects of stochastic sampling, it is characterized in that described step (4) comprises following concrete processing:

Step (41): utilize formula ${\mathrm{\θ}}^{*}=\arg \max P\left(\mathrm{\θ}\right|e,I)=\arg \max \frac{P\left(e\right|\mathrm{\θ},I)P\left(\mathrm{\θ}\right|I)}{P\left(e\right|I)}$ Calculate the maximum a posteriori probability of Multivariate Correction model parameter θ, wherein, I is defined as the probability distribution of data, and the posterior probability of θ is expressed as: $P\left(\mathrm{\θ}\right|e,I)\∝{\mathrm{\Π}}_{i=1}^{m_c}({\mathrm{\γ}}_i\frac{1}{\sqrt{2\mathrm{\π\σ}}}\exp (-\frac{e_i^2}{2{\mathrm{\σ}}^2})+(1-{\mathrm{\γ}}_i)\frac{1}{v})P\left(\mathrm{\θ}\right|I),$ Wherein γ is defined as the prior probability of interior point, and v is defined as the size of the error space;

Step (42): maximization $P\left(\mathrm{\θ}\right|e,I)\∝{\mathrm{\Π}}_{i=1}^{m_c}({\mathrm{\γ}}_i\frac{1}{\sqrt{2\mathrm{\π\σ}}}\exp (-\frac{e_i^2}{2{\mathrm{\σ}}^2})+(1-{\mathrm{\γ}}_i)\frac{1}{v})P\left(\mathrm{\θ}\right|I)$ Be equivalent to and minimize following objective function:

Step (43): minimize above-mentioned objective function $c={\mathrm{\Σ}}_i\mathrm{\ρ}\left(e_i^2\right),\mathrm{\ρ}\left(e_i^2\right)=\left\{\begin{array}{c}{e}_i\mathrm{\σ},e_i/{\mathrm{\σ}}^2\≤T\\ \sqrt{T}{\mathrm{\σ}}^2,e_i/{\mathrm{\σ}}^2>T\end{array}\right.,$ Thereby obtain current optimum model parameter θ ^*, and the likelihood function value of interior point, exterior point if an exterior point likelihood function value corresponding to sample is put the likelihood function value greatly in the inner, judges that then this sample is the exceptional sample point, eliminate these exceptional sample points after, determine corresponding interior point set u _c

Images