CN112069037A - Method and device for detecting no threshold value of cloud platform - Google Patents
Method and device for detecting no threshold value of cloud platform Download PDFInfo
- Publication number
- CN112069037A CN112069037A CN202010851477.6A CN202010851477A CN112069037A CN 112069037 A CN112069037 A CN 112069037A CN 202010851477 A CN202010851477 A CN 202010851477A CN 112069037 A CN112069037 A CN 112069037A
- Authority
- CN
- China
- Prior art keywords
- index data
- abnormal
- model
- data
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000001514 detection method Methods 0.000 claims abstract description 141
- 230000002159 abnormal effect Effects 0.000 claims abstract description 127
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 33
- 230000005856 abnormality Effects 0.000 claims abstract description 11
- 238000012549 training Methods 0.000 claims description 42
- 230000004044 response Effects 0.000 claims description 25
- 230000002547 anomalous effect Effects 0.000 claims description 3
- 238000002360 preparation method Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 5
- 238000012545 processing Methods 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000009499 grossing Methods 0.000 description 2
- YHXISWVBGDMDLQ-UHFFFAOYSA-N moclobemide Chemical compound C1=CC(Cl)=CC=C1C(=O)NCCN1CCOCC1 YHXISWVBGDMDLQ-UHFFFAOYSA-N 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 238000012896 Statistical algorithm Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Testing And Monitoring For Control Systems (AREA)
Abstract
The invention discloses a method for detecting a cloud platform without a threshold value, which comprises the following steps: acquiring index data to be detected, and respectively establishing models through a plurality of anomaly detection algorithms; respectively carrying out unsupervised anomaly detection on index data to be detected according to the models; when any model obtains abnormal index data from index data to be detected through unsupervised abnormal detection, judging whether a supervised model trained through corresponding historical index data exists according to the abnormal index data; when the unsupervised model is judged to be unsupervised, integrating all unsupervised abnormal detection results to judge whether the index data to be detected is abnormal; and when the presence of the supervision model is judged, carrying out supervision abnormality detection on the abnormal index data according to the presence of the supervision model, and judging whether the index data to be detected is abnormal or not according to the result of the supervision abnormality detection. The invention also discloses a corresponding device. The invention realizes the method for joint detection of the unsupervised model and the supervised model, and improves the accuracy of the non-threshold detection result.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for non-threshold detection of a cloud platform.
Background
The OpenStack is an open-source cloud computing management platform project and is a combination of a series of software open-source projects. The project aims to provide a cloud computing management platform which is simple to implement, can be expanded in a large scale, is rich and has a unified standard.
As OpenStack matures, the size of the OpenStack-based cloud platform is expanding from the first few, tens of, to hundreds or even thousands of platforms. With the increasing types and amounts of monitoring data, the alarm rules become more and more complex.
Performance data (memory usage, system usage of a CPU (Central Processing Unit), inflow speed of a network, and the like) of the cloud platform is time-series data of a single value. The alarm of the traditional cloud platform is basically a threshold alarm, wherein the threshold alarm is that a user sets an alarm rule according to an empirical value, and when monitoring data reaches the threshold, an alarm is generated. However, in the actual application process, the threshold setting is too dependent on experience, the threshold is too high, many alarms are missed, the hidden quality trouble is difficult to find, the threshold is too low, too many alarms cause an alarm storm, and the judgment of service operation and maintenance personnel is interfered; in addition, for the condition of some resource jitters, threshold value alarm cannot be detected, and false alarm can be generated.
Disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide a method and an apparatus for cloud platform threshold-free detection. The non-threshold value alarm is used as the supplement of the threshold value alarm, and the accuracy of the non-threshold value detection result is greatly improved by adopting a method of selecting a plurality of model results.
Based on the above purpose, an aspect of the present invention provides a method for cloud platform non-threshold detection, where the method includes:
acquiring index data to be detected, and respectively establishing models through a plurality of anomaly detection algorithms;
respectively carrying out unsupervised anomaly detection on index data to be detected according to the models;
acquiring abnormal index data from index data to be detected by responding to any model through unsupervised abnormal detection, and judging whether a supervised model trained through corresponding historical index data exists or not according to the abnormal index data;
responding to the unsupervised model, and integrating all unsupervised abnormal detection results to judge whether the index data to be detected is abnormal;
and responding to the presence of the supervision model, carrying out supervision abnormality detection on the abnormal index data according to the supervision model, and judging whether the index data to be detected is abnormal or not according to the result of the supervision abnormality detection.
In some embodiments of the cloud platform threshold-free detection methods of the present invention, the methods further comprise:
acquiring historical index data to be calibrated from a time sequence database;
detecting abnormal groups in the historical index data, and calibrating abnormal index data in the abnormal groups;
and storing the abnormal index data into a relational database.
In some embodiments of the cloud platform threshold-free detection methods of the present invention, the methods further comprise:
acquiring index data of a supervised model;
extracting characteristic data of the index data;
forming a supervised model according to the extracted feature data, and training the supervised model;
judging whether the trained supervised model meets expectations or not;
in response to determining that the supervised model is in agreement with the expectation, a model file of the trained supervised model is output.
In some embodiments of the cloud platform non-threshold detection method of the present invention, in response to the unsupervised model, the integrating results of all unsupervised anomaly detections to determine whether the index data to be detected is anomalous further includes:
judging certain index data to be detected as abnormal index data in response to half or more than half of all abnormal detection algorithms, and judging the index data to be detected as abnormal data;
and judging a certain index data to be detected as abnormal index data in response to less than half of all the abnormal detection algorithms, and judging the index data to be detected as normal data.
In some embodiments of the cloud platform non-threshold detection method of the present invention, performing supervised anomaly detection on the anomaly indicator data according to the supervised model further comprises:
loading a model file of a supervision model;
extracting characteristic data of the abnormal index data;
and inputting the characteristic data extracted according to the abnormal index data into the model file, and acquiring the detection result of the abnormal index data according to the supervised model.
In some embodiments of the method for cloud platform non-threshold detection of the present invention, forming a supervised model from the extracted feature data, and training the supervised model further comprises:
and in response to the incremental training of the trained supervised model, loading a model file in advance, extracting characteristic data of the incremental training data, and retraining the supervised model according to the characteristic data extracted from the index data and the characteristic data extracted from the incremental training data in the model file.
In another aspect of the embodiments of the present invention, there is also provided a device for detecting a cloud platform without a threshold, where the device includes:
the data preparation module is configured to obtain index data to be detected and respectively establish a model through a plurality of anomaly detection algorithms;
the unsupervised anomaly detection module is configured to perform unsupervised anomaly detection on the index data to be detected respectively according to the models;
the abnormal index data judging module is configured to respond to the fact that any model acquires abnormal index data from index data to be detected through unsupervised abnormal detection, and judge whether a supervised model trained through corresponding historical index data exists according to the abnormal index data;
the unsupervised anomaly detection result module is configured to respond to the unsupervised model and synthesize all unsupervised anomaly detection results to judge whether the index data to be detected is abnormal or not;
and the supervised anomaly detection result module is configured to respond to the supervised model, carry out supervised anomaly detection on the anomaly index data according to the supervised model, and judge whether the index data to be detected is abnormal or not according to the result of the supervised anomaly detection.
In some embodiments of the apparatus for cloud platform threshold-free detection of the present invention, the apparatus further comprises:
the abnormal index data configuration module is configured to acquire historical index data to be calibrated from a time sequence database; detecting abnormal groups in the historical index data, and calibrating abnormal index data in the abnormal groups; and storing the abnormal index data into a relational database.
In some embodiments of the apparatus for cloud platform threshold-free detection of the present invention, the apparatus further comprises:
the supervised model training module is configured to acquire index data of the supervised model; extracting characteristic data of the index data; forming a supervised model according to the extracted feature data, and training the supervised model; judging whether the trained supervised model meets expectations or not; in response to determining that the supervised model is in agreement with the expectation, a model file of the trained supervised model is output.
In some embodiments of the apparatus for cloud platform non-threshold detection of the present invention, the unsupervised anomaly detection result module is further configured to:
judging certain index data to be detected as abnormal index data in response to half or more than half of all abnormal detection algorithms, and judging the index data to be detected as abnormal data;
and judging a certain index data to be detected as abnormal index data in response to less than half of all the abnormal detection algorithms, and judging the index data to be detected as normal data.
The invention has at least the following beneficial technical effects: the method realizes the joint detection of the unsupervised model and the supervised model. Aiming at the time sequence performance data of the cloud platform, the method not only exerts the advantages of high unsupervised model construction speed, less manual participation and high supervised model accuracy, but also overcomes the defects of low unsupervised model accuracy, complex supervised model training, long time and the like, and greatly improves the comprehensiveness and accuracy of cloud platform alarm.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 shows a schematic block diagram of an embodiment of a method of cloud platform threshold-free detection in accordance with the present invention;
FIG. 2 is a schematic diagram illustrating the determination of abnormal data by the central limit theorem according to an embodiment of the method for cloud platform threshold-free detection of the present invention;
FIG. 3 illustrates a flow diagram of an embodiment of a method of cloud platform threshold-free detection in accordance with the present invention;
FIG. 4 illustrates an indicator calibration flow diagram of an embodiment of a method of cloud platform threshold-free detection in accordance with the present invention;
FIG. 5 illustrates a supervised training flow diagram of an embodiment of a method for cloud platform threshold-free detection in accordance with the present invention;
FIG. 6 shows a schematic block diagram of an embodiment of an apparatus for cloud platform threshold-free detection in accordance with the present invention;
fig. 7 is a flowchart illustrating an event process of an embodiment of the apparatus for cloud platform threshold-free detection according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it is understood that "first" and "second" are only used for convenience of description and should not be construed as limiting the embodiments of the present invention, and the descriptions thereof in the following embodiments are omitted.
In view of the foregoing, a first aspect of the embodiments of the present invention provides an embodiment of a method for cloud platform threshold-free detection. Fig. 1 is a schematic block diagram of an embodiment of a method for non-threshold detection of a cloud platform according to the present invention. In the embodiment shown in fig. 1, the method comprises at least the following steps:
s100, acquiring index data to be detected, and respectively establishing models through a plurality of anomaly detection algorithms;
s200, respectively carrying out unsupervised anomaly detection on index data to be detected according to the models;
s300, responding to any model, acquiring abnormal index data from index data to be detected through unsupervised abnormal detection, and judging whether a supervised model trained through corresponding historical index data exists or not according to the abnormal index data;
s400, responding to the unsupervised model, and integrating all unsupervised abnormal detection results to judge whether the index data to be detected is abnormal;
s500, responding to the supervised model, carrying out supervised anomaly detection on abnormal index data according to the supervised model, and judging whether the index data to be detected is abnormal or not according to the result of the supervised anomaly detection.
In some embodiments of the present invention, on the basis of large-scale practice, a method for OpenStack platform threshold-free detection is proposed. The invention realizes the method for joint detection of the unsupervised model and the supervised model. In practical application, aiming at performance data of a cloud platform, the method plays the advantages of high unsupervised model construction speed, less manual participation and high accuracy of a supervised model, particularly a neural network model, and overcomes the defects of low unsupervised model accuracy, complex supervised model training, long time and the like.
In some embodiments of the invention, the unsupervised model employs a statistical method-based model, a machine learning method-based model. The unsupervised algorithm detects anomalies according to the 3-sigma rule. The 3-sigma rule is also called as an empirical rule, when the number of times of extraction is sufficiently large, the probability distribution that the group of data is approximately in normal distribution is considered, and for the data which is not in normal distribution, the rule is still applicable according to the central limit theorem. Fig. 2 shows a schematic diagram of the method for cloud platform threshold-free detection according to the present invention for determining abnormal data by the central limit theorem, as shown in fig. 2, the data within 3 standard deviations is about 99.7 in proportion, and the data outside is considered as abnormal data.
In some embodiments of the present invention, supervised models may employ xgboost, BP ANN, LSTM, etc. models. These models require a large amount of training data and are complex and time consuming to train. In a normally operating cloud platform, the number of abnormal points is small, the workload of selection of training data and manual calibration is huge, an unsupervised model is added to detect abnormal groups for manual calibration, and in addition, when no supervised model exists, the result of the unsupervised model is adopted as an output result.
In some embodiments of the present invention, fig. 3 is a flowchart illustrating an embodiment of a cloud platform non-threshold detection method according to the present invention, and as shown in fig. 3, when index data is input into a system, unsupervised detection is first performed, a model is established in real time through an anomaly detection algorithm such as a statistics-based algorithm, an exponential smoothing algorithm, an arima algorithm, an isolated forest algorithm, and the like, and non-threshold anomaly detection is performed according to a 3-sigma rule. If the result is returned without abnormality, returning to normal and ending the process; if the result has an abnormal point (namely abnormal index data) to return, judging whether the index has a supervised model trained by historical data, if not, integrating unsupervised abnormal detection results and outputting the unsupervised abnormal detection results as a final result; if the supervision model exists, the index data is input into the supervision model for detection, and the result of the supervision abnormal detection is output as a final result.
According to some embodiments of the method of cloud platform threshold-free detection of the present invention, the method further comprises:
acquiring historical index data to be calibrated from a time sequence database;
detecting abnormal groups in the historical index data, and calibrating abnormal index data in the abnormal groups;
and storing the abnormal index data into a relational database.
In some embodiments of the present invention, the assistant personnel performs data calibration, and may use automatic calibration as an assistant to manually confirm as a main. Fig. 4 is a flowchart illustrating index calibration according to an embodiment of the cloud platform threshold-free detection method of the present invention, and as shown in fig. 4, the method includes the following steps: and acquiring historical time sequence index data needing to be calibrated from a time sequence database. The statistical method detects abnormal groups and manually calibrates abnormal index data of the abnormal groups. And storing the result of the calibrated abnormal index data into a relational database for backtracking. Wherein, the training can be directly carried out after the data calibration.
According to some embodiments of the method of cloud platform threshold-free detection of the present invention, the method further comprises:
acquiring index data of a supervised model;
extracting characteristic data of the index data;
forming a supervised model according to the extracted feature data, and training the supervised model;
judging whether the trained supervised model meets expectations or not;
in response to determining that the supervised model is in agreement with the expectation, a model file of the trained supervised model is output.
In some embodiments of the present invention, the model training refers to training an xgboost, lstm, and other models, fig. 5 is a flow chart illustrating supervised training of an embodiment of the cloud platform threshold-free detection method according to the present invention, and the processing flow is as shown in fig. 5, and specifically includes the following steps: preparing data, wherein the related data is index data which comprises calibrated data or calibrated historical data in a relational database; extracting characteristics, namely extracting the characteristics of the index data, wherein the extracted characteristics comprise statistical characteristics, fitting characteristics, classification characteristics and the like; and (4) model training, namely sending the characteristic data into a training tree integrated model for model training. Evaluating the model, and if the model is in accordance with expectation, setting a completion flag bit and outputting a trained model file; and if the expectation is not met, the follow-up training is waited.
According to some embodiments of the cloud platform non-threshold detection method, in response to the unsupervised model, the step of integrating all unsupervised anomaly detection results to determine whether the index data to be detected is abnormal further includes:
judging certain index data to be detected as abnormal index data in response to half or more than half of all abnormal detection algorithms, and judging the index data to be detected as abnormal data;
and judging a certain index data to be detected as abnormal index data in response to less than half of all the abnormal detection algorithms, and judging the index data to be detected as normal data.
In some embodiments of the invention, the detected abnormal index data is directly given based on algorithms such as a statistical algorithm, an exponential smoothing algorithm, an arima algorithm, an isolated forest algorithm and the like, and the steps are as follows: accessing index data to be detected into a system; calculating parameters of each model, and detecting data by using an anomaly detection algorithm of an unsupervised model; aiming at index data, each method has output detection results, the results are integrated, when most of the results are judged to be abnormal at a certain point, the final result is an abnormal point, otherwise, the final result is a normal point; and outputting a final abnormal detection result.
According to some embodiments of the method for cloud platform non-threshold detection, the performing supervised anomaly detection on the anomaly indicator data according to the supervised model further comprises:
loading a model file of a supervision model;
extracting characteristic data of the abnormal index data;
and inputting the characteristic data extracted from the root abnormal index data into a model file, and acquiring a detection result of the abnormal index data according to the supervised model.
In some embodiments of the present invention, the step of detecting the target data to be detected by using the existing supervised model includes: accessing index data to be detected into a system; loading a model file of a supervision model; extracting data from the index data to be detected according to characteristics including statistical characteristics, fitting characteristics, classification characteristics and the like; and (4) entering each characteristic data of the index data to be detected into the supervised model, and outputting the detection result of the supervised model.
According to some embodiments of the method for cloud platform non-threshold detection of the present invention, forming a supervised model from the extracted feature data, and training the supervised model further comprises:
and in response to the incremental training of the trained supervised model, loading a model file in advance, extracting characteristic data of the incremental training data, and retraining the supervised model according to the characteristic data extracted from the index data and the characteristic data extracted from the incremental training data in the model file.
In some embodiments of the invention, training includes "incremental training" and "retraining" functions, where "incremental training" refers to a way to train a model using page-scaled data; "retraining" refers to training using all the data in the relational database, and the model is regenerated by means of full training. If the trained model is subjected to incremental training, model files need to be preloaded, characteristic data are extracted from incremental training data, and retraining is carried out.
On the other hand, the embodiment of the invention provides an embodiment of a device for cloud platform non-threshold detection. Fig. 6 is a schematic block diagram of an embodiment of an apparatus for cloud platform threshold-free detection according to the present invention, and fig. 7 is a flowchart illustrating event processing of an embodiment of an apparatus for cloud platform threshold-free detection according to the present invention. As shown in fig. 6 and 7, the apparatus 101 includes:
the data preparation module 11 is configured to obtain index data to be detected, and establish models through a plurality of anomaly detection algorithms respectively;
the unsupervised anomaly detection module 12 is configured to perform unsupervised anomaly detection on the index data to be detected according to the models;
an abnormal index data judgment module 13 configured to acquire abnormal index data from the index data to be detected by unsupervised abnormal detection in response to any one of the models, and judge whether there is a supervised model trained by corresponding historical index data according to the abnormal index data;
an unsupervised anomaly detection result module 14 configured to, in response to the unsupervised model, synthesize all unsupervised anomaly detection results to determine whether the index data to be detected is abnormal;
and the supervised anomaly detection result module 15 is configured to respond to the supervised model, perform supervised anomaly detection on the anomaly index data according to the supervised model, and judge whether the index data to be detected is abnormal or not according to the result of the supervised anomaly detection.
According to some embodiments of the apparatus for cloud platform threshold-free detection of the present invention, the apparatus further comprises:
the abnormal index data configuration module is configured to acquire historical index data to be calibrated from a time sequence database; detecting abnormal groups in the historical index data, and calibrating abnormal index data in the abnormal groups; and storing the abnormal index data into a relational database.
According to some embodiments of the apparatus for cloud platform threshold-free detection of the present invention, the apparatus further comprises:
the supervised model training module is configured to acquire index data of the supervised model; extracting characteristic data of the index data; forming a supervised model according to the extracted feature data, and training the supervised model; judging whether the trained supervised model meets expectations or not; in response to determining that the supervised model is in agreement with the expectation, a model file of the trained supervised model is output.
According to some embodiments of the apparatus for cloud platform non-threshold detection of the present invention, the unsupervised anomaly detection result module 14 is further configured to:
judging certain index data to be detected as abnormal index data in response to half or more than half of all abnormal detection algorithms, and judging the index data to be detected as abnormal data;
and judging a certain index data to be detected as abnormal index data in response to less than half of all the abnormal detection algorithms, and judging the index data to be detected as normal data.
Likewise, it will be appreciated by a person skilled in the art that all the embodiments, features and advantages set forth above for the method for threshold-free detection of a cloud platform according to the present invention are equally applicable to the apparatus according to the present invention. For the sake of brevity of the present disclosure, no repeated explanation is provided herein.
It should be particularly noted that, as one of ordinary skill in the art can appreciate that all or part of the processes of the methods of the above embodiments can be implemented by a computer program to instruct related hardware, and the program of the method for cloud platform threshold-free detection can be stored in a computer readable storage medium, and when executed, the program can include the processes of the embodiments of the methods described above. The storage medium of the program may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A method of cloud platform threshold-free detection, the method comprising:
acquiring index data to be detected, and respectively establishing models through a plurality of anomaly detection algorithms;
respectively carrying out unsupervised anomaly detection on the index data to be detected according to the models;
acquiring abnormal index data from the index data to be detected by responding to the unsupervised abnormal detection of any model, and judging whether a supervised model trained by corresponding historical index data exists or not according to the abnormal index data;
responding to the unsupervised model, and integrating results of all unsupervised abnormal detection to judge whether the index data to be detected is abnormal;
and responding to the presence of the supervised model, carrying out supervised anomaly detection on the abnormal index data according to the supervised model, and judging whether the index data to be detected is abnormal or not according to the result of the supervised anomaly detection.
2. The method of cloud platform threshold-free detection according to claim 1, further comprising:
acquiring the historical index data to be calibrated from a time sequence database;
detecting abnormal groups in the historical index data, and calibrating the abnormal index data in the abnormal groups;
and storing the abnormal index data into a relational database.
3. The method of cloud platform threshold-free detection according to claim 1, further comprising:
acquiring index data of the supervised model;
extracting characteristic data of the index data;
forming the supervised model according to the extracted feature data, and training the supervised model;
judging whether the trained supervised model meets expectations or not;
in response to determining that the supervised model is in agreement with expectations, outputting a model file of the trained supervised model.
4. The method for cloud platform non-threshold detection according to claim 1, wherein the integrating results of all the unsupervised anomaly detections to determine whether the index data to be detected is anomalous in response to the absence of the supervised model further comprises:
judging a certain index data to be detected as abnormal index data in response to half or more than half of all the abnormal detection algorithms, and judging the index data to be detected as abnormal data;
and responding to the abnormality detection algorithm with less than half of all the abnormality detection algorithms to judge certain index data to be detected as the abnormality index data, and judging the index data to be detected as normal data.
5. The cloud platform threshold-free detection method of claim 3, wherein the supervised anomaly detection of the anomaly indicator data according to the supervised model further comprises:
loading the model file of the supervised model;
extracting the characteristic data of the abnormal index data;
inputting the feature data extracted according to the abnormal index data into the model file, and acquiring the detection result of the abnormal index data according to the supervised model.
6. The method of cloud platform threshold-free detection according to claim 3, wherein the forming the supervised model from the extracted feature data and training the supervised model further comprises:
responding to the increment training of the supervised model after training, loading the model file in advance, extracting the characteristic data of increment training data, and retraining the supervised model according to the characteristic data extracted from the index data in the model file and the characteristic data extracted from the increment training data.
7. An apparatus for non-threshold detection of a cloud platform, the apparatus comprising:
the data preparation module is configured to obtain index data to be detected and respectively establish a model through a plurality of anomaly detection algorithms;
the unsupervised anomaly detection module is configured to perform unsupervised anomaly detection on the index data to be detected respectively according to the model;
an abnormal index data judgment module configured to acquire abnormal index data from the index data to be detected in response to the unsupervised abnormal detection of any one of the models, and judge whether there is a supervised model trained by corresponding historical index data according to the abnormal index data;
an unsupervised anomaly detection result module configured to, in response to absence of the supervised model, synthesize results of all the unsupervised anomaly detections to determine whether the index data to be detected is anomalous;
and the supervised anomaly detection result module is configured to respond to the supervised model, carry out supervised anomaly detection on the abnormal index data according to the supervised model, and judge whether the index data to be detected is abnormal or not according to the result of the supervised anomaly detection.
8. The apparatus for cloud platform threshold-free detection of claim 7, wherein the apparatus further comprises:
the abnormal index data configuration module is configured to acquire the historical index data to be calibrated from a time sequence database; detecting abnormal groups in the historical index data, and calibrating the abnormal index data in the abnormal groups; and storing the abnormal index data into a relational database.
9. The apparatus for cloud platform threshold-free detection of claim 7, wherein the apparatus further comprises:
a supervised model training module configured to obtain index data of the supervised model; extracting characteristic data of the index data; forming the supervised model according to the extracted feature data, and training the supervised model; judging whether the trained supervised model meets expectations or not; in response to determining that the supervised model is in agreement with expectations, outputting a model file of the trained supervised model.
10. The cloud platform threshold-free detection apparatus of claim 7, wherein the unsupervised anomaly detection result module further comprises:
judging a certain index data to be detected as abnormal index data in response to half or more than half of all the abnormal detection algorithms, and judging the index data to be detected as abnormal data;
and responding to the abnormality detection algorithm with less than half of all the abnormality detection algorithms to judge certain index data to be detected as the abnormality index data, and judging the index data to be detected as normal data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010851477.6A CN112069037A (en) | 2020-08-21 | 2020-08-21 | Method and device for detecting no threshold value of cloud platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010851477.6A CN112069037A (en) | 2020-08-21 | 2020-08-21 | Method and device for detecting no threshold value of cloud platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112069037A true CN112069037A (en) | 2020-12-11 |
Family
ID=73659831
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010851477.6A Withdrawn CN112069037A (en) | 2020-08-21 | 2020-08-21 | Method and device for detecting no threshold value of cloud platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112069037A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114090396A (en) * | 2022-01-24 | 2022-02-25 | 华南理工大学 | Cloud environment multi-index unsupervised anomaly detection and root cause analysis method |
| CN118606872A (en) * | 2024-08-08 | 2024-09-06 | 华侨大学 | Safety water equipment abnormality detection model and abnormality detection method and equipment |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111352971A (en) * | 2020-02-28 | 2020-06-30 | 中国工商银行股份有限公司 | Bank system monitoring data anomaly detection method and system |
-
2020
- 2020-08-21 CN CN202010851477.6A patent/CN112069037A/en not_active Withdrawn
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111352971A (en) * | 2020-02-28 | 2020-06-30 | 中国工商银行股份有限公司 | Bank system monitoring data anomaly detection method and system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114090396A (en) * | 2022-01-24 | 2022-02-25 | 华南理工大学 | Cloud environment multi-index unsupervised anomaly detection and root cause analysis method |
| CN118606872A (en) * | 2024-08-08 | 2024-09-06 | 华侨大学 | Safety water equipment abnormality detection model and abnormality detection method and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111639497B (en) | Abnormal behavior discovery method based on big data machine learning | |
| CN112987675B (en) | Method, device, computer equipment and medium for anomaly detection | |
| CN113556258B (en) | Anomaly detection method and device | |
| CN110287552B (en) | Motor bearing fault diagnosis method and system based on improved random forest algorithm | |
| CN111309539A (en) | Abnormity monitoring method and device and electronic equipment | |
| CN111796957B (en) | Transaction abnormal root cause analysis method and system based on application log | |
| CN110460591B (en) | CDN flow abnormity detection device and method based on improved hierarchical time memory network | |
| CN111176953B (en) | Abnormality detection and model training method, computer equipment and storage medium | |
| CN112414694A (en) | Equipment multistage abnormal state identification method and device based on multivariate state estimation technology | |
| CN110634081A (en) | Method and device for processing abnormal data of hydropower station | |
| CN112650200A (en) | Equipment fault diagnosis method and device | |
| CN112069037A (en) | Method and device for detecting no threshold value of cloud platform | |
| CN115858606A (en) | Method, device and equipment for detecting abnormity of time series data and storage medium | |
| CN113112188B (en) | Power dispatching monitoring data anomaly detection method based on pre-screening dynamic integration | |
| CN116450482A (en) | User abnormality monitoring method and device, electronic equipment and storage medium | |
| CN112070180A (en) | Power grid equipment state judgment method and device based on information physical bilateral data | |
| CN115878598A (en) | Monitoring data processing method, electronic device and storage medium | |
| CN114356900A (en) | Power data anomaly detection method, device, equipment and medium | |
| CN115967972A (en) | Network anomaly detection method and device, electronic equipment and storage medium | |
| CN117149565A (en) | State detection method, device, equipment and medium for key performance indexes of cloud platform | |
| CN115150248A (en) | Network flow abnormity detection method and device, electronic equipment and storage medium | |
| CN116304814A (en) | Method and system for analyzing working condition of monitoring object based on classification algorithm | |
| CN113724211B (en) | Fault automatic identification method and system based on state induction | |
| CN116661954A (en) | Virtual machine abnormality prediction method, device, communication equipment and storage medium | |
| CN117705178A (en) | Wind power bolt information detection method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20201211 |