CN116304909A - Abnormality detection model training method, fault scene positioning method and device - Google Patents
Abnormality detection model training method, fault scene positioning method and device Download PDFInfo
- Publication number
- CN116304909A CN116304909A CN202310259472.8A CN202310259472A CN116304909A CN 116304909 A CN116304909 A CN 116304909A CN 202310259472 A CN202310259472 A CN 202310259472A CN 116304909 A CN116304909 A CN 116304909A
- Authority
- CN
- China
- Prior art keywords
- fault
- data
- index
- anomaly detection
- positioning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 95
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000012549 training Methods 0.000 title claims abstract description 25
- 230000005856 abnormality Effects 0.000 title claims description 24
- 238000004364 calculation method Methods 0.000 claims abstract description 14
- 230000008859 change Effects 0.000 claims abstract description 8
- 238000004422 calculation algorithm Methods 0.000 claims description 25
- 239000011159 matrix material Substances 0.000 claims description 17
- 238000012545 processing Methods 0.000 claims description 15
- 238000012216 screening Methods 0.000 claims description 9
- 230000000737 periodic effect Effects 0.000 claims description 6
- 230000002159 abnormal effect Effects 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 4
- 230000004807 localization Effects 0.000 claims description 2
- 238000012163 sequencing technique Methods 0.000 claims description 2
- 238000013145 classification model Methods 0.000 abstract description 11
- 238000012423 maintenance Methods 0.000 abstract description 10
- 230000002829 reductive effect Effects 0.000 abstract description 7
- 230000009467 reduction Effects 0.000 abstract description 4
- 230000000875 corresponding effect Effects 0.000 description 11
- 230000008569 process Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000004590 computer program Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 230000000670 limiting effect Effects 0.000 description 3
- 230000000644 propagated effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000003111 delayed effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 241000579895 Chlorostilbon Species 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000010219 correlation analysis Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 239000010976 emerald Substances 0.000 description 1
- 229910052876 emerald Inorganic materials 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000010977 jade Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- ZLIBICFPKPWGIZ-UHFFFAOYSA-N pyrimethanil Chemical compound CC1=CC(C)=NC(NC=2C=CC=CC=2)=N1 ZLIBICFPKPWGIZ-UHFFFAOYSA-N 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 239000010979 ruby Substances 0.000 description 1
- 229910001750 ruby Inorganic materials 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
- Y04S10/52—Outage or fault management, e.g. fault detection or location
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Probability & Statistics with Applications (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to the technical field of intelligent operation and maintenance, in particular to an anomaly detection model training method, a fault scene positioning method and a fault scene positioning device; the fault scene positioning is carried out based on single-index anomaly detection, so that the method can adapt to the characteristics of dynamic software services and the like under a new architecture such as micro services and the like, and the classification model trained by the traditional fixed features is difficult to adapt to the dynamic change of the features. The index-fault correlation is analyzed by combining an unsupervised mode and a supervised mode, a historical fault fingerprint mode is mined from historical fault data, and the historical fault data is not completely relied on, so that the method is more accurate and effective; by dimension reduction of the fault related indexes, calculation cost is effectively reduced, detection efficiency is improved, and low-time-consuming fault scene positioning can be achieved. The fault scene positioning method has good generalization capability, is suitable for positioning fault scenes under new architectures such as micro services, cloud native and the like, and has high adaptability to real scenes such as data loss, delay and the like.
Description
Technical Field
The application relates to the technical field of intelligent operation and maintenance, in particular to an anomaly detection model training method, a fault scene positioning method and a fault scene positioning device.
Background
With the development of cloud computing, the architecture of a software system is gradually changed into a service-oriented architecture, and a micro-service architecture is a typical representative in recent years. Meanwhile, under the trend of a cloud primary architecture, the system is oriented to container operation and maintenance, a high-complexity system, dynamic software service, multi-mode operation and maintenance data and the like become fault diagnosis challenges. Fault scenario localization is one way to analyze the type of fault that occurs and provide an unequivocal manifestation to the operation and maintenance personnel. At present, there are few effective methods for performing fault classification or realizing intelligent fault scene positioning aiming at a micro-service architecture, if timely fault identification and positioning of the fault scene can be realized, the method has important significance for ensuring the reliability and stability of the service, and the method is also an important scene of AIOps. At present, although many researches on fault detection, fault classification and the like are performed, the data characteristics under a micro-service architecture cannot be completely attached, and the direct moving effect is poor. The design of an effective algorithm or scheme realizes the rapid and accurate abnormal detection of time sequence data, and the realization of the efficient fault scene positioning under a micro-service architecture is the key of AIOps landing.
Disclosure of Invention
In order to solve the operation and maintenance problems caused by the increase of the scale and complexity of the system under the trend of cloud primordia and the like, the influence caused by service or system faults under a new architecture is effectively reduced, a low-time-consuming fault scene positioning method is designed, fault identification is rapidly and accurately carried out, fault classification is carried out, and the method and the device for training the abnormality detection model are provided.
In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:
in a first aspect, a training method for an anomaly detection model is provided, where the model is an index-fault correlation matrix, and the training method includes: acquiring historical data arranged in a time sequence data format; detecting the historical data by selecting a corresponding abnormality detection algorithm to obtain an abnormality detection result, matching the abnormality detection result with a fault result, and constructing an index-fault correlation calculation model, wherein the index-fault correlation calculation model is as follows:
wherein R is a correlation coefficient, S a To detect the total number of abnormality of the index S f S is the total number of times the fault actually occurs af The number of times that the detected index anomaly matches the actual fault; screening a plurality of indexes based on a preset index threshold value to obtain a plurality of initial indexes related to faults, processing the initial indexes based on Pearson correlation coefficients to obtain linear correlation coefficients of the initial indexes, and screening the linear correlation coefficients based on the preset threshold value to obtain target indexes; and establishing an index-fault correlation matrix based on a plurality of faults corresponding to the target indexes.
In a first implementation manner of the first aspect, before detecting the historical data by selecting a corresponding anomaly detection algorithm, the method further includes performing a classification process on the historical data, where the classification process includes: and extracting the change characteristics of the historical data to obtain the stationarity, the tendency and the periodic characteristic information associated with the historical data, and dividing the historical data into stationary type data, periodic type data and unstable type data based on the characteristic information.
With reference to the first possible implementation manner of the first aspect, a second possible implementation manner is provided, and the detecting the selection of the corresponding anomaly detection algorithm by using the historical data includes: and selecting a corresponding abnormality detection algorithm based on the type of the historical data.
In a third implementation manner of the first aspect, the R value ranges from [0,1].
With reference to the third possible implementation manner of the first aspect, a fourth possible implementation manner is provided, where the index threshold value is R > 0.8.
In a second aspect, there is provided an anomaly detection model training device, the model being an index-fault correlation matrix, the training device comprising: the data acquisition module is used for acquiring historical data arranged in a time sequence data format; the model construction module is used for constructing an index-fault correlation calculation model; the target index acquisition module is used for acquiring target indexes based on preset screening rules; and the matrix construction module is used for establishing an index-fault correlation matrix based on the target indexes and faults corresponding to the target indexes.
In a third aspect, a fault scenario positioning method is provided, the method including: acquiring real-time data; performing single-index anomaly detection on the real-time data to obtain a detection result; inputting the detection result to a trained abnormality detection model to obtain fault output; and positioning a fault scene based on the fault output result.
In a first implementation manner of the third aspect, before performing single-index anomaly detection on the real-time data, performing data conversion on the real-time data, and converting the real-time data into a time sequence data format under a time window with a uniform size.
With reference to the first implementation manner of the third aspect, in a second implementation manner of the third aspect, inputting the detection result to the trained anomaly detection model to obtain a fault output, where the fault output includes: and inputting the detection result into an index-fault correlation matrix, calculating scores of different faults, sequencing the scores, and taking all faults with scores greater than 90% of the highest score as fault output results.
In a fourth aspect, a fault scenario positioning apparatus is provided, the apparatus comprising: the data acquisition module is used for acquiring real-time data; the anomaly detection module is used for carrying out single-index anomaly detection on the real-time data to obtain a detection result; the fault identification module is used for inputting the detection result into the trained abnormal detection model to obtain fault output; and the fault positioning module is used for positioning the fault scene of the fault output result.
In the technical scheme provided by the embodiment of the application, fault scene positioning is performed based on single-index anomaly detection, so that the method can adapt to the characteristics of dynamic software services and the like under a new architecture such as micro services, and the classification model trained by the traditional fixed features is difficult to adapt to the dynamic change of the features. The index-fault correlation is analyzed by combining an unsupervised mode and a supervised mode, a historical fault fingerprint mode is mined from historical fault data, and the historical fault data is not completely relied on, so that the method is more accurate and effective; by dimension reduction of the fault related indexes, calculation cost is effectively reduced, detection efficiency is improved, and low-time-consuming fault scene positioning can be achieved. The fault scene positioning method has good generalization capability, is suitable for positioning fault scenes under new architectures such as micro services, cloud native and the like, and has high adaptability to real scenes such as data loss, delay and the like.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
The methods, systems, and/or programs in the accompanying drawings will be described further in terms of exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These exemplary embodiments are non-limiting exemplary embodiments, wherein the exemplary numbers represent like mechanisms throughout the various views of the drawings.
Fig. 1 is a schematic structural diagram of a terminal device provided in an embodiment of the present application;
FIG. 2 is a flow chart of a fault scenario locating method shown in some embodiments of the present application;
FIG. 3 is a block schematic diagram of a fault scenario positioning device provided according to an embodiment of the present application;
FIG. 4 is a flow chart of an anomaly detection model training method shown in some embodiments of the present application.
Detailed Description
In order to better understand the technical solutions described above, the following detailed description of the technical solutions of the present application is provided through the accompanying drawings and specific embodiments, and it should be understood that the specific features of the embodiments and embodiments of the present application are detailed descriptions of the technical solutions of the present application, and not limit the technical solutions of the present application, and the technical features of the embodiments and embodiments of the present application may be combined with each other without conflict.
In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it will be apparent to one skilled in the art that the present application may be practiced without these details. In other instances, well-known methods, procedures, systems, components, and/or circuits have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present application.
The flowcharts are used in this application to describe implementations performed by systems according to embodiments of the present application. It should be clearly understood that the execution of the flowcharts may be performed out of order. Rather, these implementations may be performed in reverse order or concurrently. Additionally, at least one other execution may be added to the flowchart. One or more of the executions may be deleted from the flowchart.
Before describing embodiments of the present invention in further detail, the terms and terminology involved in the embodiments of the present invention will be described, and the terms and terminology involved in the embodiments of the present invention will be used in the following explanation.
(1) In response to a condition or state that is used to represent the condition or state upon which the performed operation depends, the performed operation or operations may be in real-time or with a set delay when the condition or state upon which it depends is satisfied; without being specifically described, there is no limitation in the execution sequence of the plurality of operations performed.
(2) Based on the conditions or states that are used to represent the operations that are being performed, one or more of the operations that are being performed may be in real-time or with a set delay when the conditions or states that are being relied upon are satisfied; without being specifically described, there is no limitation in the execution sequence of the plurality of operations performed.
According to the technical scheme provided by the embodiment of the application, the main application scene is operation and maintenance management, and the positioning scheme aiming at the fault scene in the existing operation and maintenance management is generally as follows:
1. and (5) an offline training stage. And training a fault classification model by using the historical data, and detecting and classifying by using the model. 1-1, performing data preprocessing, such as missing value processing, noise processing, data formatting, feature dimension reduction and the like, on the existing historical data, and processing the data into trainable format data. 1-2, establishing a fault classification model, and performing model training based on the processed historical data to obtain the fault classification model.
2. And (5) an online detection stage. And positioning a fault scene according to the trained fault classification model. 2-1, preprocessing the real-time data, such as noise processing, data formatting and the like, and processing the real-time data into format data required by the fault classification model. 2-2, obtaining a fault classification result by the formatted data through a fault classification model, wherein the result is usually used as positioning information of a fault scene, and further processing can be introduced to complete positioning of the fault scene.
The positioning scheme for the existing fault scene has the following characteristics:
1. in the off-line training stage, the data of a plurality of indexes are required to be analyzed together, and a fault classification model is trained. The existing methods have one or more of the following problems: the change information of the time sequence is not analyzed, and the time sequence is classified according to the time data; for static software services only, various index data are fixed (for example, a service only exists on an A server, and various indexes of A.a are always processed); the supervision detection is adopted completely, the dependency on the historical fault data is large, and the generalization capability of the model is lacking; the model has insufficient interpretability.
2. In the online detection stage, fault classification is performed through a trained fault classification model so as to realize fault scene positioning, which may have a good effect on a traditional software system or an industrial process system, because the system is relatively fixed, software services are usually static, have a plurality of periodic indexes, and have relatively small magnitude. However, for the cloud primary architecture, the cloud primary architecture has the new characteristics of higher system complexity, dynamic software service, massive examples, data and the like, and a good generalization model is difficult to generate according to the conventional fault classification method. Meanwhile, the actual data may have partial data missing or delayed condition, and the fault classification model fails or is misdetected after some key indexes are missing or delayed.
Based on the above technical background, the present embodiment provides a terminal device 100, which includes a memory 110, a processor 120, and a computer program stored in the memory and executable on the processor, wherein the processor executes a fault scenario positioning method. In this embodiment, the terminal device communicates with the user terminal, and transmits the acquired detection information to the corresponding user terminal, so as to implement transmission of the detection information on hardware. The method is based on network implementation aiming at the information sending mode, and an association relation between the user terminal and the terminal equipment is required to be established before the terminal equipment is applied, and the association between the terminal equipment and the user terminal can be realized through a registration mode. The terminal device can be aimed at a plurality of user terminals or one user terminal, and the user terminal communicates with the terminal device through passwords and other encryption modes.
In this embodiment, the terminal may be a server, and includes a memory, a processor, and a communication unit for the physical structure of the server. The memory, the processor and the communication unit are electrically connected with each other directly or indirectly to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines. The memory is used for storing specific information and programs, and the communication unit is used for sending the processed information to the corresponding user side.
In this embodiment, the storage module is divided into two storage areas, where one storage area is a program storage unit and the other storage area is a data storage unit. The program storage unit is equivalent to a firmware area, the read-write authority of the area is set to be in a read-only mode, and the data stored in the area can not be erased and changed. And the data in the data storage unit can be erased or read and written, and when the capacity of the data storage area is full, the newly written data can cover the earliest historical data.
The Memory may be, but is not limited to, random access Memory (Random Access Memory, RAM), read Only Memory (ROM), programmable Read Only Memory (Programmable Read-Only Memory, PROM), erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable Read Only Memory (Ele ultrasound ric Erasable Programmable Read-Only Memory, EEPROM), etc.
The processor may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also Digital Signal Processors (DSPs)), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Referring to fig. 2, in the present embodiment, the fault scenario positioning method includes the following steps:
and S210, acquiring real-time data.
In this embodiment, the existing means in the prior art may be used for acquiring the real-time data, but the difference between the existing means and the existing means is that the format conversion is required for the real-time data after the real-time data is acquired in this embodiment, and each index data is converted into the format required by the single-index anomaly detection algorithm.
The format type required by the anomaly detection algorithm in this embodiment is time-series data, and the specific conversion mode is to convert the real-time data into a time-series data format under a time window with a uniform size.
And S220, performing single-index anomaly detection on the real-time data to obtain a detection result.
In the present embodiment, the anomaly detection is performed based on an anomaly detection algorithm, and the anomaly detection algorithm is performed based on different data types, wherein the data types are determined according to the change characteristics of the data. Therefore, the determination of the data type needs to be performed on the real-time data before the anomaly detection is performed, wherein the determination method and the determination of the data type are to primarily divide the data into stable data types, periodic data types, unstable data types and the like by extracting the characteristic information of the stability, the trend and the periodicity of the data.
Different anomaly detection algorithms are selected in advance for processing according to different data types, and the anomaly detection algorithms used in the embodiment comprise 3-sigma, EWMA, polynomial, dynamic threshold, XGBoost and other algorithms. The purpose of the processing procedure is to preliminarily judge whether the time sequence data has abnormality through different algorithms, and the time sequence data is influenced by the initial setting parameters, so that one-time detection may be inaccurate, the abnormality is detected through multiple iterations, and the result is suspected abnormality judged by the algorithm. According to the algorithm configuration and the index library generated by offline training, the method and the device preferentially select the corresponding algorithm for real-time detection on the indexes in the index library, have low calculation cost, and can rapidly detect the abnormality.
And S230, inputting the detection result into the trained abnormal detection model to obtain fault output.
In this embodiment, for the anomaly detection model index-fault correlation matrix, the scores of different faults can be obtained through the index-fault correlation matrix, and according to the score ranking, all faults with scores greater than 90% of the highest score are taken as the fault classification result.
And S240, positioning a fault scene based on the fault output result.
The final fault output can be single fault or multiple faults, and when the final fault output is single fault, the fault scene is directly positioned; if a plurality of faults exist, presetting relevant rules according to expert experience, for example, determining the priority of the fault position according to the structure, the calling relation and the like of the system, presetting the priority of the faults according to the operation and maintenance knowledge of the system, finally positioning a fault scene, and recording a fault classification result as a reference for analysis of operation and maintenance personnel.
The embodiment completes the positioning of the fault scene through the steps. Because the index is subjected to series dimension reduction, the calculation cost can be effectively reduced, the detection efficiency is improved, and the low-time-consuming fault scene positioning is realized.
For the method provided in this embodiment, an apparatus is further provided, referring to fig. 3, a fault scenario positioning apparatus 300, where the apparatus includes: the data acquisition module 310 is configured to acquire real-time data. The anomaly detection module 320 is configured to perform single-index anomaly detection on the real-time data, so as to obtain a detection result. The fault recognition module 330 is configured to input the detection result to the trained anomaly detection model, and obtain a fault output. And the fault locating module 340 is configured to locate a fault scenario for the fault output result.
In the fault scenario positioning method provided in this embodiment, the training of the anomaly detection model belongs to a key process, and referring to fig. 4, a process diagram of a training method of the anomaly detection model is provided, and the method includes:
step S410, acquiring historical data arranged in a time sequence data format.
As in step S210, the process also requires formatting for the history data, and formats the history data to be arranged in a time series data format.
And S420, selecting a corresponding abnormality detection algorithm for detecting the historical data to obtain an abnormality detection result, matching the abnormality detection result with a fault result, and constructing an index-fault correlation calculation model.
In this embodiment, the index-fault correlation calculation model is:
wherein R is a correlation coefficient, S a To detect the total number of abnormality of the index S f S is the total number of times the fault actually occurs af To match the detected index abnormality with the actual fault, the R value is in the range of 0,1]。
The calculation of the "index-fault" correlation requires multiple evaluations, as the initial algorithm design or the parameters differ to some extent in the final result. For the general understanding that the correlation is strong, the requirements are met, and the correlation and algorithm parameters of the index and the fault are directly recorded; and (3) for the index of which the correlation does not reach the threshold value, the parameter needs to be adjusted to be calculated again, iteration is repeated, and if the designated times still do not reach the standard, the value with the maximum correlation number and the algorithm parameter are taken for recording. The data with strong correlation means data with R >0.8 in the embodiment.
And S430, screening a plurality of indexes based on a preset index threshold value to obtain a plurality of initial indexes related to faults, processing the initial indexes based on Pearson correlation coefficients to obtain linear correlation coefficients of the initial indexes, and screening the linear correlation coefficients based on the preset threshold value to obtain target indexes.
It is generally considered that only the "index-fault" correlation score reaches a certain threshold (e.g., 80%) is highly correlated, and that only the index of high correlation is critically effective for locating faults, thus filtering out indexes below the threshold for each fault. If all indexes of a certain fault are filtered, the threshold value of the fault is reduced, then false alarms possibly generated in practice are comprehensively evaluated, if the requirements are not met, the fault is abandoned, and other methods are considered to locate the fault. After the above screening, there are still many high correlation indexes of each fault, and if all the faults are detected, the fault scenario is time-consuming and consumes a lot of computing resources. Typically, some index changes will cause other indexes to change, and some index change rules are almost consistent, and the indexes are detected to be one or more than one index which are not greatly different theoretically. And linear correlation degrees among different KPI indexes are analyzed through the Pearson correlation coefficient, and the indexes with extremely strong correlations (absolute value > 0.8) are screened, so that the index detection number is further reduced, the calculation cost is reduced, and the detection efficiency is improved.
And S440, establishing an index-fault correlation matrix based on the target indexes and the faults corresponding to the target indexes.
The high-correlation index of each fault can be obtained through all the steps, and an index-fault correlation matrix is established, wherein the matrix is an important basis for locating fault scenes in the online detection stage. And an index library can be obtained according to the index-fault correlation matrix, and algorithm configurations of different indexes are obtained according to index-fault correlation analysis and are used for guiding actual anomaly detection.
It is to be understood that the terminology which is not explained by terms of nouns in the foregoing description is not intended to be limiting, as those skilled in the art can make any arbitrary deduction from the foregoing disclosure.
The person skilled in the art can undoubtedly determine technical features/terms of some preset, reference, predetermined, set and preference labels, such as threshold values, threshold value intervals, threshold value ranges, etc., from the above disclosure. For some technical feature terms which are not explained, a person skilled in the art can reasonably and unambiguously derive based on the logical relation of the context, so that the technical scheme can be clearly and completely implemented. The prefixes of technical feature terms, such as "first", "second", "example", "target", etc., which are not explained, can be unambiguously deduced and determined from the context. Suffixes of technical feature terms, such as "set", "list", etc., which are not explained, can also be deduced and determined unambiguously from the context.
The foregoing of the disclosure of the embodiments of the present application will be apparent to and complete with respect to those skilled in the art. It should be appreciated that the process of deriving and analyzing technical terms not explained based on the above disclosure by those skilled in the art is based on what is described in the present application, and thus the above is not an inventive judgment of the overall scheme.
While the basic concepts have been described above, it will be apparent to those skilled in the art that the foregoing detailed disclosure is by way of example only and is not intended to be limiting. Although not explicitly described herein, various modifications, improvements, and adaptations may occur to one skilled in the art. Such modifications, improvements, and modifications are intended to be suggested within this application, and are therefore within the spirit and scope of the exemplary embodiments of this application.
Meanwhile, the present application uses specific terminology to describe embodiments of the present application. Reference to "one embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic is associated with at least one embodiment of the present application. Thus, it should be emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various portions of this specification are not necessarily all referring to the same embodiment. Furthermore, certain features, structures, or characteristics of at least one embodiment of the present application may be combined as suitable.
In addition, those of ordinary skill in the art will understand that the various aspects of the present application may be illustrated and described in terms of several patentable categories or cases, including any novel and useful processes, machines, products, or combinations of materials, or any novel and useful improvements thereto. Accordingly, aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, micro-code, etc.) or by a combination of hardware and software. The above hardware or software may be referred to as a "unit," component, "or" system. Furthermore, aspects of the present application may be embodied as a computer product in at least one computer-readable medium, the product comprising computer-readable program code.
The computer readable signal medium may comprise a propagated data signal with computer program code embodied therein, for example, on a baseband or as part of a carrier wave. The propagated signal may take on a variety of forms, including electro-magnetic, optical, etc., or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code located on a computer readable signal medium may be propagated through any suitable medium including radio, electrical, fiber optic, RF, or the like, or any combination of the foregoing.
Computer program code required for execution of aspects of the present application may be written in any combination of one or more programming languages, including an object oriented programming such as Java, scala, smalltalk, eiffel, JADE, emerald, C ++, c#, vb net, python, etc., or similar conventional programming languages such as the "C" programming language, visual Basic, fortran 2003,Perl,COBOL 2002,PHP,ABAP, dynamic programming languages such as Python, ruby and Groovy or other programming languages. The programming code may execute entirely on the user's computer, or as a stand-alone software package, or partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any form of network, such as a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet), or in a cloud computing environment, or as a service, such as software as a service (SaaS).
Furthermore, the order in which the processing elements and sequences are described, the use of numerical letters, or other designations are used is not intended to limit the order in which the processes and methods of the present application are performed, unless specifically indicated in the claims. While in the foregoing disclosure there has been discussed, by way of various examples, some embodiments of the invention which are presently considered to be useful, it is to be understood that this detail is solely for that purpose and that the appended claims are not limited to the disclosed embodiments, but, on the contrary, are intended to cover all modifications and equivalent arrangements that are within the spirit and scope of the embodiments of this application. For example, while the system components described above may be implemented by hardware devices, they may also be implemented solely by software solutions, such as installing the described system on an existing server or mobile device.
It should also be appreciated that in the foregoing description of the embodiments of the present application, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure aiding in the understanding of at least one of the embodiments of the invention. This method of disclosure, however, is not intended to imply that more features than are presented in the claims are required for the subject application. Indeed, less than all of the features of a single embodiment disclosed above.
Claims (10)
1. An anomaly detection model training method, wherein the model is an index-fault correlation matrix, the training method comprising:
acquiring historical data arranged in a time sequence data format;
detecting the historical data by selecting a corresponding abnormality detection algorithm to obtain an abnormality detection result, matching the abnormality detection result with a fault result, and constructing an index-fault correlation calculation model, wherein the index-fault correlation calculation model is as follows:
wherein R is a correlation coefficient, S a To detect the total number of abnormality of the index S f S is the total number of times the fault actually occurs af The number of times that the detected index anomaly matches the actual fault;
screening a plurality of indexes based on a preset index threshold value to obtain a plurality of initial indexes related to faults, processing the initial indexes based on Pearson correlation coefficients to obtain linear correlation coefficients of the initial indexes, and screening the linear correlation coefficients based on the preset threshold value to obtain target indexes;
and establishing an index-fault correlation matrix based on a plurality of faults corresponding to the target indexes.
2. The anomaly detection model training method of claim 1, further comprising classifying the historical data prior to selecting the corresponding anomaly detection algorithm for detection, the classifying comprising:
and extracting the change characteristics of the historical data to obtain the stationarity, the tendency and the periodic characteristic information associated with the historical data, and dividing the historical data into stationary type data, periodic type data and unstable type data based on the characteristic information.
3. The anomaly detection model training method of claim 2, wherein selecting the corresponding anomaly detection algorithm for detection of the historical data comprises:
and selecting a corresponding abnormality detection algorithm based on the type of the historical data.
4. The anomaly detection model training method of claim 1, wherein the R value range is [0,1].
5. The anomaly detection model training method of claim 4, wherein the indicator threshold value is R > 0.8.
6. An anomaly detection model training device for implementing the method of any one of claims 1 to 5, the anomaly detection model training device comprising:
the data acquisition module is used for acquiring historical data arranged in a time sequence data format;
the model construction module is used for constructing an index-fault correlation calculation model;
the target index acquisition module is used for acquiring target indexes based on preset screening rules;
and the matrix construction module is used for establishing an index-fault correlation matrix based on the target indexes and faults corresponding to the target indexes.
7. A fault scenario positioning method, the method comprising:
acquiring real-time data;
performing single-index anomaly detection on the real-time data to obtain a detection result;
inputting the detection result to a trained abnormality detection model to obtain fault output;
and positioning a fault scene based on the fault output result.
8. The fault scenario positioning method according to claim 7, wherein the step of performing single-index anomaly detection on the real-time data further comprises performing data conversion on the real-time data, and converting the real-time data into a time sequence data format under a time window with a uniform size.
9. The fault scenario localization method of claim 7, wherein inputting the detection result to a trained anomaly detection model to obtain a fault output comprises:
and inputting the detection result into an index-fault correlation matrix, calculating scores of different faults, sequencing the scores, and taking all faults with scores greater than 90% of the highest score as fault output results.
10. A fault scenario positioning device for implementing the method of any one of claims 7 to 9, the fault scenario positioning device comprising:
the data acquisition module is used for acquiring real-time data;
the anomaly detection module is used for carrying out single-index anomaly detection on the real-time data to obtain a detection result;
the fault identification module is used for inputting the detection result into the trained abnormal detection model to obtain fault output;
and the fault positioning module is used for positioning the fault scene of the fault output result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310259472.8A CN116304909A (en) | 2023-03-13 | 2023-03-13 | Abnormality detection model training method, fault scene positioning method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310259472.8A CN116304909A (en) | 2023-03-13 | 2023-03-13 | Abnormality detection model training method, fault scene positioning method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116304909A true CN116304909A (en) | 2023-06-23 |
Family
ID=86800959
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310259472.8A Pending CN116304909A (en) | 2023-03-13 | 2023-03-13 | Abnormality detection model training method, fault scene positioning method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116304909A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117035297A (en) * | 2023-08-02 | 2023-11-10 | 瀚能科技有限公司 | Campus intelligent task allocation method and system based on big data |
| CN117113260A (en) * | 2023-10-19 | 2023-11-24 | 深圳市磐锋精密技术有限公司 | Intelligent laminating equipment fault early warning system based on data analysis |
| CN117310394A (en) * | 2023-11-29 | 2023-12-29 | 天津市英环信诚科技有限公司 | Big data-based power failure detection method and device, electronic equipment and medium |
| CN117527611A (en) * | 2023-11-07 | 2024-02-06 | 北京太极信息系统技术有限公司 | Gaussian distribution-based fault dynamic prediction method and related device |
-
2023
- 2023-03-13 CN CN202310259472.8A patent/CN116304909A/en active Pending
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117035297A (en) * | 2023-08-02 | 2023-11-10 | 瀚能科技有限公司 | Campus intelligent task allocation method and system based on big data |
| CN117035297B (en) * | 2023-08-02 | 2024-04-19 | 瀚能科技有限公司 | Campus intelligent task allocation method and system based on big data |
| CN117113260A (en) * | 2023-10-19 | 2023-11-24 | 深圳市磐锋精密技术有限公司 | Intelligent laminating equipment fault early warning system based on data analysis |
| CN117113260B (en) * | 2023-10-19 | 2024-01-30 | 深圳市磐锋精密技术有限公司 | Intelligent laminating equipment fault early warning system based on data analysis |
| CN117527611A (en) * | 2023-11-07 | 2024-02-06 | 北京太极信息系统技术有限公司 | Gaussian distribution-based fault dynamic prediction method and related device |
| CN117527611B (en) * | 2023-11-07 | 2024-06-07 | 北京太极信息系统技术有限公司 | Gaussian distribution-based fault dynamic prediction method, system, electronic equipment and storage medium |
| CN117310394A (en) * | 2023-11-29 | 2023-12-29 | 天津市英环信诚科技有限公司 | Big data-based power failure detection method and device, electronic equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Schlegel et al. | Towards a rigorous evaluation of XAI methods on time series | |
| CN116304909A (en) | Abnormality detection model training method, fault scene positioning method and device | |
| US12007745B2 (en) | Apparatus for predicting equipment damage | |
| KR102247179B1 (en) | Xai-based normal learning data generation method and device for unsupervised learning of abnormal behavior detection model | |
| CN113537642B (en) | Product quality prediction method and device, electronic equipment and storage medium | |
| US20230351158A1 (en) | Apparatus, system and method for detecting anomalies in a grid | |
| CN114661994B (en) | User interest data processing method and system based on artificial intelligence and cloud platform | |
| CN117538710B (en) | Intelligent early warning method and system for local dynamic discharge monitoring | |
| CN112801316A (en) | Fault positioning method, system equipment and storage medium based on multi-index data | |
| US11004002B2 (en) | Information processing system, change point detection method, and recording medium | |
| US6522768B1 (en) | Predicting system behavior of a managed system using pattern recognition techniques | |
| US20240192672A1 (en) | System and method of monitoring an industrial environment | |
| CN115128438A (en) | Chip internal fault monitoring method and device | |
| US20230297095A1 (en) | Monitoring device and method for detecting anomalies | |
| CN117826771B (en) | Cold rolling mill control system abnormality detection method and system based on AI analysis | |
| CN118395126A (en) | Fault diagnosis method, system and storage medium for equipment rotation state analysis | |
| CN116054910B (en) | Fault analysis and device for earth station equipment based on knowledge graph construction | |
| CN112733015B (en) | User behavior analysis method, device, equipment and medium | |
| CN113535458B (en) | Abnormal false alarm processing method and device, storage medium and terminal | |
| CN115455407A (en) | Machine learning-based GitHub sensitive information leakage monitoring method | |
| CN114003466A (en) | Fault root cause positioning method for micro-service application program | |
| CN114358581A (en) | Method and device for determining abnormal threshold of performance index, equipment and storage medium | |
| CN114564853B (en) | Evaluation report generation method based on FMEA data and electronic equipment | |
| CN115456101B (en) | Data security transmission method and system based on data center | |
| CN115987376B (en) | Method and device for testing performance of radio frequency equipment in earth station |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |