CN111274218A - Multi-source log data processing method for power information system - Google Patents

Abstract

The invention discloses a multisource log data processing method of an electric power information system, which comprises the steps of collecting log data, scanning and dividing each piece of collected log information into field sequences, and storing the field sequences in a database, wherein the log data sources comprise network equipment logs, safety equipment logs and system logs; the data preprocessing is used for removing redundant data to obtain effective data, removing the redundant data from the network equipment log, the safety equipment log and the system log and respectively storing the redundant data in corresponding databases; and (4) associating and fusing multi-source log data, and establishing an association rule base according to the collected network equipment logs, the safety equipment logs and log data in a system log database. In the data preprocessing, in order to remove redundant data which is not in a research range, the log data are classified by adopting a fuzzy clustering algorithm, and the log data are divided into categories in an uncertain manner by adopting a fuzzy clustering method, so that the method is more suitable for actual conditions.

Description

A method for processing multi-source log data of power information system

technical field

The invention relates to the technical field of power system log analysis, in particular to a method for processing multi-source log data of a power information system.

Background technique

Log information can comprehensively record the operation of the system, a large number of events and attacks, and it is very important to analyze the network situation. The main key points and difficulties are as follows:

(1) The log information formats of different data sources are different. When analyzing logs from multiple data sources, you have to deal with log data in different formats. Logs from different data sources not only have different formats, but also different storage methods. Some log information can be easily understood, but some log information is not easy to understand. Therefore, how to understand all log fields clearly when comprehensively analyzing multi-source log information is a key issue in multi-source log analysis.

(2) The amount of log data to be analyzed is large. No matter which source log records are accumulated over time, a huge amount of information will be accumulated. When log data from multiple data sources are combined, the amount of logs to be analyzed will be very large. How to quickly filter out Getting the required information and processing it is also a challenge.

(3) Data alignment. The content provided by the log information from different sources is generated according to its own parameter settings. To combine and analyze different information, it is necessary to put them into the same parameter system. At this time, attribute merging and deletion and data alignment are the most important factors. important issues.

(4) Contradictory information. Because the functions of each device and application are different, when measuring the same network environment, false information may be generated due to some noise, resulting in very different log conclusions from different sources. How to deal with these false information and effectively integrate the correct information will is a key question.

SUMMARY OF THE INVENTION

In view of the problems existing in the above-mentioned prior art, the present invention provides a method for processing multi-source log data of a power information system, including:

Step 1: Collect log data, which is used to obtain log data from various sources, and scan and segment each log information collected into field sequences and store them in the database. The log data sources include network device logs, security device logs and system logs. ;

Step 2: data preprocessing, for removing redundant data to obtain valid data, removing redundant data from network device logs, security device logs and system logs and storing them in corresponding databases respectively;

Step 3: Correlate and integrate multi-source log data, and establish an association rule base according to the collected network device logs, security device logs, and log data in the system log database.

As a further optimization of the above solution, the multi-source log data preprocessing includes:

Remove redundant attribute data of each log information: For each source of log data, according to the preset attribute fields to be retained, the attribute fields to be retained in each log information are extracted and stored by regular expression matching;

Remove the log information whose attribute value exceeds the range: filter out the data that exceeds the range according to the preset range of the attribute value of a certain attribute;

Unify the log data from different sources, and convert all log data into the same format according to the attributes required by each log information in the preset unified format;

As a further optimization of the above scheme, to remove the log information whose attribute value exceeds the range, the data after removing the redundant attribute data is first classified by the fuzzy clustering method, and after the classification is completed:

Fuzzy classification of host system logs according to event types, and keep log data with event types as errors, audit failures and warnings;

Fuzzy classification of security device logs according to the severity of events, and retention of events with the severity of attacks and suspicious events;

Fuzzy classification of network device logs according to event severity, and events with severity levels emergency, alert, critical, error, and warnings are retained.

As a further optimization of the above scheme, the classification by the fuzzy clustering method is specifically:

4.1. Record the collected log data as X={x1, x2, x3,.....xn}, where Xi is the vector data composed of the attribute values corresponding to the retained attributes;

4.2. Obtain the similarity of each two log data according to the matrix X, and obtain the similarity matrix R, wherein the calculation of the similarity adopts the angle cosine method;

4.3. According to the similarity matrix R, use the transitive closure method to obtain the equivalent matrix, and the formula is: R ^2k =(R ^2k ) ² , R ^2k is the transitive closure;

4.4. According to the equivalent matrix, take λ∈[0, 1] to obtain the clustering results under different λ values of the log data, and judge the most realistic λ and clustering results according to the actual situation.

As a further optimization of the above solution, the association and fusion of multi-source log data is specifically:

Arrange the log data in the unified format according to the occurrence time, set the sliding time window according to the preset time interval, divide the log data into different sliding time windows, and obtain the premise of the occurrence of the event for each sliding time window. The conditions and resulting results form the correlation rules form for each occurrence.

A method for processing multi-source log data of a power information system of the present invention has the following beneficial effects:

1. A method for processing multi-source log data in a power information system of the present invention, in data preprocessing, in order to remove redundant data that is not within the scope of research, a fuzzy clustering algorithm is used to classify the log data, and the fuzzy clustering method is used to classify the log data. The data is divided into categories indefinitely, which is more in line with the actual situation.

2. A method for processing multi-source log data of a power information system of the present invention integrates log data from various sources of network equipment logs, security equipment logs and system logs, making the data more complete and facilitating comprehensive analysis through logs from multiple sources. Security situation of power information system.

Description of drawings

1 is an overall flow chart of a method for processing multi-source log data in a power information system according to the present invention;

2 is a flowchart of a fuzzy clustering algorithm in a method for processing multi-source log data in a power information system according to the present invention;

Implementation

The following describes a method for processing multi-source log data of a power information system according to an embodiment of the present invention with reference to the accompanying drawings.

As shown in the figure, a method for processing multi-source log data of a power information system of the present invention includes:

Step 1: Collect log data, which is used to obtain log data from various sources, and scan and segment each log information collected into field sequences and store them in the database. The log data sources include network device logs, security device logs and system logs. ;

Among them, the collected log data includes a Linux host log collection module, a Windows host log collection module, a security device log collection module and a network device log collection module; the method for collecting the Linux host log is: perform the syslog configuration service on the Linux host group, Install the connection software on the Windows host that collects log information, use the functions provided in the Syslog server to transfer the logs collected on the Linux host cluster to the Windows host, and transfer the collected and transmitted log files as text files. Storage; when collecting Windows log information, use the eventquery script tool in the system to collect logs; for security device log collection, choose to use the open source snort system to obtain; in network device log collection, choose to use switches to As a network device to collect logs, a syslog server is created on the windows system, and udp receives the log information of the switch through the port; the log data of each source is stored in its corresponding database.

Step 2: data preprocessing, for removing redundant data to obtain valid data, removing redundant data from network device logs, security device logs and system logs and storing them in corresponding databases respectively;

Step 3: Correlate and integrate multi-source log data, and establish an association rule base according to the collected network device logs, security device logs, and log data in the system log database.

The multi-source log data preprocessing in step 2 includes:

Remove redundant attribute data of each log information: For each source of log data, according to the preset attribute fields to be retained, the attribute fields to be retained in each log information are extracted and stored by regular expression matching;

In the host system log information, the attributes that need to be preserved include event type, event ID, event source, computer name, user, event description, log type;

Attributes required in the security device log: time node, the time when the event occurred; alarm message (describes what vulnerability or attack was found); protocol; source IP address, attack initiation point; target IP address, attack destination; target port; source mac address; destination mac address.

Attributes required in the switch: time node, switch IP address, switch host, event detail description, severity description.

Remove the log information whose attribute value exceeds the range: filter out the data that exceeds the range according to the preset range of the attribute value of a certain attribute;

Unify the log data from different sources, and convert all log data into the same format according to the attributes required by each log information in the preset unified format; unify the log data from all sources into the same format and save, the format is { Event number, event name, event start time node, event end time node, source IP address, destination IP address, source port number and destination port number, event detailed description}.

To remove the log information whose attribute value exceeds the range, first classify the data after removing redundant attribute data by fuzzy clustering method. After the classification is completed:

Fuzzy classification of host system logs according to event types, and keep log data with event types as errors, audit failures and warnings;

Fuzzy classification of security device logs according to the severity of events, and retention of events with the severity of attacks and suspicious events;

Fuzzy classification of network device logs according to event severity, and events with severity levels emergency, alert, critical, error, and wamings are retained.

The log data is classified by fuzzy clustering method, specifically:

Record the collected log data as X={x1, x2, x3,.....xn}, where Xi is the vector data composed of the attribute values corresponding to the retained attributes;

Obtain the similarity of each two log data according to the matrix X, and obtain the similarity matrix R, wherein the calculation of the similarity adopts the angle cosine method;

According to the similarity matrix R, the equivalent matrix is obtained by the transitive closure method, and the formula is: R ^2k =(R ^2k ) ² , R ^2k is the transitive closure;

According to the equivalent matrix, take λ∈[0, 1] to obtain the clustering results under different λ values of the log data, and judge the most realistic λ and clustering results according to the actual situation.

First use the following table to represent all the data:

log data attribute 1 property 2 ...... attribute m 1st log data Article 2 Log Data ...... The nth log data

The above data is represented by a corresponding matrix, X={x1, x2,.....xn}, wherein Xi is the vector data composed of the attribute values corresponding to the retained m attributes;

For each piece of data, that is, each row of data in the matrix X, the angle cosine method is used to obtain the similarity of each two data. The angle cosine method formula is:

Among them, rij represents the similarity of each two pieces of data, and k represents the kth column, that is, the kth attribute;

After obtaining the similarity matrix R, by calculating R ^2k , k takes 1, 2..n; until R ^2k =(R ^2k ) ² , at this time R ^2k is the closure required for the similarity matrix to obtain the equivalent matrix;

After obtaining the equivalent matrix, take λ∈[0, 1] to obtain the clustering results under different λ values, compare the different clustering results, and determine the most realistic clustering results and the corresponding λ values.

After the log data with redundant data removed is obtained, data from different sources is integrated, and the method for associating and fusing multi-source log data is as follows:

Arrange the log data in the unified format according to the occurrence time, set the sliding time window according to the preset time interval, divide the log data into different sliding time windows, and obtain the premise of the occurrence of the event for each sliding time window. The conditions and resulting results form the correlation rules form for each occurrence.

Obtain the preconditions and results of the occurrence of the event. The specific method is: find the set of event types that frequently occur in the same sliding time window. If event A and event B often appear in a sliding time window, then event A can be considered as event A. and event B are frequently occurring event types, so it is inferred that there may be a certain correlation between them, and event A and event B have a causal relationship.

The present invention is not limited to the above-mentioned specific embodiments, and various transformations made by those of ordinary skill in the art from the above-mentioned concept without creative work all fall within the protection scope of the present invention.

Claims (5)

1. A multi-source log data processing method of a power information system is characterized by comprising the following steps: the method comprises the following steps:

step 1: collecting log data, wherein the log data are used for acquiring log data with different sources, scanning and dividing each piece of collected log information into field sequences and storing the field sequences in a database, and the sources of the log data comprise a network equipment log, a safety equipment log and a system log;

step 2: the data preprocessing is used for removing redundant data to obtain effective data, removing the redundant data from the network equipment log, the safety equipment log and the system log and respectively storing the redundant data in corresponding databases;

and step 3: and (4) associating and fusing multi-source log data, and establishing an association rule base according to the collected network equipment logs, the safety equipment logs and log data in a system log database.

2. The multi-source log data processing method of the power information system according to claim 1, wherein: the multi-source log data preprocessing comprises:

removing redundant attribute data of each piece of log information: according to the log data of each source, extracting and storing the attribute fields needing to be reserved in each log information in a regular expression matching mode according to the preset attribute fields needing to be reserved;

removing log information with property values out of range: filtering out data beyond a range according to a preset range of an attribute value of a certain attribute;

and unifying the formats of the log data from different sources, and converting all the log data into the same format according to the attribute required by each piece of log information in the preset unified format.

3. The multi-source log data processing method of the power information system according to claim 2, characterized in that: the log information with the attribute value out of range is removed, the data with the redundant attribute data removed is classified by a fuzzy clustering method, and after the classification is finished:

carrying out fuzzy classification on the logs of the host system according to event types, and keeping log data of which the event types are errors, auditing failures and warnings;

the safety device logs are classified in a fuzzy mode according to the severity of the events, and the severity of the events is reserved as attacks and suspicious events;

the network device logs are classified according to the event severity in a fuzzy mode, and events with the severity levels of emergency, alert, critic, error and warns are reserved.

4. The multi-source log data processing method of the power information system according to claim 3, characterized in that: the classification by the fuzzy clustering method specifically comprises the following steps:

4.1, recording the collected log data as X ═ { X1, X2, X3,.. times.xn }, wherein xi is vector data composed of attribute values corresponding to the reserved attributes;

4.2, obtaining the similarity of every two log data according to the matrix X to obtain a similarity matrix R, wherein the similarity is calculated by adopting an included angle cosine method;

4.3, obtaining the equivalent matrix by adopting a transmission closed-packet method according to the similar matrix R, wherein the formula is as follows: r^2k＝(R^2k)²，R^2kIs a transitive closure;

and 4.4, according to the equivalent matrix, taking lambda belonging to [0, 1], obtaining clustering results of the log data under different lambda values, and judging the lambda and the clustering result which most accord with the reality according to the practical situation.

5. The multi-source log data processing method of the power information system according to claim 1, wherein: the associated and fused multi-source log data specifically comprises the following steps:

arranging the log data with the unified format according to occurrence time, setting sliding time windows according to preset time intervals, dividing the log data into different sliding time windows, and acquiring preconditions and results of occurrence of events for the occurrence of the events in each sliding time window to form an association rule form of each occurrence.

Images