CN113676464A - Network security log alarm processing method based on big data analysis technology - Google Patents
Network security log alarm processing method based on big data analysis technology Download PDFInfo
- Publication number
- CN113676464A CN113676464A CN202110910555.XA CN202110910555A CN113676464A CN 113676464 A CN113676464 A CN 113676464A CN 202110910555 A CN202110910555 A CN 202110910555A CN 113676464 A CN113676464 A CN 113676464A
- Authority
- CN
- China
- Prior art keywords
- log
- analysis
- alarm
- logs
- network security
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0604—Management of faults, events, alarms or notifications using filtering, e.g. reduction of information by using priority, element types, position or time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a network security log alarm processing method based on big data analysis technology, which is characterized in that standardized log data with uniform format is obtained through an automatic analysis engine, useless and meaningless logs are filtered out through log filtering rules, then the logs are converted into security alarm events through network attack event classification rules, and then the alarm events which are related within a certain time range are combined through alarm rules and a related analysis algorithm, so that repeated alarms are effectively removed, the repeated occurrence frequency is recorded, and the analysis quality of the alarm events is improved. The invention establishes a set of intuitive, reliable and stable network security log analysis method, carries out paradigm analysis processing on the collected network security log data in a simple and convenient and quick way to generate log data with unified log attributes, and then carries out de-coincidence and processing on alarm events generated by the logs, thereby effectively improving the network security detection efficiency, reducing the false alarm and repetition of the alarm and further reducing the pressure of alarm analysis processing.
Description
Technical Field
The invention relates to the technical field of power information systems, in particular to a network security log alarm processing method based on a big data analysis technology.
Background
In recent years, information network security events are frequent, security situations become more severe day by day, log data generated by various network security devices are rapidly increased along with the expansion of network boundaries and the increase of network security devices, a large number of log events need to be identified and processed by security operation and maintenance personnel, the log data comprise a plurality of repeated logs and invalid garbage logs, different security manufacturers have different attack feature libraries, different monitoring and alarming strategies are adopted, in order to improve the overall perception capability of network security, only an alarming threshold value can be reduced, so that a plurality of attack alarms are reported in wrong, the monitoring pressure of the monitoring personnel is high, in addition, the same flow is captured by different security devices due to a deployment mode of the network security devices, and further, the same attack behavior triggers the alarming of a plurality of security devices passing along the way, so that the phenomenon of repeated alarming is caused.
The method has the advantages that false alarm and repetition of alarm are effectively reduced, the capacity of alarm analysis processing is improved, two important tasks need to be completed, firstly, high-efficiency and accurate standardized analysis in a unified format is carried out on multi-source heterogeneous log data, secondly, the standardized log data in the unified format is subjected to correlation analysis so as to effectively remove the duplicate of log alarm events, and the purposes of reducing false alarm and repetition of alarm are achieved.
The standardized analysis of the logs needs to compile a regular expression to perform field escape on log data according to the device logs from multiple sources, so that the standardized analysis of the logs is completed.
Most of existing log association analysis deduplication algorithms are based on Apriori algorithm or FP-growth algorithm.
The Apriori algorithm is an algorithm for mining a frequent item set of a Boolean association rule, and has the disadvantages that the scanning times of a database are too many, a large number of candidate item sets are possibly generated, the operation time is obviously increased under the condition that the length of the frequent item set is increased, a unique support degree is adopted, and the difference of the importance degrees of various attributes is not considered;
the FP-growth algorithm is constructed based on an Apriori algorithm, a frequent item set or a frequent item pair is found after a data set is stored in a specific structure called as an FP tree, and the FP-growth algorithm has the defect that the efficiency of the algorithm is greatly reduced if too many child nodes of the tree are generated, for example, a tree only containing a prefix is generated; in addition, the FP-Growth algorithm needs to generate a conditional database and a conditional FP-tree recursively, so that the memory overhead is large, and the FP-Growth algorithm can only be used for mining a single-dimensional Boolean association rule.
Disclosure of Invention
The invention provides a network security log alarm processing method based on a big data analysis technology, which is used for solving the problem that the generated alarm event message cannot be processed timely and efficiently due to the fact that the types and the quantity of network security logs are large in the prior art.
The invention adopts the following technical scheme:
a network security log alarm processing method based on big data analysis technology comprises the following steps:
step S1: collecting logs generated by various types of network security equipment needing alarm analysis;
step S2: the log analysis engine automatically traverses a built-in grok expression analysis rule base, if the rule base has a corresponding rule, the step S3 is carried out, and if the rule base does not have the corresponding rule, the step S4 is carried out;
step S3: the log analysis engine carries out automatic word segmentation processing on the logs according to the analysis rules according to the matched rules, then automatically generates a normal file, and the analysis engine carries out escape on the logs according to standard normal fields through a logstack filter plug-in unit to generate standard logs with a uniform format;
step S4: the logs which cannot traverse the corresponding rules are sent to an unresolved log library, the auxiliary function of an analysis engine is used manually, the semantic mapping rules are adjusted according to the comparison between the reference analysis result given by the analysis engine and the original log data by selecting proper word segmentation symbols and proper semantic mapping rules of each field after word segmentation until standardized logs in a uniform format can be generated correctly and effectively, the semantic mapping rules are supplemented into the analysis rule library, and the logs are analyzed by the analysis engine;
step S5: filtering the logs by the written log filtering rules to remove meaningless logs;
step S6: converting the standardized log data into a network security alarm event;
step S7: for log data which is not in the definition range of the network attack event classification rule, defining the log data as a network security alarm event by customizing a newly added alarm rule;
step S8: performing correlation analysis on network security alarm events within a period of time by a cosine similarity algorithm, merging the alarm events meeting the similarity rule, and recording the number of events;
step S9: and downloading and analyzing the associated flow data packet through the log data merged by the alarm event, and adjusting the similarity rule value of the cosine similarity algorithm of the association analysis according to the analysis.
Further, in step S1, log collection of various types of security devices is performed by the big data component logstack based on the syslog protocol, and original logs of different encoding formats are received.
Further, the normalized fields in step S3 include 40 types of fields including the defined system identification content, event body part, event object part, event semantic part, event generation part, event source part, other attributes, and 26 customizable reserved fields for parsing the log message.
Further, after filtering the log in step S5, the stored log is saved in the Elasticsearch database, and an index is created for log query.
Further, step S6 specifically includes: according to preset network security attack stages and attack event classification rules, log events are matched and associated for analysis according to the attack stages and the event classifications, log data are stored in a data warehouse according to the event classifications of different attack stages, and a security event analysis subject library is formed.
Further, in the step S7, the writing of the newly added customized alarm rule is performed by manually combining logical operators such as and, or, not, containing, greater than, less than, and equal to, and screening security events in combination with the value range of the log standard field, and the customized rule is generated by dragging, selecting, and filling in gaps with numerical values of low codes.
Further, in the step S7, the writing of the newly added customized alarm rule is performed by manually combining logical operators such as and, or, not, containing, greater than, less than, and equal to, and screening security events in combination with the value range of the log standard field, and the customized rule is generated by dragging, selecting, and filling in gaps with numerical values of low codes.
A network security log alarm deduplication processing system, comprising:
the acquisition module is used for acquiring logs generated by various types of network safety equipment needing alarm analysis;
the normalized processing module is internally embedded with a log analysis engine of a grok expression analysis rule base and used for automatically segmenting the logs according to the analysis rules according to the matched rules and then automatically generating normalized files, and the analysis engine is used for enabling the normalized files to perform escape on the logs according to standard normalized fields through a logstack filter plug-in to generate standardized logs with uniform formats;
the filtering module is used for filtering the logs according to the written log filtering rules to remove meaningless logs;
the alarm processing module is used for converting the standardized log data into a network security alarm event;
the correlation analysis module is used for performing correlation analysis on the network security alarm events within a time range by a cosine similarity algorithm, merging the alarm events meeting the similarity rule and recording the number of the events;
and the associated flow downloading module is used for downloading and analyzing the associated flow data packet through the log data merged by the alarm event and adjusting the similarity rule value of the cosine similarity algorithm of the association analysis according to the analysis.
The invention obtains standardized log data with uniform format through the automatic analysis engine, filters useless and meaningless logs through the log filtering rule, converts the logs into safety alarm events through the network attack event classification rule, and combines the related alarm events within a certain time range through the alarm rule and the related analysis algorithm, thereby effectively removing repeated alarms, recording the repeated occurrence frequency and improving the analysis quality of the alarm events.
Drawings
FIG. 1 is a flow chart of the network security log alarm processing method based on big data analysis technology of the present invention;
FIG. 2 is a schematic diagram of the cosine similarity algorithm of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, an embodiment of the present invention provides a network security log alarm processing method based on big data analysis technology, including the following steps:
step S1: and collecting logs generated by various types of network security equipment needing alarm analysis.
The invention is based on syslog protocol and carries out log collection of various safety devices by a big data component logstack, and receives original logs with different coding formats.
Step S2: the log parsing engine will automatically traverse the built-in grok expression parsing rule base, if there is a corresponding rule in the rule base, go to step S3, if there is no rule, go to step S4.
Step S3: and the log analysis engine automatically performs word segmentation processing on the log according to the matched rule according to the analysis rule, then automatically generates a normalized file, and the analysis engine performs escape on the log according to the standard normalized field by the normalized file through a logstack filter plug-in to generate the standardized log with a uniform format.
The standard normalized field comprises 40 types of fields of defined system identification content, an event body part, an event object part, an event semantic part, an event generation part, an event original part and other attributes, and 26 reserved fields which can be customized so as to clearly analyze the log message, and the fields obtained by standardized analysis are used for carrying out a merging and de-duplication step on subsequent alarm event correlation analysis.
The data information after the normalization processing comprises:
1) the system identifies the content: event name, event abstract, event type, event grade, network protocol and network application protocol;
2) the event main part: source name, source MAC address, source IP, source translation IP address, source port, source translation port;
3) event object part: destination name, destination MAC address, destination IP, destination conversion IP address, destination port and destination conversion port;
4) event semantic part: user name, program name, operation, object, result;
5) an event generation section: response, equipment name, equipment type, equipment IP, generation time and monitoring value;
6) event primitive part: original grade, original type, sending flow, receiving flow, duration and request information;
7) other attributes: event receiving time, collector IP address, collection type, original message, merging number.
Step S4: the logs which cannot traverse the corresponding rules (namely the logs which cannot be correctly and effectively analyzed by the analysis engine) are sent to an unresolved log library, the auxiliary function of the analysis engine is used manually, the semantic mapping rules are adjusted according to the comparison between the reference analysis result given by the analysis engine and the original log data by selecting proper word segmentation symbols and proper semantic mapping rules of each field after word segmentation until the standardized logs with the uniform format can be correctly and effectively generated, the semantic mapping rules are supplemented into the analysis rule library, and the logs are analyzed by the analysis engine.
Step S5: and filtering the logs by the written log filtering rules, removing meaningless logs such as device state logs and device login logs, and filtering, screening and storing the logs meeting the conditions by taking the log filtering rules as screening conditions.
In this step, the stored log is reserved and stored in the Elasticsearch database, and an index is established and provided for log query.
Step S6: the standardized log data are converted into network security alarm events, the log events are matched and subjected to correlation analysis according to attack stages and event classifications according to preset network security attack stages and attack event classification rules, and the log data are stored in a data warehouse according to the event classifications of different attack stages to form a security event analysis subject library.
The network security attack stage and the attack event comprise:
1) attack preparation: information collection (basic information collection, scanning, network monitoring, data theft), others (malicious domain name access)
2) Attack implementation: exploit attacks (misconfiguration, security vulnerabilities, service vulnerabilities, injection attacks, system vulnerabilities, application vulnerabilities, protocol vulnerabilities), advanced Attacks (APT), password attacks (brute force, social engineering, weak passwords), denial of service attacks, spoofing attacks (email spoofing, WEB spoofing), hijacking attacks (session hijacking, package hijacking, domain hijacking), others (general)
3) And (3) a sinking stage: promoting attack authority, Windows \ Linux trace clearing, advanced attack (backdoor program), security system backdoor (system backdoor, webpage backdoor)
4) And (3) transverse movement: intranet penetration (host penetration, intranet bounce, domain penetration).
Step S7: for log data which is not in the definition range of the network attack event classification rule, defining the log data as a network security alarm event by customizing a newly added alarm rule, redefining the attribute of the log alarm event, and establishing another index in an Elasticissearch database for the generated security alarm event.
In the step, the customized newly added alarm rule is compiled by manually combining logical operators such as and, or, not, containing, greater than, less than and equal to, and the like, screening the security events by combining the value range of the log standard field, and generating the customized rule by dragging and selecting low codes and filling numerical values in a blank mode.
Step S8: and performing correlation analysis on the network security alarm events within a period of time by a cosine similarity algorithm, merging the alarm events meeting the similarity rule, and recording the number of the events. The default time period is set to 1 minute.
Cosine similarity algorithm: the cosine value between the included angles of the two vectors in a vector space is used as the measure of the difference between the two individuals, the cosine value is close to 1, the included angle tends to 0, the more similar the two vectors are, the cosine value is close to 0, and the included angle tends to 90 degrees, the more dissimilar the two vectors are.
The cosine similarity algorithm is shown in fig. 2, in which the angle formula of the two-dimensional vector is as follows:
the formula of the included angle further extended to the N-dimensional vector is as follows:
A＝{x1，x2，x3...xn} B＝{y1，y2，y3...yn}
in the calculation of the similarity of the log data, the extracted log data is an attribute field from a standardized log with a uniform format, the calculation model may select the extracted log attribute field, and the selected attribute field is added into a calculation formula, for example, A, B in the above formula represents vectors a and B formed by respectively extracting log attribute fields of two pieces of log data, where the extracted key elements include 40 defined type fields and 26 reserved fields in step S3, and the calculation steps are described as follows:
such as: after the original log data is subjected to standardized analysis and vector component element selection, the main effective data of the standardized log participating in calculation comprises the following data: { time of occurrence of event: 2021-08-0610:39: 18, source address: 59.38.139.62, source port: 8080, destination address: 192.168.181.221, destination port: 8080, protocol: http, event name: scan, event type: malicious behavior, level: low risk, equipment type: safety device/flow detection device }, in this data, time takes interval value according to a selected time range and event occurrence time, the time interval is set as 1 minute by default, log data to be compared is log data in the 1 minute range, in order to facilitate understanding of the time here is calculated according to occurring seconds, the value after the value of 39 minutes is 18, a source address is converted into int type number 992381758, a port value is 8080, a destination address is converted into int type number 3232282077, a port value is 8080, a protocol corresponds to the value in the database as 6, an event name corresponds to 11 in an attack stage event classification rule table of a scanning event in the database, a malicious behavior event type corresponds to 9 in the database, a class corresponds to 1, a classification of the device type corresponds to the value in the database as 26, and the value digit and elements calculated comprehensively correspond to the value rules in the database, thus, vector a is {0.18, 0.992381758, 0.8080, 0.3232282077, 0.8080, 0.6, 0.11, 0.9, 0.1, 0.26 }.
After the original log data is subjected to standardized analysis and vector component element selection, the main effective data of the standardized log participating in calculation comprises the following data: { time of occurrence of event: 2021-08-0610:39:47, source address: 59.38.130.31, source port: 8080, destination address: 192.168.181.221, destination port: 8080, protocol: http, event name: basic information collection, event type: information collection, rating: low risk, equipment type: safety device/flow detection device }, the time of occurrence value is 47, the source address is converted into int type number 992379423, the port value is 8080, the destination address is converted into int type number 3232282077, the port value is 8080, the protocol corresponds to 6 in the database, the event name corresponds to 10 in the event classification rule table of the attack stage in the database for the basic information collection event, the event type of information collection corresponds to 5 in the database, the class corresponds to 1, the device type classification corresponds to 26 in the database, the value digit and the element of the comprehensive calculation correspond to the value rule in the database, so the vector B ═ 0.47, 0.992379423, 0.8080, 0.63232282077, 0.8080, 0.6, 0.1, 0.5, 0.1, 0.26 }.
The calculation result is 0.998327, and the two logs can be judged to be associated according to the threshold setting (for example, currently set to 0.995) of the similarity calculation.
The method comprises the steps of setting a threshold value of similarity calculation in a system, defining alarm events which are greater than or equal to the threshold value as associated events (such as 0.995), combining the calculated associated alarm events, and recording the number of the events.
In addition, the value of other elements which need to be added into the calculation formula can be set to 0 if the condition that the value is empty is met.
Except that five-element group data of a source IP, a destination IP, a source port, a destination port and a network protocol are listed in a vector of a calculation formula, normalized data fields contained in other log alarm events can be used as calculation vectors, the closer the calculation result is to 1, the more similar the log alarm events are, according to a set similarity threshold, the alarm events reaching and exceeding the threshold are defined as associated events, the calculated associated alarm events are combined, and the number of the events is recorded.
Step S9: and downloading and analyzing the associated flow data packet through the log data merged by the alarm event, and adjusting the similarity rule value of the cosine similarity algorithm of the association analysis according to the analysis, thereby achieving the effect of adjusting the association analysis. Specifically, the adjustment of the optimization is achieved by adjusting the cosine similarity calculation result value after analyzing the journal data lower ampacity data packet associated with the alarm event, setting a new threshold value, and adjusting the vector element selection range and the vector element value rule.
The embodiment of the invention also provides a system for processing network security log alarm duplicate removal, which comprises:
the acquisition module is used for acquiring logs generated by various types of network safety equipment needing alarm analysis; the invention is based on syslog protocol and carries out log collection of various safety devices by a big data component logstack, and receives original logs with different coding formats.
The normalized processing module is internally embedded with a log analysis engine of a grok expression analysis rule base and used for automatically segmenting the logs according to the analysis rules according to the matched rules and then automatically generating normalized files, and the analysis engine is used for enabling the normalized files to carry out the escape on the logs according to standard normalized fields through a logstack filter plug-in to generate the normalized logs with a uniform format.
And sending the logs which cannot be correctly and effectively analyzed by the analysis engine to an unresolved log library, carrying out word segmentation display on the logs by manually giving word segmentation characters by the log analysis engine, giving a reference result to be compared with the original logs, and completing standardized analysis on the logs by manually supplementing the escape mapping rules.
The filtering module is used for filtering the logs by the written log filtering rules, removing meaningless logs such as device state logs and device login logs, and filtering, screening and storing the logs meeting the conditions by taking the log filtering rules as screening conditions;
the alarm processing module is used for converting the standardized log data into network security alarm events, specifically, the log events can be matched and associated for analysis according to attack stages and event classifications according to preset network security attack stages and attack event classification rules, and the log data can be stored in a data warehouse according to the event classifications of different attack stages to form a security event analysis subject library.
For log data which is not in the definition range of the network attack event classification rule, defining the log data as a network security alarm event by customizing a newly added alarm rule, redefining the attribute of the log alarm event, and establishing another index in an Elasticissearch database for the generated security alarm event.
And the association analysis module is used for performing association analysis on the network security alarm events within a time range by a cosine similarity algorithm, combining the alarm events meeting the similarity rule and recording the number of the events.
And the associated flow downloading module is used for downloading and analyzing the associated flow data packet through the log data merged by the alarm event and adjusting the similarity rule value of the cosine similarity algorithm of the association analysis according to the analysis, thereby achieving the effect of adjusting the association analysis.
The invention establishes a set of intuitive, reliable and stable network security log analysis method, carries out paradigm analysis processing on the collected network security log data in a simple and convenient and quick way to generate log data with unified log attributes, and then carries out de-coincidence and processing on alarm events generated by the logs, thereby effectively improving the network security detection efficiency, reducing the false alarm and repetition of the alarm and further reducing the pressure of alarm analysis processing.
The above description is only an embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (7)
1. A network security log alarm processing method based on big data analysis technology is characterized in that: the method comprises the following steps:
step S1: collecting logs generated by various types of network security equipment needing alarm analysis;
step S2: the log analysis engine automatically traverses a built-in grok expression analysis rule base, if the rule base has a corresponding rule, the step S3 is carried out, and if the rule base does not have the corresponding rule, the step S4 is carried out;
step S3: the log analysis engine carries out automatic word segmentation processing on the logs according to the analysis rules according to the matched rules, then automatically generates a normal file, and the analysis engine carries out escape on the logs according to standard normal fields through a logstack filter plug-in unit to generate standard logs with a uniform format;
step S4: the logs which cannot traverse the corresponding rules are sent to an unresolved log library, the auxiliary function of an analysis engine is used manually, the semantic mapping rules are adjusted according to the comparison between the reference analysis result given by the analysis engine and the original log data by selecting proper word segmentation symbols and proper semantic mapping rules of each field after word segmentation until standardized logs in a uniform format can be generated correctly and effectively, the semantic mapping rules are supplemented into the analysis rule library, and the logs are analyzed by the analysis engine;
step S5: filtering the logs by the written log filtering rules to remove meaningless logs;
step S6: converting the standardized log data into a network security alarm event;
step S7: for log data which is not in the definition range of the network attack event classification rule, defining the log data as a network security alarm event by customizing a newly added alarm rule;
step S8: performing correlation analysis on network security alarm events within a period of time by a cosine similarity algorithm, merging the alarm events meeting the similarity rule, and recording the number of events;
step S9: and downloading and analyzing the associated flow data packet through the log data merged by the alarm event, and adjusting the similarity rule value of the cosine similarity algorithm of the association analysis according to the analysis.
2. The big data analysis technology-based network security log alarm processing method of claim 1, wherein: in step S1, log collection of various security devices is performed by the big data component logstack based on the syslog protocol, and original logs of different encoding formats are received.
3. The big data analysis technology-based network security log alarm processing method of claim 1, wherein: the normalized fields in step S3 contain 40 types of fields including the defined system identification content, event subject part, event object part, event semantic part, event generation part, event original part, other attributes, and 26 customizable reserved fields for parsing the log message.
4. The big data analysis technology-based network security log alarm processing method of claim 1, wherein: after filtering the log in step S5, storing the stored log into the Elasticsearch database, and creating an index for log query.
5. The big data analysis technology-based network security log alarm processing method of claim 1, wherein: step S6 specifically includes: according to preset network security attack stages and attack event classification rules, log events are matched and associated for analysis according to the attack stages and the event classifications, log data are stored in a data warehouse according to the event classifications of different attack stages, and a security event analysis subject library is formed.
6. The big data analysis technology-based network security log alarm processing method of claim 1, wherein: in the step S7, the customized newly added alarm rule is compiled by manually combining logical operators such as and, or, not, containing, greater than, less than, or equal to, and screening security events in combination with the value range of the log standard field, and the customized rule is generated by dragging and selecting low codes and filling numerical values in a null manner.
7. The big data analysis technology-based network security log alarm processing method of claim 1, wherein: in the cosine similarity calculation method in step S8, the cosine value between the included angles of two vectors in a vector space is used as a measure of the difference between two individuals, the cosine value is close to 1, the included angle tends to 0, which indicates that the two vectors are more similar, the cosine value is close to 0, the included angle tends to 90 degrees, which indicates that the two vectors are more dissimilar.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110910555.XA CN113676464B (en) | 2021-08-09 | 2021-08-09 | Network security log alarm processing method based on big data analysis technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110910555.XA CN113676464B (en) | 2021-08-09 | 2021-08-09 | Network security log alarm processing method based on big data analysis technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113676464A true CN113676464A (en) | 2021-11-19 |
| CN113676464B CN113676464B (en) | 2023-07-04 |
Family
ID=78542201
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110910555.XA Active CN113676464B (en) | 2021-08-09 | 2021-08-09 | Network security log alarm processing method based on big data analysis technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113676464B (en) |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114143178A (en) * | 2021-12-03 | 2022-03-04 | 中国电信集团系统集成有限责任公司 | TR069 protocol-combined alarm root positioning visualization method and device |
| CN114257417A (en) * | 2021-11-26 | 2022-03-29 | 中国南方电网有限责任公司 | Network security data processing method and device, computer equipment and storage medium |
| CN114257414A (en) * | 2021-11-25 | 2022-03-29 | 国网山东省电力公司日照供电公司 | Intelligent network security duty method and system |
| CN114281864A (en) * | 2021-12-17 | 2022-04-05 | 东南大学 | Correlation analysis method for power network alarm information |
| CN114301712A (en) * | 2021-12-31 | 2022-04-08 | 西安交通大学 | Industrial internet alarm log correlation analysis method and system based on graph method |
| CN114416060A (en) * | 2022-03-28 | 2022-04-29 | 北京人人云图信息技术有限公司 | Complex event processing method and system for time sequence data |
| CN114430348A (en) * | 2022-02-07 | 2022-05-03 | 云盾智慧安全科技有限公司 | Web site search engine optimization backdoor identification method and device |
| CN114584365A (en) * | 2022-03-01 | 2022-06-03 | 北京优炫软件股份有限公司 | Security event analysis response method and system |
| CN114780810A (en) * | 2022-04-22 | 2022-07-22 | 中国电信股份有限公司 | Data processing method, data processing device, storage medium and electronic equipment |
| CN114826773A (en) * | 2022-06-02 | 2022-07-29 | 合肥卓讯云网科技有限公司 | User-defined log alarm method and device based on log data |
| CN114944964A (en) * | 2022-07-21 | 2022-08-26 | 北京未来智安科技有限公司 | Network security event processing method and device |
| CN115129494A (en) * | 2022-08-31 | 2022-09-30 | 浙江工业大学 | Event log collection method and system based on Windows kernel |
| CN115333916A (en) * | 2022-07-19 | 2022-11-11 | 广州爱浦路网络技术有限公司 | Network element alarm information processing method, device and storage medium in communication network |
| CN115495427A (en) * | 2022-11-22 | 2022-12-20 | 青岛远洋船员职业学院 | Log data storage method based on intelligent security management platform |
| CN115549953A (en) * | 2022-08-15 | 2022-12-30 | 国家管网集团北方管道有限责任公司 | Network security alarm method and system |
| CN115658637A (en) * | 2022-12-26 | 2023-01-31 | 北京六方云信息技术有限公司 | Log normalization processing method and device, storage medium and processor |
| CN115934782A (en) * | 2023-02-13 | 2023-04-07 | 山东星维九州安全技术有限公司 | Method for analyzing and processing security log and computer storage medium |
| CN116232751A (en) * | 2023-03-16 | 2023-06-06 | 中国华能集团有限公司北京招标分公司 | Safety alarm analysis method |
| CN116633667A (en) * | 2023-06-20 | 2023-08-22 | 云南固垒科技有限公司 | Network security log auditing method and device based on self-adaptive rule generation |
| CN118368124A (en) * | 2024-05-06 | 2024-07-19 | 中国电子科技集团公司第十五研究所 | Network security event processing method and device |
| CN114281864B (en) * | 2021-12-17 | 2024-11-19 | 东南大学 | Correlation analysis method for power network alarm information |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103546312A (en) * | 2013-08-27 | 2014-01-29 | 中国航天科工集团第二研究院七〇六所 | Massive multi-source isomerism log correlation analyzing method |
| US20140283083A1 (en) * | 2013-03-15 | 2014-09-18 | Tenable Network Security, Inc. | System and method for correlating log data to discover network vulnerabilities and assets |
| CN104539626A (en) * | 2015-01-14 | 2015-04-22 | 中国人民解放军信息工程大学 | Network attack scene generating method based on multi-source alarm logs |
| CN107766167A (en) * | 2017-10-23 | 2018-03-06 | 郑州云海信息技术有限公司 | A kind of fault log repeats to report an error the method for merger |
| CN108243060A (en) * | 2017-01-19 | 2018-07-03 | 上海直真君智科技有限公司 | A kind of network security alarm risk determination method presorted based on big data |
| CN109376532A (en) * | 2018-10-31 | 2019-02-22 | 云南电网有限责任公司 | Power network security monitoring method and system based on the analysis of ELK log collection |
| CN110620790A (en) * | 2019-10-10 | 2019-12-27 | 国网山东省电力公司信息通信公司 | Network security device linkage processing method and device |
| CN111885012A (en) * | 2020-07-03 | 2020-11-03 | 安徽继远软件有限公司 | Network situation perception method and system based on information acquisition of various network devices |
-
2021
- 2021-08-09 CN CN202110910555.XA patent/CN113676464B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140283083A1 (en) * | 2013-03-15 | 2014-09-18 | Tenable Network Security, Inc. | System and method for correlating log data to discover network vulnerabilities and assets |
| CN103546312A (en) * | 2013-08-27 | 2014-01-29 | 中国航天科工集团第二研究院七〇六所 | Massive multi-source isomerism log correlation analyzing method |
| CN104539626A (en) * | 2015-01-14 | 2015-04-22 | 中国人民解放军信息工程大学 | Network attack scene generating method based on multi-source alarm logs |
| CN108243060A (en) * | 2017-01-19 | 2018-07-03 | 上海直真君智科技有限公司 | A kind of network security alarm risk determination method presorted based on big data |
| CN107766167A (en) * | 2017-10-23 | 2018-03-06 | 郑州云海信息技术有限公司 | A kind of fault log repeats to report an error the method for merger |
| CN109376532A (en) * | 2018-10-31 | 2019-02-22 | 云南电网有限责任公司 | Power network security monitoring method and system based on the analysis of ELK log collection |
| CN110620790A (en) * | 2019-10-10 | 2019-12-27 | 国网山东省电力公司信息通信公司 | Network security device linkage processing method and device |
| CN111885012A (en) * | 2020-07-03 | 2020-11-03 | 安徽继远软件有限公司 | Network situation perception method and system based on information acquisition of various network devices |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114257414A (en) * | 2021-11-25 | 2022-03-29 | 国网山东省电力公司日照供电公司 | Intelligent network security duty method and system |
| CN114257417A (en) * | 2021-11-26 | 2022-03-29 | 中国南方电网有限责任公司 | Network security data processing method and device, computer equipment and storage medium |
| CN114143178A (en) * | 2021-12-03 | 2022-03-04 | 中国电信集团系统集成有限责任公司 | TR069 protocol-combined alarm root positioning visualization method and device |
| CN114143178B (en) * | 2021-12-03 | 2024-06-04 | 中电信数智科技有限公司 | Alarm root-cause positioning visualization method and device combined with TR069 protocol |
| CN114281864A (en) * | 2021-12-17 | 2022-04-05 | 东南大学 | Correlation analysis method for power network alarm information |
| CN114281864B (en) * | 2021-12-17 | 2024-11-19 | 东南大学 | Correlation analysis method for power network alarm information |
| CN114301712A (en) * | 2021-12-31 | 2022-04-08 | 西安交通大学 | Industrial internet alarm log correlation analysis method and system based on graph method |
| CN114430348A (en) * | 2022-02-07 | 2022-05-03 | 云盾智慧安全科技有限公司 | Web site search engine optimization backdoor identification method and device |
| CN114430348B (en) * | 2022-02-07 | 2023-12-05 | 云盾智慧安全科技有限公司 | Web site search engine optimization backdoor identification method and device |
| CN114584365A (en) * | 2022-03-01 | 2022-06-03 | 北京优炫软件股份有限公司 | Security event analysis response method and system |
| CN114416060A (en) * | 2022-03-28 | 2022-04-29 | 北京人人云图信息技术有限公司 | Complex event processing method and system for time sequence data |
| CN114780810A (en) * | 2022-04-22 | 2022-07-22 | 中国电信股份有限公司 | Data processing method, data processing device, storage medium and electronic equipment |
| CN114780810B (en) * | 2022-04-22 | 2024-02-27 | 中国电信股份有限公司 | Data processing method and device, storage medium and electronic equipment |
| CN114826773B (en) * | 2022-06-02 | 2024-04-16 | 合肥卓讯云网科技有限公司 | User-defined log alarming method and device based on log data |
| CN114826773A (en) * | 2022-06-02 | 2022-07-29 | 合肥卓讯云网科技有限公司 | User-defined log alarm method and device based on log data |
| CN115333916A (en) * | 2022-07-19 | 2022-11-11 | 广州爱浦路网络技术有限公司 | Network element alarm information processing method, device and storage medium in communication network |
| CN115333916B (en) * | 2022-07-19 | 2023-07-25 | 广州爱浦路网络技术有限公司 | Network element alarm information processing method, device and storage medium in communication network |
| CN114944964A (en) * | 2022-07-21 | 2022-08-26 | 北京未来智安科技有限公司 | Network security event processing method and device |
| CN114944964B (en) * | 2022-07-21 | 2022-10-21 | 北京未来智安科技有限公司 | Network security event processing method and device |
| CN115549953A (en) * | 2022-08-15 | 2022-12-30 | 国家管网集团北方管道有限责任公司 | Network security alarm method and system |
| CN115129494A (en) * | 2022-08-31 | 2022-09-30 | 浙江工业大学 | Event log collection method and system based on Windows kernel |
| CN115495427A (en) * | 2022-11-22 | 2022-12-20 | 青岛远洋船员职业学院 | Log data storage method based on intelligent security management platform |
| CN115658637A (en) * | 2022-12-26 | 2023-01-31 | 北京六方云信息技术有限公司 | Log normalization processing method and device, storage medium and processor |
| CN115934782B (en) * | 2023-02-13 | 2023-05-12 | 山东星维九州安全技术有限公司 | Method for analyzing and processing security log and computer storage medium |
| CN115934782A (en) * | 2023-02-13 | 2023-04-07 | 山东星维九州安全技术有限公司 | Method for analyzing and processing security log and computer storage medium |
| CN116232751A (en) * | 2023-03-16 | 2023-06-06 | 中国华能集团有限公司北京招标分公司 | Safety alarm analysis method |
| CN116633667A (en) * | 2023-06-20 | 2023-08-22 | 云南固垒科技有限公司 | Network security log auditing method and device based on self-adaptive rule generation |
| CN116633667B (en) * | 2023-06-20 | 2024-04-19 | 云南固垒科技有限公司 | Network security log auditing method and device based on self-adaptive rule generation |
| CN118368124A (en) * | 2024-05-06 | 2024-07-19 | 中国电子科技集团公司第十五研究所 | Network security event processing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113676464B (en) | 2023-07-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113676464B (en) | Network security log alarm processing method based on big data analysis technology | |
| CN112114995B (en) | Terminal abnormality analysis method, device, equipment and storage medium based on process | |
| CN107172022B (en) | APT threat detection method and system based on intrusion path | |
| CN111722984B (en) | Alarm data processing method, device, equipment and computer storage medium | |
| CN117473571B (en) | Data information security processing method and system | |
| CN114915479B (en) | Web attack stage analysis method and system based on Web log | |
| CN113094707B (en) | Lateral movement attack detection method and system based on heterogeneous graph network | |
| EP3272097B1 (en) | Forensic analysis | |
| CN113904881B (en) | Intrusion detection rule false alarm processing method and device | |
| CN109218321A (en) | A kind of network inbreak detection method and system | |
| CN115883213B (en) | APT detection method and system based on continuous time dynamic heterogeneous graph neural network | |
| CN112560029A (en) | Website content monitoring and automatic response protection method based on intelligent analysis technology | |
| CN111274218A (en) | Multi-source log data processing method for power information system | |
| WO2024051017A1 (en) | Distributed website tampering detection system and method | |
| CN117749535B (en) | Network traffic abnormality detection method and device | |
| CN113032774B (en) | Training method, device and equipment of anomaly detection model and computer storage medium | |
| CN118264473A (en) | Method and system for detecting network attack of telecommunication network signaling system | |
| CN109447177B (en) | Account clustering method and device and server | |
| CN117097571A (en) | Method, system, device and medium for detecting network transmission sensitive data | |
| CN117033501A (en) | Big data acquisition and analysis system | |
| CN110912753A (en) | Cloud security event real-time detection system and method based on machine learning | |
| CN112597498A (en) | Webshell detection method, system and device and readable storage medium | |
| CN109218305B (en) | Network evidence obtaining method and device based on alarm aggregation | |
| CN113076355A (en) | Method for sensing data security flow situation | |
| Tao et al. | An intrusion alarm data association analysis method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |