CN117792727A - Threat early warning model training and network threat early warning method, device and equipment - Google Patents
Threat early warning model training and network threat early warning method, device and equipment Download PDFInfo
- Publication number
- CN117792727A CN117792727A CN202311786204.8A CN202311786204A CN117792727A CN 117792727 A CN117792727 A CN 117792727A CN 202311786204 A CN202311786204 A CN 202311786204A CN 117792727 A CN117792727 A CN 117792727A
- Authority
- CN
- China
- Prior art keywords
- vector
- flow data
- threat
- model
- warning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012549 training Methods 0.000 title claims abstract description 81
- 238000000034 method Methods 0.000 title claims abstract description 73
- 239000013598 vector Substances 0.000 claims abstract description 276
- 230000008569 process Effects 0.000 claims abstract description 22
- 230000015654 memory Effects 0.000 claims description 25
- 238000003860 storage Methods 0.000 claims description 18
- 241000700605 Viruses Species 0.000 claims description 15
- 238000001514 detection method Methods 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 7
- 238000010276 construction Methods 0.000 claims description 2
- 230000004044 response Effects 0.000 abstract description 2
- 230000000694 effects Effects 0.000 description 11
- 238000000605 extraction Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000012544 monitoring process Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000004220 aggregation Methods 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 3
- 230000006399 behavior Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000002155 anti-virotic effect Effects 0.000 description 1
- 238000012098 association analyses Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000005111 flow chemistry technique Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000011524 similarity measure Methods 0.000 description 1
Landscapes
- Alarm Systems (AREA)
Abstract
The invention relates to the technical field of data security, and discloses a method, a device and equipment for training a threat early-warning model and early-warning network threats, wherein the training method comprises the following steps: acquiring a flow data set, wherein the flow data set is provided with a threat early warning label; extracting characteristic information of flow data, and obtaining corresponding characteristic vectors based on the characteristic information, so as to construct a characteristic vector group; traversing the characteristic vector group, determining the vector accessed each time in the traversing process as an anchor point vector, and obtaining a positive sample vector group; taking threat early warning labels corresponding to anchor point vectors as supervision, and performing supervision training on a preset threat early warning model; in the traversing process, the preset threat early-warning model is updated successively until the characteristic vector group is traversed to obtain the target threat early-warning model, and the same type of vector is obtained according to the labels of the flow data in the training process to perform model training, so that the recognition capability of the model to different data is ensured to be obtained through training, and the threat early-warning accuracy and response speed are improved.
Description
Technical Field
The invention relates to the technical field of data security model training, in particular to a method, a device and equipment for training a threat early-warning model and early-warning network threat.
Background
In the prior art, when the data security early-warning model is trained, various types of flow data are often mixed, the model is trained through the mixed flow data, so that the early-warning model obtained through training cannot quickly and accurately identify different types of flow data when the different types of flow data are identified, the effect of identifying different types of dangerous flow data by the model obtained through training is poor, and the early-warning effect when the dangerous flow is faced is not good enough.
Disclosure of Invention
In view of the above, the invention provides a method, a device and equipment for training a threat early-warning model and network threat early-warning, which are used for solving the problem that the model obtained by training cannot quickly and accurately identify different types of flow data when the early-warning model is trained.
In a first aspect, the present invention provides a method for training a threat early warning model, the method comprising:
obtaining a flow data set, the flow data set comprising: traffic data with threat early warning tags;
extracting characteristic information of the flow data set, obtaining characteristic vectors corresponding to all flow data based on the characteristic information, and constructing a characteristic vector group based on the characteristic vectors;
traversing the characteristic vector group, determining a current access vector accessed each time in the traversing process as an anchor point vector, and obtaining a positive sample vector group based on the anchor point vector;
taking threat early warning labels corresponding to flow data corresponding to the anchor point vectors as supervision, and performing supervision training on a preset threat early warning model based on the positive sample vector group;
and in the traversing process, successively updating the preset threat early-warning model until the traversing of the characteristic vector group is finished, and obtaining a target threat early-warning model.
After extracting characteristic information of flow data and obtaining corresponding characteristic vectors, anchor point vectors are determined, the vectors are classified according to labels of data corresponding to the vectors, a positive sample vector group is obtained, a preset model is trained according to the positive sample vector group, and therefore the early warning model obtained through training can be better in recognition effect and faster in distinguishing dangerous data of different types.
In an optional implementation manner, the extracting feature information of the flow data set, and obtaining feature vectors corresponding to each flow data based on the feature information, includes: and extracting the characteristic information of the current flow data in the flow data set through a residual error learning network, and converting the characteristic information into a characteristic vector corresponding to the current flow data through projection mapping.
The flow data in the data set is subjected to feature extraction through the residual error learning network, so that the feature information can be accurately extracted, and the subsequent training effect is ensured.
In an alternative embodiment, the obtaining the positive sample vector set based on the anchor point vector includes: based on threat early warning labels of the feature vectors corresponding to the flow data, determining vector categories of the feature vectors in the feature vector group; and extracting target feature vectors with the same vector category as the anchor point vector from the feature vector group, and constructing a positive sample vector group based on the target feature vectors.
The feature vectors which are the same as the labels of the data corresponding to the anchor point vectors are gathered through the labels of the data corresponding to the different feature vectors, so that the data types corresponding to the feature vectors in the constructed positive sample vector group are the same as the anchor point vectors, and the certain type of flow data can be trained in a targeted manner during the subsequent model training.
In an alternative embodiment, the threat alert tag includes: security data, website virus data, mail virus data, and protocol virus data.
Different types of flow data correspond to different labels, so that the subsequent model can be trained for various types of flow data during training, and the recognition capability of the model for different types of dangerous data is improved.
In an optional implementation manner, the monitoring and training the preset threat early-warning model based on the positive sample vector set with the threat early-warning label corresponding to the traffic data corresponding to the anchor point vector as a monitor includes:
normalizing the positive sample vector group and the anchor point vector to obtain a standard positive sample vector group and a standard anchor point vector;
taking threat early warning labels corresponding to the flow data corresponding to the anchor point vectors as supervision, and performing supervision training on a preset threat early warning model based on the standard anchor point vectors, the standard positive sample vector group and preset model parameters.
Before training, the positive sample vector set and the anchor point vector are normalized so as to meet the training requirement, and when training is performed, the threat early warning label is used as supervision to supervise and train the preset threat early warning model, so that the recognition effect of the model obtained by training on different types of flow data and the early warning capability of threat behaviors are ensured.
In a second aspect, the present invention provides a network threat early warning method, the method comprising:
collecting network real-time traffic and extracting metadata of the network real-time traffic;
extracting feature information of the metadata, and obtaining a corresponding first feature vector based on the feature information;
and inputting the first feature vector into a target threat early-warning detection model to obtain a target early-warning result, wherein the target threat early-warning detection model is trained by adopting the training method of the threat early-warning model in the first aspect or any corresponding implementation mode.
The network flow data is identified through the early warning model obtained through training, dangerous flow data can be effectively early warned, and data safety is guaranteed.
In an alternative embodiment, the collecting network real-time traffic includes: and correlating various access chains of the network data to obtain the multipath network real-time traffic.
And through correlating various access chains, the flow data of various paths is obtained, so that the safety of global flow data is ensured.
In a third aspect, the present invention provides a computer device comprising: the processor executes the computer instructions, thereby executing the training method of the threat early warning model of the first aspect or any corresponding embodiment thereof, or executing the network threat early warning method of the second aspect or any corresponding embodiment thereof.
In a fourth aspect, the present invention provides a computer readable storage medium, where computer instructions are stored on the computer readable storage medium, where the computer instructions are configured to cause a computer to perform the training method of the threat early warning model of the first aspect or any embodiment corresponding thereto, or perform the network threat early warning method of the second aspect or any embodiment corresponding thereto.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method of training a threat alert model in accordance with an embodiment of the invention;
FIG. 2 is a flow chart of a training method of another threat alert model in accordance with an embodiment of the invention;
FIG. 3 is a flow chart of a network threat early warning method in accordance with an embodiment of the invention;
FIG. 4 is a block diagram of a training device for threat alert models in accordance with an embodiment of the invention;
fig. 5 is a schematic diagram of a hardware structure of a computer device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In order to early warn the safety of the flow data, the model is often trained so that the model can distinguish the acquired data and judge whether the received flow data is dangerous flow data or not, so that early warning is timely carried out and attack by the dangerous flow data is avoided. In the related art, when training a model, various data are usually mixed and input into the model, and the model is trained, so that the trained model cannot be timely and effectively distinguished when facing different types of dangerous flow data, and the recognition speed of the dangerous flow data is not enough, so that the use effect of the trained model is not good enough.
Therefore, the embodiment of the invention provides a training method of a threat early-warning model, which is used for obtaining the vectors of the same type according to the labels of flow data to train the model, so that the recognition capability of the model to different data obtained through training is ensured, and the threat early-warning accuracy and response speed are improved.
In accordance with an embodiment of the present invention, there is provided a training method embodiment of a threat alert model, it being noted that the steps illustrated in the flowchart of the figures may be performed in a computer system, such as a set of computer executable instructions, and that although a logical sequence is illustrated in the flowchart, in some cases steps illustrated or described may be performed in a different order than that illustrated herein.
In this embodiment, a method for training a threat early-warning model is provided, which may be used in the foregoing early-warning model training process, and fig. 1 is a flowchart of a method for training a threat early-warning model according to an embodiment of the present invention, as shown in fig. 1, where the flowchart includes the following steps:
step S101, acquiring a flow data set, where the flow data set includes: traffic data with threat alert tags.
When the model is trained, a plurality of data sets which are already open-sourced and have high approval are available, the data sets comprise safe flow data and dangerous flow data for training, various types of flow data are obtained by processing the data sets which are open-sourced, wherein the flow data sets comprise safe flow and flow data corresponding to various dangerous behaviors, the flow data sets jointly form the flow data set in the step S101, and all the flow data sets are provided with self labels, namely threat early warning labels, and are used for representing the types of flow data to which the self belongs.
Step S102, extracting characteristic information of the flow data set, obtaining characteristic vectors corresponding to the flow data based on the characteristic information, and constructing a characteristic vector group based on the characteristic vectors.
The method comprises the steps of providing a large amount of flow data in a flow data set, extracting feature information in the flow data, processing the feature information after extracting the feature information corresponding to each flow data in the data set, converting the feature information into a vector form to obtain feature vectors corresponding to each flow data, extracting features of a large amount of flow data in the flow data set, converting the feature information to obtain corresponding feature vectors, and forming a feature vector group by the feature vectors.
And step S103, traversing the feature vector group, determining the current access vector accessed each time in the traversing process as an anchor point vector, and obtaining a positive sample vector group based on the anchor point vector.
After the feature vector group is obtained, traversing the feature vector group, taking each feature vector of the feature vector group as an anchor point vector in sequence, and obtaining a positive sample vector group according to threat early-warning labels of flow data corresponding to the anchor point vector after each anchor point vector is determined, wherein the threat early-warning labels of the flow data corresponding to each feature vector of the positive sample vector group are identical to the threat early-warning labels of the flow data corresponding to the anchor point vector. After the set of positive sample vectors is obtained, the model may be subsequently trained from the set of positive sample vectors corresponding to this anchor vector.
Step S104, taking threat early-warning labels corresponding to flow data corresponding to anchor point vectors as supervision, and performing supervision training on a preset threat early-warning model based on a positive sample vector group.
After the positive sample vector group is obtained by aggregation, threat early-warning labels of flow data corresponding to vectors in the positive sample vector group are identical to anchor point vectors, namely the vectors have identical threat early-warning labels, and the threat early-warning labels corresponding to the flow data corresponding to the anchor point vectors are used as supervision. And then training a preset threat early-warning model through vectors in the positive sample vector groups, wherein the trained model can effectively identify the traffic data of the type corresponding to the threat early-warning label.
Step S105, in the traversing process, the preset threat early-warning model is updated successively until the traversing of the feature vector group is finished, and the target threat early-warning model is obtained.
In the process of traversing the feature vector group, each feature vector is sequentially used as an anchor point vector, so that for each feature vector in the feature vector group, a corresponding positive sample vector group is obtained in an aggregation mode, and each new anchor point vector and a corresponding positive sample vector group are obtained, the preset threat early warning model is trained again according to the content in the step S104, and the threat early warning model is updated to further improve the recognition capability of the threat early warning model. Until the traversal of the feature vector group is finished, at this time, the threat early-warning model after the updating training of the plurality of rounds is the target threat early-warning model, and can effectively identify various types of dangerous flow data.
After extracting characteristic information of flow data and obtaining corresponding characteristic vectors, anchor point vectors are determined, the vectors are classified according to labels of data corresponding to the vectors, a positive sample vector group is obtained, a preset model is trained according to the positive sample vector group, and therefore the early warning model obtained through training can be better in recognition effect and faster in distinguishing dangerous data of different types.
According to an embodiment of the present invention, another embodiment of a training method for a threat early-warning model is provided, which may be used in the foregoing early-warning model training process, and fig. 2 is a flowchart of another training method for a threat early-warning model according to an embodiment of the present invention, as shown in fig. 2, where the flowchart includes the following steps:
step S201, acquiring a flow data set, where the flow data set includes: traffic data with threat alert tags. Please refer to step S101 in the embodiment shown in fig. 1 in detail, which is not described herein.
Step S202, extracting characteristic information of a flow data set, obtaining characteristic vectors corresponding to each flow data based on the characteristic information, and constructing a characteristic vector group based on the characteristic vectors.
Specifically, in step S202, feature information of the flow data set is extracted, and feature vectors corresponding to each flow data are obtained based on the feature information, including: and extracting the characteristic information of the current flow data in the flow data set through the residual error learning network, and converting the characteristic information into a characteristic vector corresponding to the current flow data through projection mapping.
When the feature extraction is performed, the same feature network parameter weight can be shared through residual network learning, so that the feature extraction is performed on the input data, namely, the feature information corresponding to each flow data in the flow data set is extracted. Before extracting the characteristic information of the flow data, the flow data can be enhanced so as to facilitate the extraction of the characteristic information. After the feature information corresponding to each flow data is extracted, the feature information is mapped into a feature vector suitable for contrast loss through projection mapping.
And step S203, traversing the feature vector group, determining the current access vector accessed each time in the traversing process as an anchor point vector, and obtaining a positive sample vector group based on the anchor point vector.
Specifically, in step S203, a positive sample vector set is obtained based on the anchor point vector, including:
based on threat early warning labels of the feature vectors corresponding to the flow data, determining vector categories of the feature vectors in the feature vector group;
and extracting target feature vectors with the same vector category as the anchor point vectors from the feature vector group, and constructing a positive sample vector group based on the target feature vectors.
It will be understood that each feature vector in the feature vector set is obtained by extracting features from flow data in the flow data set, so that each feature vector corresponds to one flow data, and each flow data has a threat early warning label, and a vector class is determined according to the threat early warning label. After the anchor point vector is determined, threat early warning labels of flow data corresponding to the anchor point vector are obtained, vectors with the same threat early warning labels of the flow data corresponding to the anchor point vector are searched in the characteristic vector group, and the vectors are combined into a positive sample vector group.
Specifically, the threat early warning label of the traffic data may include: security data, website virus data, mail virus data, and protocol virus data. These tags are used to indicate which type of dangerous data the traffic data belongs to, such as website virus data, mail virus data or protocol virus data, and the traffic data may also be security data, and in particular implementations, the tags may also include other types of dangerous data, without limitation, the traffic data being determined by threat alert tags to be of that type in particular.
And step S204, taking threat early-warning labels corresponding to flow data corresponding to anchor point vectors as supervision, and performing supervision training on a preset threat early-warning model based on the positive sample vector group.
Specifically, in step S204, the method includes:
and step S2041, normalizing the positive sample vector group and the anchor point vector to obtain a standard positive sample vector group and a standard anchor point vector.
After the positive sample vector group is obtained, the characteristic vector and the anchor point vector in the positive sample vector group are normalized, so that the formats of the characteristic vector and the anchor point vector in the positive sample vector group meet the requirement of model training.
Step S2042, taking threat early warning labels corresponding to flow data corresponding to anchor point vectors as supervision, and performing supervision training on a preset threat early warning model based on standard anchor point vectors, standard positive sample vector sets and preset model parameters.
When the monitoring training is carried out, because each feature vector in the positive sample vector group has the threat early warning label which is the same as the anchor vector, namely the vectors are all feature vectors corresponding to the same type of flow data, the early warning model obtained by the monitoring training can effectively identify the type of flow data so as to carry out early warning, and the monitoring training is carried out on the preset threat early warning model based on the standard anchor vector, the standard positive sample vector group and the preset model parameters.
Specifically, during training, the anchor vector can be defined as z i Defining a set of positive sample vectors asRepresentation and z i The formula for supervised training is as follows:
wherein,representing anchor z i Is equal to the positive sample number, τ>0 scalar temperature coefficient, for each positive sample vector group corresponding to each anchor point vector, presetting a threat early warning model L by the formula SCL Training is performed. In contrast learning, the temperatureThe coefficient of degree τ plays a key role in contrast loss, τ is a scale that controls the similarity measure between samples, and can affect the distribution and clustering effect of the samples in the feature space. When the temperature coefficient tau is larger, the contrast loss is more focused on distinguishing the differences among different categories, namely focusing on the characteristics of category layers, samples belonging to different categories in the characteristic space are more dispersed, and the distance between groups is increased, so that the distinction among the categories is emphasized in the representation learning, and the semantic meaningful category distinction is facilitated. In contrast, when the temperature coefficient τ is smaller, the contrast loss focuses more on the difference between the examples, that is, focuses more on the characteristics of the example layers, so that the samples in the same category are more dispersed in the characteristic space, the distance between the examples is increased, the model can better distinguish different examples in the same category, and the characteristics of each example can be accurately captured.
Step S205, in the traversing process, the preset threat early-warning model is updated successively until the traversing of the feature vector group is finished, and the target threat early-warning model is obtained.
The specific following formula in step S204 can be understood by the following formula:
after training is carried out by adopting the positive sample vector group corresponding to each anchor point vector to obtain a trained preset threat early warning model, the training result corresponding to each positive sample vector group is accumulated to achieve the effect of updating the training effect, and after all the positive sample vector groups are trained by continuously updating and accumulating, a target threat early warning model is obtained and used for analyzing and early warning traffic data and guaranteeing the safety of network traffic.
According to the training method embodiment of the threat early warning model, after the characteristic information of the flow data is extracted and the corresponding characteristic vectors are obtained, the anchor point vectors are determined, the vectors are classified according to the labels of the data corresponding to the vectors, a positive sample vector group is obtained, the preset model is trained according to the positive sample vector group, and therefore the early warning model obtained through training can be better in recognition effect and faster in distinguishing different types of dangerous data.
According to an embodiment of the present invention, there is further provided a cyber threat early warning method, which can perform cyber threat early warning through the target threat early warning model provided in the above embodiment, and fig. 3 is a flowchart of a cyber threat early warning method according to an embodiment of the present invention, as shown in fig. 3, where the flowchart includes the following steps:
step S301, collecting network real-time traffic and extracting metadata of the network real-time traffic.
And receiving data traffic received by the whole network system in real time through a data monitoring port of the network system, and extracting metadata of the data traffic.
Specifically, the method for collecting the real-time traffic of the network comprises the following steps: and correlating various access chains of the network data to obtain the multipath network real-time traffic.
By associating various access chains of network data, network traffic of various paths in the network system can be obtained, for example, traffic data of the access system is actively detected through traffic detection, the traffic data is obtained through a log of the network security device, and the traffic data is obtained through an antivirus module of the network system. The safety of the flow data is comprehensively ensured by collecting the flow data of various paths.
Step S302, extracting feature information of the metadata, and obtaining a corresponding first feature vector based on the feature information.
The method for extracting the obtained metadata to obtain the corresponding feature vector and converting the extracted metadata into the corresponding vector may refer to the above embodiment, and will not be described herein.
Step S303, inputting the first feature vector into a target threat early-warning detection model to obtain a target early-warning result.
And detecting the feature vector through the trained model so as to determine whether the flow data is safe or not and determining which type of dangerous flow data the flow data specifically belongs to if the flow data is dangerous data. The network flow data is identified through the early warning model obtained through training, dangerous flow data can be effectively early warned, and data safety is guaranteed.
The present invention provides an exemplary example to assist understanding of the model training method provided in the foregoing embodiment, and when performing security early warning of network data, the early warning is usually performed by a system, which is a set of system solutions for solving the problem of complex enterprise information security developed based on big data technology, machine learning and association analysis, and the system logic architecture includes a data acquisition layer, a data processing layer, a data storage layer, an interface layer and a service interaction layer.
The data acquisition layer adopts an advanced network flow processing architecture in the industry, processes the real-time acquisition of the meganetwork flow in a second level, acquires logs of safety equipment of the network, and realizes the functions of application analysis, session restoration and the like;
the data processing layer is used for carrying out data extraction, data cleaning, data conversion and other processes on the accessed data content (structured data record, semi-structured text, unstructured file and the like) according to a standardized standard flow so as to realize the increment of data, data preparation and data abstraction;
data storage layer: and the accessed data resources are subjected to aggregation storage in a hierarchical, classifying and layering mode, so that unified fusion of various network security data is realized, and the functions of an original library, a resource library, a subject library, a configuration library and the like are provided, so that storage, analysis query and real-time streaming data processing analysis capability of mass data are provided.
Interface layer: through standardized data management, the transparency, the manageability and the controllability of data resources are realized, the data landing is perfected, the data processing flow is standardized, and the safe use of the data is ensured. Providing data resources for business interaction layer, including original data, business knowledge and metadata, indexing data resource catalogue, etc
The business interaction layer comprises six functional modules of data analysis, graph mining, threat detection, retrospective evidence obtaining, threat information and situation awareness.
When the model is trained, the correlation algorithm is utilized to correlate according to the network behavior data, the network communication protocol data characteristics (comprising the data packet length and the data packet arrival interval time sequence) are extracted, and the machine learning algorithm is utilized to train the threat intelligent early warning model.
Using an encrypted traffic data set with high acceptance that is already sourced: such as CTU-13 data set, UNSW-NB15 data set, or ISCX VPN-nonVPN data set, to obtain data samples, to obtain bi-directional traffic sessions for network communications using dynamic sandboxed techniques, to obtain normal traffic and a set of network threat traffic sessions. First, distinguishing the original data packets: grouping the data packets in a quintuple form according to the source IP address, the target IP address, the source port, the target port and the protocol type, deleting header information of an Ethernet layer, unifying header length of a transmission layer, unifying the data packet length by cutting or filling application layer data, normalizing each byte in the data packets, and normalizing each byte to a section [0,1] for model training.
Feature information extraction of input data using ResNet network learning branches sharing the same feature network parameter weights, the contrast learning branches first enhancing data v 1 And v 2 Extracting characteristic representation information by characteristic extraction, and mapping the characteristic representation information into vector representation suitable for contrast loss through one projectionAnd further a normalization operation is used on vector z.
Recording input data as anchor point x i Other data of the same class as the anchor point may be noted as positive samplesData different from the anchor point category can be marked as negative sample +.>The feature representation vector z after mapping the comparison learning branch also meets the following requirementsThe symbol definition of the contrast learning can be that +.>Andrespectively denoted as anchor point z i Corresponding positive and negative samples. And obtaining a trained target threat early warning model through supervised learning.
The embodiment also provides a training device for threat early warning models, which is used for implementing the foregoing embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
The embodiment provides a training device for threat early warning model, whose structural block diagram is shown in fig. 4, comprising:
a flow data acquisition module 401, configured to acquire a flow data set, where the flow data set includes: traffic data with threat early warning tags;
the feature information construction module 402 is configured to extract feature information of a flow data set, obtain feature vectors corresponding to each flow data based on the feature information, and construct a feature vector group based on the feature vectors;
the anchor point vector determining module 403 is configured to traverse the feature vector set, determine a current access vector accessed each time in the traversal process as an anchor point vector, and obtain a positive sample vector set based on the anchor point vector;
the model supervision and training module 404 is configured to supervise a preset threat early warning model based on a positive sample vector set by taking a threat early warning label corresponding to flow data corresponding to an anchor point vector as supervision;
the target model determining module 405 is configured to update the preset threat early-warning model successively in the traversing process until the traversing of the feature vector set is completed, so as to obtain a target threat early-warning model.
In some optional embodiments, extracting feature information of the flow data set, and obtaining feature vectors corresponding to each flow data based on the feature information includes: and extracting the characteristic information of the current flow data in the flow data set through the residual error learning network, and converting the characteristic information into a characteristic vector corresponding to the current flow data through projection mapping.
In some alternative embodiments, deriving the set of positive sample vectors based on the anchor vector includes: based on threat early warning labels of the feature vectors corresponding to the flow data, determining vector categories of the feature vectors in the feature vector group; and extracting target feature vectors with the same vector category as the anchor point vectors from the feature vector group, and constructing a positive sample vector group based on the target feature vectors.
In some alternative embodiments, the threat alert tag includes: security data, website virus data, mail virus data, and protocol virus data.
In some optional embodiments, taking threat early warning labels corresponding to traffic data corresponding to anchor point vectors as supervision, and performing supervision training on a preset threat early warning model based on a positive sample vector group, the method comprises the following steps: normalizing the positive sample vector group and the anchor point vector to obtain a standard positive sample vector group and a standard anchor point vector; and taking threat early warning labels corresponding to flow data corresponding to the anchor point vectors as supervision, and performing supervision training on a preset threat early warning model based on the standard anchor point vectors, the standard positive sample vector group and preset model parameters.
Further functional descriptions of the above respective modules and units are the same as those of the above corresponding embodiments, and are not repeated here.
The training device of the threat alert model in this embodiment is presented in the form of functional units, where the units are ASIC (Application Specific Integrated Circuit ) circuits, processors and memories executing one or more software or fixed programs, and/or other devices that can provide the above described functionality.
The embodiment of the invention also provides computer equipment, which is provided with the training device of the threat early warning model shown in the figure 4.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a computer device according to an alternative embodiment of the present invention, as shown in fig. 5, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are communicatively coupled to each other using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the computer device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In some alternative embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple computer devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). One processor 10 is illustrated in fig. 5.
The processor 10 may be a central processor, a network processor, or a combination thereof. The processor 10 may further include a hardware chip, among others. The hardware chip may be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general-purpose array logic, or any combination thereof.
Wherein the memory 20 stores instructions executable by the at least one processor 10 to cause the at least one processor 10 to perform a method for implementing the embodiments described above.
The memory 20 may include a storage program area that may store an operating system, at least one application program required for functions, and a storage data area; the storage data area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, memory 20 may optionally include memory located remotely from processor 10, which may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
Memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk, or solid state disk; the memory 20 may also comprise a combination of the above types of memories.
The computer device further comprises input means 30 and output means 40. The processor 10, memory 20, input device 30, and output device 40 may be connected by a bus or other means, for example in fig. 5.
The input device 30 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the computer apparatus, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointer stick, one or more mouse buttons, a trackball, a joystick, and the like. The output means 40 may include a display device, auxiliary lighting means (e.g., LEDs), tactile feedback means (e.g., vibration motors), and the like. Such display devices include, but are not limited to, liquid crystal displays, light emitting diodes, displays and plasma displays. In some alternative implementations, the display device may be a touch screen.
The embodiments of the present invention also provide a computer readable storage medium, and the method according to the embodiments of the present invention described above may be implemented in hardware, firmware, or as a computer code which may be recorded on a storage medium, or as original stored in a remote storage medium or a non-transitory machine readable storage medium downloaded through a network and to be stored in a local storage medium, so that the method described herein may be stored on such software process on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, a solid state disk or the like; further, the storage medium may also comprise a combination of memories of the kind described above. It will be appreciated that a computer, processor, microprocessor controller or programmable hardware includes a storage element that can store or receive software or computer code that, when accessed and executed by the computer, processor or hardware, implements the methods illustrated by the above embodiments.
Although embodiments of the present invention have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope of the invention as defined by the appended claims.
Claims (10)
1. A method of training a threat alert model, the method comprising:
obtaining a flow data set, the flow data set comprising: traffic data with threat early warning tags;
extracting characteristic information of the flow data set, obtaining characteristic vectors corresponding to all flow data based on the characteristic information, and constructing a characteristic vector group based on the characteristic vectors;
traversing the characteristic vector group, determining a current access vector accessed each time in the traversing process as an anchor point vector, and obtaining a positive sample vector group based on the anchor point vector;
taking threat early warning labels corresponding to flow data corresponding to the anchor point vectors as supervision, and performing supervision training on a preset threat early warning model based on the positive sample vector group;
and in the traversing process, successively updating the preset threat early-warning model until the traversing of the characteristic vector group is finished, and obtaining a target threat early-warning model.
2. The method according to claim 1, wherein the extracting feature information of the flow data set, and obtaining feature vectors corresponding to each flow data based on the feature information, includes:
and extracting the characteristic information of the current flow data in the flow data set through a residual error learning network, and converting the characteristic information into a characteristic vector corresponding to the current flow data through projection mapping.
3. The method of claim 1, wherein the deriving the set of positive sample vectors based on the anchor vector comprises:
based on threat early warning labels of the feature vectors corresponding to the flow data, determining vector categories of the feature vectors in the feature vector group;
and extracting target feature vectors with the same vector category as the anchor point vector from the feature vector group, and constructing a positive sample vector group based on the target feature vectors.
4. The method of claim 1, wherein the threat alert tag comprises:
security data, website virus data, mail virus data, and protocol virus data.
5. The method of claim 1, wherein the supervising the pre-set threat early-warning model based on the positive sample vector set with the threat early-warning label corresponding to the traffic data corresponding to the anchor vector as a supervision, comprises:
normalizing the positive sample vector group and the anchor point vector to obtain a standard positive sample vector group and a standard anchor point vector;
taking threat early warning labels corresponding to the flow data corresponding to the anchor point vectors as supervision, and performing supervision training on a preset threat early warning model based on the standard anchor point vectors, the standard positive sample vector group and preset model parameters.
6. A network threat pre-warning method, the method comprising:
collecting network real-time traffic and extracting metadata of the network real-time traffic;
extracting feature information of the metadata, and obtaining a corresponding first feature vector based on the feature information;
inputting the first feature vector into a target threat early-warning detection model to obtain a target early-warning result, wherein the target threat early-warning detection model is trained by the training method of the threat early-warning model according to any one of claims 1-5.
7. The method of claim 6, wherein the collecting network real-time traffic comprises:
and correlating various access chains of the network data to obtain the multipath network real-time traffic.
8. A training device for threat alert models, the device comprising:
a flow data acquisition module, configured to acquire a flow data set, where the flow data set includes: traffic data with threat early warning tags;
the characteristic information construction module is used for extracting characteristic information of the flow data set, obtaining characteristic vectors corresponding to all flow data based on the characteristic information, and constructing a characteristic vector group based on the characteristic vectors;
the anchor point vector determining module is used for traversing the characteristic vector group, determining the current access vector accessed each time in the traversing process as an anchor point vector, and obtaining a positive sample vector group based on the anchor point vector;
the model supervision training module is used for taking threat early warning labels corresponding to the flow data corresponding to the anchor point vectors as supervision and carrying out supervision training on a preset threat early warning model based on the positive sample vector group;
and the target model determining module is used for successively updating the preset threat early-warning model in the traversing process until the traversing of the feature vector group is finished, so as to obtain a target threat early-warning model.
9. A computer device, comprising:
a memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the method of training the threat early warning model of any of claims 1 to 5 or the method of cyber threat early warning of claims 6-7.
10. A computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the training method of the threat alert model of any of claims 1 to 5 or the cyber threat alert method of claims 6-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311786204.8A CN117792727A (en) | 2023-12-22 | 2023-12-22 | Threat early warning model training and network threat early warning method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311786204.8A CN117792727A (en) | 2023-12-22 | 2023-12-22 | Threat early warning model training and network threat early warning method, device and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117792727A true CN117792727A (en) | 2024-03-29 |
Family
ID=90401215
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311786204.8A Pending CN117792727A (en) | 2023-12-22 | 2023-12-22 | Threat early warning model training and network threat early warning method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117792727A (en) |
-
2023
- 2023-12-22 CN CN202311786204.8A patent/CN117792727A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11614990B2 (en) | Automatic correlation of dynamic system events within computing devices | |
| CN110351150B (en) | Fault source determination method and device, electronic equipment and readable storage medium | |
| CN111191767B (en) | Vectorization-based malicious traffic attack type judging method | |
| CN112866023B (en) | Network detection method, model training method, device, equipment and storage medium | |
| CN111309539A (en) | Abnormity monitoring method and device and electronic equipment | |
| CN106815125A (en) | A kind of log audit method and platform | |
| WO2019084072A1 (en) | A graph model for alert interpretation in enterprise security system | |
| CN113645232A (en) | Intelligent flow monitoring method and system for industrial internet and storage medium | |
| CN106534146A (en) | Safety monitoring system and method | |
| US20170295068A1 (en) | Logical network topology analyzer | |
| CN109587125A (en) | Network security big data analysis method, system and related device | |
| CN110543506A (en) | Data analysis method and device, electronic equipment and storage medium | |
| CN112487208A (en) | Network security data association analysis method, device, equipment and storage medium | |
| CN114329450A (en) | Data security processing method, device, equipment and storage medium | |
| CN114116811A (en) | Log processing method, device, equipment and storage medium | |
| CN111767739B (en) | PPTL-based system 3 WeChat group on-line monitoring method and system | |
| CN113282920B (en) | Log abnormality detection method, device, computer equipment and storage medium | |
| CN113886829A (en) | Method and device for detecting defect host, electronic equipment and storage medium | |
| WO2024088025A1 (en) | Automated 5gc network element management method and apparatus based on multi-dimensional data | |
| CN117792727A (en) | Threat early warning model training and network threat early warning method, device and equipment | |
| CN114124509B (en) | Spark-based network abnormal flow detection method and system | |
| EP4254237A1 (en) | Security data processing device, security data processing method, and computer-readable storage medium for storing program for processing security data | |
| CN116186019B (en) | Function integrated intelligent analysis method and device applied to civil aircraft avionics system | |
| CN118487872B (en) | Nuclear power industry-oriented network abnormal behavior detection and analysis method | |
| CN117768193A (en) | Safety monitoring method, device, equipment and medium for industrial control network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |