CN112733954A

CN112733954A - Abnormal traffic detection method based on generation countermeasure network

Info

Publication number: CN112733954A
Application number: CN202110072729.XA
Authority: CN
Inventors: 黎文伟; 岳子乔
Original assignee: Hunan University
Current assignee: Hunan University
Priority date: 2021-01-20
Filing date: 2021-01-20
Publication date: 2021-04-30

Abstract

The invention relates to an abnormal traffic detection method based on a generation countermeasure network, which is used for detecting abnormal network traffic of unknown types by learning normal traffic samples. The invention mainly comprises the following steps: (1) providing an abnormal flow detection method framework based on a generation countermeasure network; (2) a preprocessing method based on network traffic characteristics is provided. Compared with the prior art, the abnormal traffic detection method based on the generation countermeasure network provided by the invention can still realize the detection capability under the condition of facing the abnormal traffic of unknown types, and can provide better performance in statistics and calculation. The method is feasible and effective, and the trained model can achieve good identification accuracy and anti-interference capability.

Description

Abnormal traffic detection method based on generation countermeasure network

Technical Field

The invention relates to the field of deep learning and anomaly detection, in particular to an anomaly traffic detection method based on a generation countermeasure network.

Background

The 46 th statistical report of the development conditions of the Chinese Internet, which is issued by a China Internet information center (CNNIC), shows that as long as 6 months in 2020, the scale of the netizen in China reaches 9.40 hundred million, and the internet popularity reaches 67.0 percent compared with 3625 ten thousand in 3 months in 2020, and the development of the internet in China is still in and will be in an important strategic opportunity for a long time. However, people are also confronted with increasingly serious network security problems while enjoying the great convenience brought by internet technology and new internet businesses. The development of computer networks has greatly exacerbated network security problems, particularly in modern network environments and advanced computing devices. With the arrival of the information era, abnormal traffic in the network is increased rapidly, the problem of network congestion is increasingly highlighted, the load of the network traffic is reduced through abnormal traffic detection, and the problem of network congestion is relieved. The abnormal flow detection is a technology for discovering abnormality by collecting and analyzing information of a protection system, mainly by monitoring a computer system and a network in real time, discovering and identifying abnormal flow in network flow, giving an abnormal flow alarm, and regarding the abnormal flow detection as a two-classification problem for distinguishing normal or abnormal.

Firewalls cannot deal with all network security issues individually, so Intrusion Detection Systems (IDS), which are an important way to work in conjunction with firewalls, have become one of the hot research directions for network security today. The traditional abnormal flow detection adopts methods such as threshold setting, feature detection and statistics, and an expert system based on the methods and an intrusion detection system based on a rule base have great limitations, so a machine learning technology capable of fitting a complex function becomes an excellent solution for the intrusion detection system. Meanwhile, although Machine learning has been widely studied for network anomaly detection, the detection accuracy is not significantly improved by using conventional Machine learning techniques, such as Support Vector Machine (SVM), random forest and adaboost, which are observed in the evaluation of NSL-KDD dataset, and is less than 83%. With the improvement of computer performance and the rise of machine learning methods, deep learning is used as an important branch of the machine learning method and is gradually applied to network anomaly detection, and the characteristic of the characteristic learning can automatically extract the characteristics of a data set, so that the characteristic engineering is greatly simplified; meanwhile, the deep learning application method in the aspects of images and natural languages which have good expression in the fields of data classification and anomaly detection can be used for reference.

A Generative Adaptive Network (GAN) is an excellent generative model, and can learn high-dimensional and complex data distribution without relying on any prior assumption. The training of the builder is promoted by fighting the mutual game of the network and the builder, and the strong performance makes the training of the builder a hotspot of research in recent years and achieves remarkable research results in a plurality of application fields.

The article "Schleglt, SeebckP, Waldstein S M, equivalent, upstream analysis Detection with general adaptive Networks to Guide Marker Discovery [ C ] International reference on Information Processing in Medical imaging. Springer, Cham, 2017" proposes a framework to realize the Detection of abnormal images by using the idea of generating countermeasure Networks.

In view of the fact that emerging attack means are more and more complex, and the conventional technology can only detect some existing abnormal attack behaviors (worms, gray pigeons, trojans, DDOS, and the like), can not detect unknown abnormal behaviors and recently-occurring abnormal behaviors in the network, or can cause huge errors in the judgment of the real-time network once the historical data of the feature library which needs to be updated in real time is out of date. Therefore, it is required to have the capability of detecting an abnormal flow rate of unknown type on the premise that the known abnormal flow rate can be detected.

Based on the above, the present patent aims to propose a method for detecting abnormal traffic based on a generation countermeasure network, which can still embody the detection capability in the case of the abnormal traffic of unknown kind.

Disclosure of Invention

The invention provides an abnormal flow detection method based on a generation countermeasure network, which can achieve the excellent effect of detecting abnormal flow by learning normal flow characteristics. The method mainly comprises the following steps:

(1) providing an abnormal flow detection framework based on a generation countermeasure network;

(2) a preprocessing method based on network traffic characteristics is provided.

The specific contents are as follows:

(1) an abnormal traffic detection framework based on a generative countermeasure network is proposed, this networkThe structure of the collaterals is shown in figure 1. The model consists of a preprocessor, a generator based on a self-coding structure, a countermeasure network and a reconstruction encoder, wherein the preprocessor is used for preprocessing the flow S and generating a flow characteristic image W; a generator receives a feature image W generated after preprocessing, a one-dimensional feature vector Z of the feature image W is generated in the generator through an encoder, and a feature image W' is generated through a decoder; inputting the generated feature image W 'into a reconstruction encoder to generate a one-dimensional feature vector Z'; the flow characteristic image W generated by the network input preprocessor and the characteristic image W 'generated by the generator are resisted, information feedback is output to promote the training of the generator until an image close to the characteristic image W is generated finally, and the minimum error is kept between the one-dimensional characteristic vector Z and the one-dimensional characteristic vector Z' as far as possible; when a model receives an abnormal sample in a testing stage, because an encoder, a decoder module and a reconstruction encoder in a generator in the model are not suitable for the abnormal sample, the difference between a one-dimensional vector Z obtained at the moment and a one-dimensional vector Z 'obtained by encoding of a reconstruction encoder is very obvious, a threshold value theta is set by comparing the difference between Z and Z', and the threshold value theta is compared with the threshold value theta₁To determine whether the traffic is abnormal.

In the training process, firstly, the generator network parameters are fixed, the flow characteristic image W generated by the preprocessor and the characteristic image W 'generated by the generator are input into the countermeasure network, and supervision training is carried out to adjust parameters of the countermeasure network so as to better distinguish the flow characteristic image W from the generated image W'. And then parameters of the countermeasure network are fixed, a generator and a reconstruction encoder module are trained, so that the characteristic image W ' generated by the generator is closer to the characteristic image W before input, the countermeasure network can not make correct judgment finally, and a one-dimensional vector Z ' obtained by the characteristic image W ' generated by the generator through a reconstruction encoder is closer to a one-dimensional vector Z obtained by the characteristic image W through reconstruction encoding. With the increase of the training times, the countermeasure network and the generator reach balance, the characteristic image W' generated by the generator gradually approaches the preprocessed characteristic image W, at the moment, the accuracy approaches 0.5, the countermeasure network cannot be distinguished, and the true or false cannot be judged. It is noted that since pattern collapse is likely to occur during training of the countermeasure network, spectral normalization (spectral normalization) is added to the last layer of the countermeasure network to stabilize the training.

In training, the challenge network is trained alternately with the generator reconstruction encoder module. The countermeasure network is essentially similar to a classifier, with the output layer being a fully connected layer output, with the purpose of determining whether the input image is the generated feature image W' or the original image W.

Wherein a loss function L is used_DTraining a countermeasure network in the abnormal flow detection model, defining y as a real label value 0 or 1, defining Adv as the countermeasure network, and then L_DCan be expressed as:

L_D＝-ylog(Adv(W'))-(1-y)log(1-Adv(W')) (1)

loss function L of the generator_GThe medicine consists of three parts:

the first part is to generate reconstruction loss L of W and W' in the network_conAnd the flow characteristic image is used for reducing the difference between the preprocessed flow characteristic image W and the reconstructed image W' generated by the generator on the pixel level. Expressed as:

L_con＝||W-W'||₁ (2)

second part in the countermeasure network, for optimization loss L in terms of image features_advAnd the method is used for reducing the difference between the preprocessed flow characteristic image W and a reconstructed image W' generated by the generation network. Expressed as:

L_adv＝||Adv(W)-Adv(W')||₂ (3)

in the third part, aiming at the one-dimensional vector Z ' obtained by encoding the reconstructed feature image W ', the one-dimensional vector Z ' expected to be obtained by encoding the image W can have no difference with the one-dimensional vector Z obtained by encoding the image W for normal flow data, so that an error optimization loss function L between potential vectors is introduced_encExpressed as:

L_enc＝||Z-Z'||₁ (4)

using a loss function L for the entire generative network model_GTraining is carried out:

L_G＝αL_enc+βL_con+δL_adv (5)

where α, β, δ are weights that adjust the assignment of each loss, and need to be tried in experiments.

(2) Providing a preprocessing method based on network flow characteristics; in a preprocessor, carrying out batch processing on original flow, and counting the basic characteristics of TCP connection for collected flow data, wherein the basic characteristics comprise continuous duration, protocol type of TCP connection, network service type of a target host, normal or error state representation of network connection, byte number of data from a source host to the target host, byte number of data from the target host to the source host and the number of error segments; the content characteristics of TCP connection comprise the times of accessing sensitive files and directories of the system, a sign of whether login is successful or not, whether super user authority is obtained or not, the access times of root users, the times of file creation operation, the times of accessing control files and the times of outbound connection in the last two seconds in an FTP session; the network flow statistical characteristics based on time comprise the number of connections with the same target host as the current connection, the number of connections with the same service as the current connection in the last two seconds, the percentage of connections with the same service as the current connection and the percentage of connections with different service from the current connection with the same target host in the last two seconds; the statistical characteristics of the network flow based on the host computer comprise the number of the connections with the same target host computer in the first 100 connections, the number of the connections with the same service as the current connection and the same target host computer in the first 100 connections, the percentage of the connections with the same service as the current connection and the same source port as the current connection in the first 100 connections, the percentage of the connections with the same target host computer in the first 100 connections, the percentage of the connections with the same service as the current connection and the different source host computer in the current connection in the connections with the same target host computer in the first 100 connections, the percentage of the connections with the same SYN error in the current connection and the percentage of the connections with the same target host computer in the first 100 connections, and the connections with the same service as the current connection and the target host computer in the first 100 connections, percentage of connections with SYN error, percentage of connections with REJ error, among the first 100 connections, with the same target host as the current connection, and percentage of connections with REJ error, among the first 100 connections, with the same service as the current connection, with the same target host.

For continuous values in the features, considering that the data are widely different from each other, the standard deviation normalization is used for processing, the data are transformed into the range with the mean value of 0 and the standard deviation of 1 by transforming the original data, and the transformation function is as follows:

x^*＝(x-u)/σ (6)

where x represents the data value, u represents the mean of the feature for all sample data, and σ represents the standard deviation of the feature for all sample data.

For discrete values in the features, one-hot coding (one-hot) is used to expand attribute dimensions, which is the most accurate, and on the premise of retaining all information, no additional information is added, and all variables are processed in a preprocessor in this way, so that the dimensions of data are greatly increased, all information of original data is completely retained, and missing values are not considered. By the method, the network flow characteristics are expanded, and then the one-dimensional network flow characteristics are converted into a two-dimensional network flow characteristic image through a reshape function.

Compared with the prior art, the technical scheme at least has the following remarkable effects:

1. the invention provides an abnormal flow detection method based on a generated countermeasure network (GAN). A self-coding structure model based on a convolutional neural network is used for designing a functional module, and the whole method is composed of a preprocessor, a generator, the countermeasure network and a reconstruction encoder. The method not only reduces the difference between the flow characteristic image and the reconstructed flow characteristic image, but also focuses on the difference between the one-dimensional vector obtained by the flow characteristic image generated by the reduction preprocessor and the one-dimensional vector obtained by the flow characteristic image generated by the generator through reconstruction coding.

2. Compared with the traditional scheme, the invention provides an effective and novel abnormal flow detection method capable of identifying unknown types under the training of normal flow samples, and better performance can be provided in statistics and calculation.

3. The invention provides a preprocessing method based on network flow characteristics. In the preprocessor, original traffic is processed in batch, and for collected traffic data, the basic characteristics of TCP connection, the content characteristics of TCP connection, the time-based network traffic statistical characteristics and the host-based network traffic statistical characteristics are counted. For continuous values in the features, considering that the difference between data is large, standard deviation standardization is used for processing, for discrete values in the features, one-hot coding (one-hot) is used, attribute dimensionality is expanded, and then a reshape function is used for converting the one-dimensional flow features into two-dimensional flow feature images.

Drawings

FIG. 1 is a schematic diagram of a framework structure of an abnormal traffic detection method based on a generation countermeasure network according to the present invention;

FIG. 2 is a diagram of the network sub-modules of the present invention, an abnormal traffic detection method based on generation of countermeasure network;

Detailed Description

The invention relates to an abnormal flow detection method based on a generation countermeasure network. For convenience of explanation, the present implementation describes a specific implementation of the present invention by taking NLS-KDD traffic data set as an example, but those skilled in the art should understand that the technical solution of the present patent application does not limit the kind of network traffic data. These embodiments are merely to explain the technical principles of the present invention and are not intended to limit the scope of the present invention.

The embodiment can be implemented according to the following steps, is not limited to any programming language, in this example, a python programming language is taken as an example, and the model is built on a pytorch deep learning platform, specifically including the following steps:

the method comprises the following steps: implementing an abnormal traffic detection model based on generating a countermeasure network

Fig. 1 is a general framework structure of an abnormal traffic detection scheme based on a generation countermeasure network according to the present invention, and fig. 2 is a network sub-module structure, i.e., a preprocessor, a generator, a countermeasure network and a reconstruction encoder, of an abnormal traffic detection method based on a generation countermeasure network according to the present invention. The input data item is NLS-KDD data set data, and the output data item is a difference value between a one-dimensional vector Z obtained by encoding a flow characteristic image W through an encoder submodule in a generator and a one-dimensional vector Z 'obtained by encoding a characteristic image W' generated by the generator through a reconstruction encoder.

Step two: data set preparation.

And randomly selecting M (more than ten thousand) positive sample flow data from an NLS-KDD train data set A to generate a training set, and randomly selecting N flow data from an NLS-KDD test data set B in any proportion to generate a test set.

Step three: and (5) training an abnormal flow detection model.

Training is carried out in the abnormal flow detection model in the first step, and specific experimental parameters are as follows:

in the preprocessor, the flow data in the NLS-KDD data set is preprocessed by the preprocessor, and the characteristic attribute in the flow characteristic is expanded into a flow characteristic image of 11 × 11 to be used as the input of the generator. The generator and countermeasure network and the reconstruction encoder model all use Adam optimizers, the initial learning rate is 0.0001, and the momentum parameter is (beta)₁＝0.9,β₂0.999), batch _ size is set to 64. During training, the parameters of the generator are fixed, the parameters of the countermeasure network are updated through the loss function of the optimization formula (1) of the Adam optimizer, and then the parameters of the countermeasure network are fixedAnd parameters of the generator and the reconstruction encoder are updated through an Adam optimizer optimization formula (5) loss function, the parameters of the countermeasure network, the generator and the reconstruction encoder are alternately updated once every iteration, the training parameters are saved once every iteration for 1000 times, and the training is finished when the iteration is 12000 times.

Step four: and (6) testing.

And testing on the abnormal network flow detection model trained in the step three. The testing process is specifically that | Z-Z' | calculation of calculation₁Setting a threshold theta by comparing theta with | | | Z-Z' |₁To determine whether the traffic is abnormal.

In summary, for the abnormal traffic of the network, the invention provides an abnormal traffic detection method based on a generation countermeasure network (GAN), which trains a generator, a reconstruction encoder and a countermeasure network in stages, meets the functional requirement of detecting the abnormal traffic of a known type or even an unknown type under the environment of training normal traffic samples, and has practical application value in the environment of abnormal network traffic detection.

It will be appreciated by persons skilled in the art that the scope of the present invention is not limited to the specific embodiments described. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and it is noted that the technical solutions after the changes or substitutions will fall within the protection scope of the invention.

Claims

1. An abnormal traffic detection method based on a generation countermeasure network, which is characterized by comprising the following steps:

(1) providing an abnormal flow detection method framework based on a generation countermeasure network;

2. The abnormal traffic detection method based on the generative countermeasure network as claimed in claim 1, wherein the abnormal traffic detection method based on the generative countermeasure network framework specifically comprises:

the input flow is classified by a preprocessor, a generator and a countermeasure network and a reconstruction encoder. The method comprises the steps that a countermeasure network and a generator are connected, a reconstruction encoder conducts countermeasure training, training of the generator and the reconstruction encoder is promoted, a flow characteristic image generated by the generator is optimized, meanwhile, the difference between a one-dimensional vector obtained by the flow characteristic image generated by a preprocessor through encoding of an encoder sub-module in the generator and a one-dimensional vector obtained by the flow characteristic image generated by the generator through encoding of the reconstruction encoder is reduced, meanwhile, in the design of the countermeasure network, the spectrum normalization is added into the last layer of network, and the countermeasure network is stabilized in the training process.

3. The abnormal traffic detection method based on the generation countermeasure network according to claim 1, wherein the preprocessing method based on the network traffic characteristics specifically includes:

in the preprocessor, original flow is processed in batches, collected flow data are subjected to statistics of basic characteristics of TCP connection, content characteristics of TCP connection, time-based network flow statistical characteristics and host-based network flow statistical characteristics, continuous values in the characteristics are processed by standard deviation standardization in consideration of large difference among data, discrete values in the characteristics are subjected to one-hot encoding (one-hot), attribute dimensions are expanded, and finally a reshape function is used for converting one-dimensional flow characteristics into two-dimensional flow characteristic images.