CN104468262B - A kind of network protocol identification method and system based on semantic sensitivity - Google Patents

Abstract

The invention relates to a network protocol identification method and system based on semantic sensitivity. In the modeling stage, the network data packet collection of a specific application protocol is used as input, and the keyword model of the analyzed protocol is constructed using the Latent Dirichlet Allocation method; in the training stage, the classification feature information of the data packet is extracted according to the protocol keyword model, The obtained keyword feature vector is used as input, and the supervised machine learning method is used to learn and train the offline training data set to obtain the classification model of the analyzed protocol; in the classification stage, the classification feature information of the data message is extracted according to the protocol keyword model, The protocol classification model is used to judge the protocol attribute of the network data message to be tested, and judge whether it belongs to the network data message of the target protocol. The invention can fully excavate the potential protocol semantic information in the network message message, and effectively identify various network protocols.

Description

A semantic-sensitive network protocol identification method and system

technical field

The invention belongs to the field of network technology, and in particular relates to a semantic-sensitive network protocol identification method and system.

Background technique

Protocol identification technology is the process of corresponding network data flow with specific application protocols. It has very important applications in many network and security fields, such as network measurement, intrusion detection and prevention, and botnet behavior detection. Taking its application in intrusion detection and prevention systems as an example, traditional intrusion detection and prevention systems regard each packet payload as a series of byte sequences, and compare the sequence information of these bytes with malicious programs (malware ) signature (usually represented by a set of regular expressions) for matching operations. This kind of coarse-grained signature checking mechanism is greatly limited in reliability because it ignores the application protocol structure of the data packet payload.

Modern intrusion detection and prevention systems are evolving to be more semantically sensitive. Specifically, it obtains the field information of the analyzed application protocol according to the information format of the application protocol, so as to realize a reasonable analysis of the network data packet. Several types of application protocol parsing tools, such as FlowSifter, UltraPAC, binpac, and GAPA, have been proposed in previous research works. All these application protocol analysis tools need the protocol specification information of the protocol to be analyzed, so as to generate the analysis tool corresponding to the protocol. However, many application protocols in the Internet are proprietary protocols, and there is no publicly available protocol fingerprint specification information for these protocols. According to the Internet2NetFlow organization's statistics on the traffic in the backbone network, it is found that more than 40% of the network data flows belong to unidentified application protocols. Network communication protocols used by malware and botnets have no protocol specification information from their designers. In order to analyze the network data flow of an unknown application protocol, it is first necessary to infer the protocol to obtain the protocol fingerprint information. Network monitoring tools, such as Wireshark, NetDude, SNORT and BRO, etc. also need application protocol analysis tools to realize their functions.

Network protocol identification methods can be divided into three categories based on the transport layer port, based on the data packet load and based on the statistical behavior characteristics of network data flow according to the different research objects. The present invention takes the load of network data message as the basic research object. Existing methods in this field can be roughly divided into two categories: (1) methods based on protocol analysis; (2) methods based on protocol signatures. The present invention belongs to the second category, methods based on protocol signatures. The method based on the protocol signature only relies on the analysis of the data message load in the analysis process, and does not depend on the executable code of the application program. Previously, the research work on the automatic construction of protocol signatures did not use the latent semantic information existing in the data message, that is, the association relationship between the syntactic elements in the data message. It is worth noting that such research work cannot achieve the research goal of using fewer classification features and achieving higher accuracy. At the same time, compared with previous research work, the present invention makes fewer assumptions on the analyzed network protocol itself.

Contents of the invention

The purpose of the present invention is to design and implement a network protocol recognition method and system based on semantic sensitivity, so that in the network protocol recognition process, the potential protocol semantic information in the network message message can be fully tapped; while ensuring a higher recognition accuracy Under the premise of the recall rate, it has strong universality and robustness in practice.

The invention motivation of the present invention comes from the continuously increasing variety of unknown network traffic, and the novel protocol identification method and system designed are based on the minimum manpower requirement to realize the comprehensive automation of the specific application protocol identification process.

Specifically, the present invention adopts the following technical solutions:

A semantic-sensitive network protocol recognition method, including a modeling stage, a training stage and a classification stage;

In the modeling stage, the network data packet collection of a specific application protocol is used as input, and the keyword model of the analyzed protocol is constructed using the LatentDirichlet Allocation method;

In the training stage, according to the protocol keyword model obtained in the modeling stage, the classification feature information of the data message is extracted, and the obtained keyword feature vector is used as input, and the offline training data set is learned and trained by a supervised machine learning method, so as to obtain all Classification models for analysis protocols;

In the classification stage, the classification feature information of the data packet is extracted according to the protocol keyword model obtained in the modeling stage, and the protocol classification model output in the training stage is used to judge the protocol attributes of the network data packet to be tested, and determine whether it belongs to the target Protocol network data packets.

Further, the specific steps of the modeling phase include:

1) Collect network data packets belonging to a specific application protocol, thereby dividing network data packets into two categories: one is a collection of data packets belonging to the application protocol to be analyzed; the other is a collection of data packets that do not belong to the application protocol to be analyzed datagram collection;

2) Utilize the n-gram model to convert the network data message into a network data message with n-gram elements as the basic unit; the n-gram model is a subsequence of n consecutive elements of a given sequence;

3) Construct the protocol keyword model of the protocol to be analyzed by using the Latent Dirichlet Allocation method.

Further, the specific steps of using the Latent Dirichlet Allocation method to construct the protocol keyword model include:

1) For all n-grams in the set D containing M data packets Assign a random keyword index number Here w _{(m, i)} represents the i-th n-gram in the data message m, z _{(m, i)} is the keyword index number of the n-gram, and N _m is the n-gram element in the data message m the number of

2) with Represents the keyword index number of all other n-grams except z _(m,i) , in When the value remains constant, according to the posterior probability distribution Generate a new keyword index value z _(m,i) for n-gram w _(m,i) by sampling; where α and β are given hyperparameters, Represents the number of times element t in the n-gram dictionary is assigned to key k, Represents the number of occurrences of keyword k in the message message m, and W represents the number of n-gram elements in the n-gram dictionary;

3) According to the value of z _{(m, i)} obtained by the Gibbs sampling method, the expired value in the posterior probability distribution is updated;

4) Repeat the above sampling operation for all tuples (m, i) in the data set. If the Gibbs sampling convergence condition L is reached, the algorithm stops and returns the final keyword index number Otherwise repeat steps 1) to 3);

5) Construct a protocol keyword model using the keyword index number obtained through steps 1) to 4)

Where K represents the number of protocol keywords,

A semantically sensitive network protocol recognition system based on the above method, comprising:

The modeling unit uses the network data packet collection of the specific application protocol as input, and uses the LatentDirichlet Allocation model to construct the keyword model of the analyzed protocol;

The training unit extracts the classification feature information of the data message according to the protocol keyword model obtained by the modeling unit, and takes the obtained keyword feature vector as input, and uses the supervised machine learning method to learn and train the offline training data set, so as to obtain the analyzed The classification model of the agreement;

The classification unit extracts the classification feature information of the data message according to the protocol keyword model obtained by the modeling unit, and uses the protocol classification model output by the training unit to make a judgment on the protocol attribute of the network data message to be tested, and judge whether it belongs to the target protocol network data packets.

Key technical points of the present invention are:

1) Make full use of the latent semantic information in the protocol message. The present invention can distinguish different meanings represented by the same n-grams element in different messages. These different messages may have different semantics and thus should be classified as different protocol keywords. It is worth noting that the previous protocol information format inference method based on network data flow cannot deal with the above-mentioned problems well. Because most of the previous methods rely on counting the frequency of occurrences of strings, thus ignoring the context in which each string appears.

2) In addition, the present invention can discover the correlation between different n-grams. In a protocol message, multiple n-grams together form an element in the protocol message format. For example, in an SMTP message, the 3-grams, "250" and "OK" together represent a protocol element that can be used to confirm a mail session. Utilizing protocol keyword recognition, the present invention can aggregate interrelated n-grams to form a protocol keyword.

Utilize the method of the present invention to carry out effective protocol identification to multiple network protocols, compared with the disclosed related technology, it has the following advantages:

1. The method can solve the application protocol identification problem of connection-oriented protocols (such as TCP) and connectionless protocols (such as UDP);

2. The method is based on the payload statistics of data packets, which does not assume any prior knowledge of the protocol specification. Therefore, it can be applied to the identification of text, binary and encrypted protocols;

3. As a packet-based network protocol identification solution, the method does not need to assemble IP data packets into application layer messages. Therefore, it applies to both packet-by-packet and per-flow protocol classification schemes.

4. This method is applicable to both large and long flows (such as SMTP) and small and short flows (such as FTP) in real network environments.

Description of drawings

Figure 1 is a flowchart of the modeling stage of the network protocol recognition method based on semantic sensitivity.

Fig. 2 is a flowchart of the training phase of the network protocol recognition method based on semantic sensitivity.

Fig. 3 is a flowchart of the classification stage of the network protocol recognition method based on semantic sensitivity.

Fig. 4 is a flow chart of protocol keyword model construction based on Latent Dirichlet Allocation method.

Fig. 5 is an architecture diagram of a semantic-sensitive network protocol recognition system.

detailed description

In order to make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described below through specific embodiments and accompanying drawings.

The network protocol identification method based on semantic sensitivity of the present invention takes the network data flow as input, and automatically and accurately recognizes the network data flow of the analyzed protocol from the mixed network flow. The method only analyzes the load part of the IP data message, and does not need to reverse-analyze the executable code of the program, and does not rely on prior knowledge in the protocol specification. At the same time, the method can solve the identification problem of connection-oriented protocols (such as TCP) and connection-oriented protocols (such as UDP). The method consists of three main phases: a modeling phase, a training phase, and a classification phase.

The modeling stage consists of three modules: data collection, data message n-gram generation, and keyword model construction (keyword recognition). Its flow chart is shown in Figure 1, and the specific description is as follows:

1. Data collection: The function of the data collection module is to collect network data packets belonging to specific application protocols. Therefore, the network data packets are divided into two categories: one is a collection of data packets belonging to the application protocol to be analyzed; the other is a collection of data packets not belonging to the application protocol to be analyzed.

2. Data packet n-gram generation: the data packet n-gram generation operation uses the n-gram model to convert the network data packet into a network data packet with n-gram elements as the basic unit. The n-gram model described in the present invention is a subsequence of n consecutive elements of a given sequence (at least a sequence of n elements). Given a set of network data packets, the n-gram model decomposes the network packet sequence b ₁ , b ₂ ,…,b _m with byte size m into a sequence of n-grams (n≤m): b ₁ , b ₂ ,...,b _n , b ₂ ,b ₃ ,...,b _n+1 ,...,b _m-n+1 ,b _m-n+2 ,...,b _m . In practice, usually only the first W n-gram elements with higher statistical frequency are selected and their n-gram dictionary is formed.

3. Keyword recognition: The keyword recognition module uses the Latent Dirichlet Allocation (LDA) method to construct the protocol keyword model of the protocol to be analyzed. The output of the modeling phase is a protocol keyword model of the analyzed network protocol.

The training phase consists of four modules: data collection, data packet n-gram generation, feature extraction, and classifier learning. Its flow chart is shown in Figure 2, and the specific description is as follows:

1. Data collection: Same as step 1 in the modeling phase.

2. Generation of datagram n-grams: same as step 2 in the modeling phase.

3. Feature extraction: perform classification feature extraction on network data packets according to the protocol keyword model obtained in step 3 of the modeling stage. This step calculates the occurrence probability of different keywords in the data message, and accordingly forms the K-dimensional feature vector of the data message.

4. Classifier learning: use a supervised learning method to construct a binary classifier for the analyzed application protocol according to the classification features obtained by the packet feature extraction module in step 3.

The classification stage consists of three modules: data packet n-gram generation, feature extraction, and classifier. Its flow chart is shown in Figure 3, and the specific description is as follows:

1. Data packet n-gram generation: same as step 2 in the training phase.

2. Feature extraction: Same as step 3 in the training phase.

3. Classifier: according to the classification feature vectors obtained in steps 1 and 2 and the classification model obtained in step 4 of the training phase, the protocol category is determined for unidentified network data packets. There are two types of output results: one is the network data message belonging to the target protocol, and the other is the network data message of the non-target protocol.

The innovation of the whole method lies in the construction process of the protocol keyword model, which can be divided into the following steps. Figure 4 shows the flow chart of protocol keyword identification and keyword model construction based on the Latent Dirichlet Allocation (LDA) method .

The input of the protocol keyword model construction process is a collection of message packets belonging to a specific application protocol. L is the termination condition of the protocol keyword model construction process. The output of the protocol keyword model building process is the protocol keyword model of the analyzed network protocol. This method builds the protocol keyword model based on the Latent Dirichlet Allocation (LDA) method, and its specific implementation steps are as follows:

1. First, for all n-grams in the set D containing M datagrams Assign a random keyword index number Here w _{(m, i)} represents the i-th n-gram in the data message m, z _{(m, i)} is the keyword index number of the n-gram, and N _m is the n-gram element in the data message m the number of

2. Next, use Keyword index numbers representing all other n-grams except z _(m,i) . exist When the value remains constant, according to the posterior probability distribution Generate a new keyword index value z _(m,i) for n-gram w _(m,i) by sampling. where α and β are given hyperparameters, Represents the number of times element t in the n-gram dictionary is assigned to key k, Represents the number of occurrences of keyword k in message message m. W represents the number of n-gram elements in the n-gram dictionary.

3. According to the value of z _{(m, i)} obtained by the Gibbs sampling method, the expired value in the posterior probability distribution is updated;

4. Repeat the above sampling operation for all tuples (m, i) in the data set. If the Gibbs sampling convergence condition L is reached, the algorithm is terminated and the final keyword index number is returned Otherwise repeat steps 1-3.

5. Construct a protocol keyword model using the keyword index number obtained through steps 1-4

Where K represents the number of protocol keywords,

Combining with the above-mentioned network protocol identification method based on semantic sensitivity, the present invention also discloses a network protocol identification system based on semantic sensitivity. The system is mainly composed of three parts: modeling unit, training unit and classification unit, which correspond to the modeling phase, training phase and classification phase respectively. The system diagram architecture is shown in Figure 5.

1. Modeling unit: take the network data packet collection of a specific application protocol as input, and use the LatentDirichlet Allocation model to construct the keyword model of the analyzed protocol. The output of this unit is a protocol keyword model of the analyzed protocol.

2. Training unit: According to the protocol keyword model obtained by the modeling unit, the classification characteristic information of the data message is extracted. Taking the keyword feature vector obtained by the feature extraction module as input, the supervised machine learning method is used to learn and train the offline training data set, so as to obtain the classification model of the analyzed protocol.

3. Classification unit: extract the classification feature information of the data message according to the protocol keyword model obtained by the modeling unit, and use the protocol detection model (i.e. the above classification model) output by the training unit to make the protocol attribute of the network data message to be tested. judge. There are two types of output results: one is the network data message belonging to the target protocol, and the other is the network data message of the non-target protocol.

In the verification experiment, the present invention conducts experiments respectively under the situation that the total number W of n-gram elements is different values to the DNS protocol and the FTP protocol, and compares its accuracy rate under different supervised learning algorithms, the recall rate and F measure. Given a certain application protocol to be analyzed by the system, the present invention first defines the following three data sets:

●True Positives (TP): It is recognized by the system as a network data packet of a certain protocol, and it is indeed a set of network data packets belonging to the protocol.

●False Positives (FP): It is recognized by the system as a network packet of a certain protocol, but it does not belong to the set of network packets of the protocol.

●False Negatives (FN): It is recognized by the system as a network data packet not of a certain protocol, but it is actually a collection of network data packets belonging to the protocol.

●True Negatives (TN): The network data packets identified by the system as not belonging to a certain protocol, and indeed do not belong to the set of network data packets of the protocol.

Based on the above three data sets, the present invention uses three evaluation indicators commonly used in the field of machine learning (precision), recall (recall) and F-Measure (F-Measure) to evaluate the effectiveness and reliability of the system Evaluation. The three evaluation indicators are defined as follows:

Since the accuracy rate and the recall rate describe two aspects of system performance respectively, the single use of the accuracy rate and the recall rate as an evaluation index has limitations. Therefore, this paper chooses the F measure index to comprehensively consider these two indicators, so as to select the optimal solution. . The experimental results of the semantic-sensitive network protocol recognition method in the DNS protocol and the FTP protocol are shown in the table below.

Table 1: Experimental results of DNS protocol

Table 2: Experimental results of FTP protocol

Table 1 shows the experimental results of the DNS protocol. The present invention pays attention to the numerical value of the accuracy rate of the DNS protocol, and the variation range is 94.16% to 99.74% under different parameter settings. The value of the recall rate varies from 98.21% to 99.85% under different parameter settings. For the DNS protocol, the present invention finds that the best experimental result is the C4.5 decision tree, and the corresponding W value is 1000.

Table 2 shows the experimental results of the FTP protocol. The present invention pays attention to the value of the accuracy rate of the FTP protocol. Under different parameter settings, the variation range is 97.20% to 99.56%. The value of the recall rate varies from 87.16% to 97.28% under different parameter settings. For the FTP protocol, the present invention finds that the best experimental result is to use the C4.5 decision tree, and the corresponding W value is 1500.

The above embodiments are only used to illustrate the technical solution of the present invention and not to limit it. Those of ordinary skill in the art can modify or equivalently replace the technical solution of the present invention without departing from the spirit and scope of the present invention. The scope of protection should be determined by the claims.

Claims (6)

1. A semantically sensitive network protocol recognition method, characterized in that, comprises a modeling stage, a training stage and a classification stage; In the modeling stage, the network data packet collection of a specific application protocol is used as input, and the keyword model of the analyzed protocol is constructed using the Latent DirichletAllocation method; In the training stage, according to the protocol keyword model obtained in the modeling stage, the classification feature information of the data message is extracted, and the obtained keyword feature vector is used as input, and the offline training data set is learned and trained by a supervised machine learning method, so as to obtain all Classification models for analysis protocols; In the classification stage, the classification feature information of the data packet is extracted according to the protocol keyword model obtained in the modeling stage, and the protocol classification model output in the training stage is used to judge the protocol attributes of the network data packet to be tested, and determine whether it belongs to the target Protocol network data packets; The modeling phase includes the following steps: a) Collect network data packets belonging to a specific application protocol, thereby dividing the network data packets into two categories: one is a collection of data packets belonging to the application protocol to be analyzed; the other is a collection of data packets that do not belong to the application protocol to be analyzed datagram collection; b) Utilizing the n-gram model to convert the network data message into a network data message with an n-gram element as a basic unit; the n-gram model is a subsequence of n consecutive elements of a given sequence; c) Construct the protocol keyword model of the protocol to be analyzed by using the Latent Dirichlet Allocation method; The steps to construct a protocol keyword model using the Latent Dirichlet Allocation method include: 1) For all n-grams in the set D containing M data packets Assign a random keyword index number Here w _{(m, i)} represents the i-th n-gram in the data message m, z _{(m, i)} is the keyword index number of the n-gram, and N _m is the n-gram element in the data message m the number of 2) with Represents the keyword index number of all other n-grams except z _(m,i) , in When the value remains constant, according to the posterior probability distribution Generate a new keyword index value z _(m,i) for n-gram w _(m,i) by sampling; where α and β are given hyperparameters, Represents the number of times element t in the n-gram dictionary is assigned to key k, Represents the number of occurrences of keyword k in the message message m, and W represents the number of n-gram elements in the n-gram dictionary; 3) According to the value of z _{(m, i)} obtained by the Gibbs sampling method, the expired value in the posterior probability distribution is updated; 4) Repeat the above sampling operation for all tuples (m, i) in the data set. If the Gibbs sampling convergence condition L is reached, the algorithm stops and returns the final keyword index number Otherwise repeat steps 1) to 3); 5) Construct a protocol keyword model using the keyword index number obtained through steps 1) to 4) Where K represents the number of protocol keywords, 2. The method according to claim 1, characterized in that: when generating the data message n-gram, only select the first W n-gram elements with higher statistical frequency, and form its n-gram dictionary. 3. The method according to claim 1, wherein the specific steps of the training phase include: 1) Data collection, the same as the operation of step 1) in the modeling stage; 2) Generation of data message n-gram, the same as the operation of step 2) in the modeling stage; 3) Carry out classification feature extraction to the network data message according to the protocol keyword model obtained in step 3) of the modeling stage; 4) Using a supervised learning method to construct a binary classifier for the analyzed application protocol according to the extracted classification features. 4. The method according to claim 3, wherein the specific steps of the modeling phase include: 1) Data packet n-gram is generated, same as step 2 in the training phase; 2) feature extraction, same as training stage step 3); 3) According to the classification feature vector obtained in step 1) and step 2) and the classification model obtained in step 4) of the training phase, the protocol type is determined for the unmarked network data message. 5. The method according to claim 1, wherein the network protocol to be tested is a connection-oriented protocol and/or a connectionless protocol. 6. A semantically sensitive network protocol recognition system based on the method according to claim 1, characterized in that, comprising: The modeling unit uses the network data packet collection of the specific application protocol as input, and uses the Latent DirichletAllocation model to construct the keyword model of the analyzed protocol; The training unit extracts the classification feature information of the data message according to the protocol keyword model obtained by the modeling unit, and takes the obtained keyword feature vector as input, and uses the supervised machine learning method to learn and train the offline training data set, so as to obtain the analyzed The classification model of the agreement; The classification unit extracts the classification feature information of the data message according to the protocol keyword model obtained by the modeling unit, uses the protocol classification model output by the training unit to make a judgment on the protocol attribute of the network data message to be tested, and judges whether it belongs to the target protocol network data packets.