CN114996287B - Automatic equipment identification and capacity expansion method based on feature library - Google Patents

Abstract

The invention discloses an automatic equipment identification and capacity expansion method based on a feature library. After the unknown equipment is accessed, the method can compare the similarity with the existing equipment characteristics in the equipment library, and the known equipment is identified by combining equipment identification rules and an identification algorithm model. And for the unknown equipment, under a feature vector space model, the invention performs classification management on the unknown equipment type according to a similarity criterion, locks or reduces the range of the equipment to be identified, automatically generates equipment feature information and stores the equipment feature information into a designated area, and automatically updates the equipment feature information into an equipment library after manual intervention confirmation. Compared with the fully manual extraction of the device characteristics, the workload of heterogeneous edge device access is reduced by at least half of the original workload. Therefore, the invention can solve the problem that the equipment feature library in the current stage can not be automatically expanded, and better improves the management efficiency of the access and configuration of the edge equipment.

Description

A device automatic identification and expansion method based on feature library

Technical Field

The invention relates to a device automatic identification and capacity expansion method based on a feature library, and belongs to the field of industrial control automation.

Background technique

With the rapid development of Internet of Things technology, more and more edge devices are developing towards networking and intelligence, and the workload of edge networking devices is increasing rapidly. The existing edge device identification method is mainly based on the device feature information in the feature library, and can only identify known devices in the device feature library. Devices not in the device library cannot be identified.

In recent years, more and more unknown and private network communication protocol devices have emerged one after another. The diversification and heterogeneous development trend of edge devices has not only increased the difficulty of protocol identification and analysis, but also the workload of protocol feature extraction has increased rapidly, bringing unprecedented challenges to the automation upgrade and transformation of manufacturing enterprises. At the same time, with the increase in the types of access devices, the device feature library needs to be continuously upgraded. However, the current device feature library lacks an automatic update mechanism. The work of extracting features from such a large number of devices is extremely heavy and boring, and requires a lot of manual work.

The identification of heterogeneous edge devices is an important prerequisite for establishing IoT connections. Since the protocols and performance of different types of edge devices vary greatly, a classification strategy should be adopted for the identification of edge devices. The first step in classification management is to accurately identify the device type, quickly and accurately identify the device in cyberspace, and judge its device attributes in a fine-grained manner. This will not only help to continuously expand the device library and support the access of more devices, but also reduce the workload of technicians in extracting device features and improve work efficiency.

The invention patent application with application number 202110974559.4 proposes a real-time detection and identification method and system for distribution network IoT terminal equipment. This patent application focuses more on the field of power distribution network. Based on the existing basic information database of distribution network IoT terminal equipment, it realizes the identification of online terminal equipment through comparative quantitative methods, thereby increasing the credibility and transparency of distribution network operation. However, the basic information database of equipment lacks an automatic update mechanism.

The invention patent application with application number 202010187111.3 proposes a terminal device identification system and method. For unknown devices in the device feature library, the patent application creates a corresponding relationship between the monitoring module and the device, realizes rapid identification and configuration of the terminal device, and improves the efficiency of batch networking and integrated management of terminal devices. It does not involve the device library update mechanism.

The invention patent application with application number 202011643313.0 proposes a feature library update method, device, network device and readable storage medium. This patent application focuses more on the field of network security technology. By loading and compiling a second feature library used to replace the first feature library in a specified data structure of the shared memory of the network device, and setting a synchronization lock, the network security problem during the update of the feature library is improved. However, its feature library update method is not suitable for the needs of automatic identification of edge devices and feature library expansion in the industrial control field.

Summary of the invention

The technical problem to be solved by the present invention is that in the existing edge device identification method, the device feature library lacks an automatic update mechanism, but as the types of access devices increase, the device feature library needs to be continuously upgraded.

In order to solve the above technical problems, the technical solution of the present invention is to provide a method for automatic device identification and expansion based on a feature library, which is characterized by comprising the following steps:

Step 1: Abstract the characteristic message of the IoT device into a word frequency vector composed of characteristic words. After feature engineering processing, the IoT device information is converted into a multi-dimensional feature vector. Based on the accumulation of connected device services, a device feature library is established in the cloud.

Step 2: When a new IoT device comes online, the HTTP response packet of the IoT device is obtained as the original sample through the sample collection module;

Step 3: The feature extraction module extracts the sample features of the original sample:

The feature extraction module extracts information that can reflect IoT devices from the HTTP response package, and then uses feature engineering to obtain the corresponding vectorized word vector information as sample features;

Step 4: The data preprocessing module preprocesses the sample features extracted by the feature extraction module, converts the sample features of the text type into the sample features of the numerical type, and thus converts the IoT device information into a multi-dimensional feature vector;

Step 5. The algorithm recognition module uses the multidimensional feature vector of the current online IoT device as input, and performs feature matching on the multidimensional feature vector with the multidimensional feature vector of the IoT device of known type in the device feature library. If the multidimensional feature vector of the current online IoT device is consistent with the multidimensional feature vector of any IoT device of known type in the device feature library, the current online IoT device belongs to a known device, and the current online IoT device is recognized. Otherwise, the current online IoT device belongs to an unknown device. The algorithm recognition module uses the improved constrained seed K-means recognition algorithm to calculate vector similarity and recognize and classify the current online IoT device. The improved constrained seed K-means recognition algorithm uses the cosine similarity of two multidimensional feature vectors to measure the similarity, and based on the cosine similarity Clustering is performed using the K-means recognition algorithm; during clustering, when the cosine similarity value between the multidimensional feature vector corresponding to the unknown device and the cluster center of a cluster of a known device type is greater than a given threshold ε, the unknown device is classified into the cluster, the device type of the current unknown device is the device type corresponding to the cluster, and the corresponding device and communication features are generated based on the multidimensional feature vector of the current unknown device and stored in a designated area; when the cosine similarity value between the multidimensional feature vector corresponding to the unknown device and the cluster centers of all clusters is not greater than a given threshold ε, it indicates that the current unknown device belongs to a new device category, and a new device type is automatically created based on the multidimensional feature vector of the current unknown device, and the current unknown device is classified into the new device type, and then the corresponding device and communication features are generated based on the multidimensional feature vector of the current unknown device and stored in a designated area;

Step 6: Manually read the device and communication characteristics stored in the designated area, obtain the new device type, manually check the manufacturer, device category, model, communication characteristics and other information of the unknown device, and manually confirm whether the classification of the unknown device and the new device type are correct; after manual intervention and confirmation, the unknown device is identified, and then the new device type and the device and communication characteristics of the identified unknown device are automatically updated to the device feature library, thereby realizing semi-automatic expansion of the device feature library.

Preferably, step 2 comprises the following steps:

Step 201: The sample collection module performs port scanning in the entire IP address space to obtain the IP addresses of untagged unknown IoT devices, and adds the IP addresses of tagged IoT devices of known types in the device feature library to form a device IP address set;

Step 202: The sample collection module sends a request to all IP addresses in the device IP address set to obtain a complete HTTP response packet header as an original sample of the corresponding online IoT device.

Preferably, step 3 comprises the following steps:

Step 301, the feature extraction module counts the total number of header fields in the HTTP response packet and removes redundant information;

Step 302: The feature extraction module selects the field with the highest frequency of occurrence from all header fields as device feature information, and then obtains the corresponding vectorized word vector information through feature engineering processing. The vectorized word vector information is the sample feature of the original sample.

Preferably, in step 5, the cosine similarity cosθ between vector X and vector Y is calculated using the following formula:

Where x _i and y _i are the i-th elements in vector X and vector Y respectively.

After the unknown device is connected, the present invention can compare the similarity with the existing device features in the device library, and use a combination of device identification rules and identification algorithm models to identify known devices. For unknown devices, under the feature vector space model, the present invention classifies and manages unknown device types according to similarity criteria, locks or narrows the scope of devices to be identified, automatically generates device feature information and stores it in a designated area, and automatically updates it to the device library after manual intervention and confirmation. Compared with fully manual extraction of device features, the workload of heterogeneous edge device access is reduced by at least half of the original. Therefore, the present invention can overcome the problem that the current device feature library cannot be automatically expanded, and better improve the management efficiency of edge device access and configuration.

Specifically, compared with the prior art, the present invention has the following beneficial effects:

(1) After the unknown device goes online, the cloud driver scans and obtains the original sample information of the device, vectorizes the feature values, and uses the multi-dimensional feature vector of the device as input to calculate the vector similarity using the improved constrained seed K-means recognition algorithm. According to the similarity of the device features, the unknown devices are grouped to narrow the device range.

(2) Expansion of the equipment library. The original manual extraction of unknown equipment features has been changed to a machine recognition algorithm that first defines the unknown equipment by grouping, and then automatically generates unknown equipment features and stores them in a designated area to narrow the scope. Manual intervention is then required to achieve semi-automatic expansion of the equipment library, greatly reducing the workload of on-site personnel.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a device library expansion method provided by the present invention;

FIG2 is a flow chart of the recognition algorithm used in the present invention.

Detailed ways

The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

The traditional device identification method based on the existing feature library rule method has the problems of limited new device identification types and poor scalability. Taking into account the characteristics of IoT device information, under the vector space model, the present invention adopts an improved constrained seed K-means recognition algorithm. On the basis of the original K-means recognition algorithm, the cosine distance function is introduced as the similarity measurement function. The cosine distance pays more attention to the difference in direction and is insensitive to the absolute value. This optimization algorithm makes up for the problem of inconsistent metric standards between different device samples. The cosine distance is selected as the distance measurement standard to improve the device clustering effect and the ability to discover new device categories.

Based on the above principles, the method for automatic device identification and expansion based on a feature library provided by the present invention mainly includes the following key points: first, the configuration of the cloud device library; second, the identification of edge devices; third, the classification management of unknown devices by improving the K-means recognition algorithm, which specifically includes the following steps:

Step 1: Abstract the characteristic message of the IoT device into a word frequency vector composed of characteristic words. After feature engineering processing, the IoT device information is converted into a multi-dimensional feature vector. Based on the accumulation of connected device services, a device feature library is established in the cloud.

Step 2: After the IoT device is online, the original sample of the IoT device is obtained through the sample collection module, including the following steps:

Step 201: The sample collection module performs port scanning in the entire IP address space to obtain the IP addresses of untagged unknown IoT devices, and adds the IP addresses of tagged IoT devices of known types in the device feature library to form a device IP address set;

Step 202: The sample collection module sends a request to all IP addresses in the device IP address set to obtain a complete HTTP response packet header as an original sample of the corresponding online IoT device.

Step 3: The feature extraction module extracts the sample features of the original sample:

The feature extraction module extracts information that can reflect the IoT device from the HTTP response packet as sample features, including the following steps:

Step 301: The feature extraction module counts the total number of header fields in the HTTP response packet and removes redundant information to reduce computational complexity and improve device recognition efficiency.

Step 302: The feature extraction module selects the field with the highest frequency of occurrence from all header fields as device feature information, and then obtains the corresponding vectorized word vector information through feature engineering processing. The vectorized word vector information is the sample feature of the original sample.

Step 4: The data preprocessing module preprocesses the sample features extracted by the feature extraction module, converts the text type sample features into numerical type sample features, and thus converts the IoT device information into a multi-dimensional feature vector.

The present invention converts text content into vectors in a multidimensional vector space through a data preprocessing module, and the similarity of two vectors in the multidimensional vector space can be used to represent the similarity of the corresponding text content. Therefore, the algorithm recognition module provided by the present invention adopts an improved K-means recognition algorithm to realize device category recognition.

Step 5. The algorithm recognition module takes the multidimensional feature vector of the current online IoT device as input, and performs feature matching on the multidimensional feature vector with the multidimensional feature vector of the tagged IoT device of known types in the device feature library. If the multidimensional feature vector of the current online IoT device is consistent with the multidimensional feature vector of any tagged IoT device of known types in the device feature library, the current online IoT device is a known device, and the current online IoT device is recognized. Otherwise, the current online IoT device is an unknown device, and the algorithm recognition module uses the improved constrained seed K-means recognition algorithm to perform vector similarity calculation to identify and classify the current online IoT device.

Traditional K-means recognition algorithms often use the Euclidean distance between vectors as a measurement indicator. Assuming that the multidimensional feature vectors corresponding to IoT device 1 and IoT device 2 are X and Y, respectively, and vector X and vector Y are n-dimensional vectors, then vector X and vector Y are represented as X(x ₁ ,x ₂ ,…,x _n ) and Y(y ₁ ,y ₂ ,…,y _n ) respectively, and the Euclidean absolute distance d(X,Y) ^* between vector X and vector Y is calculated using the following formula:

x _i , y _i are the i-th elements in vector X and vector Y respectively.

However, the Euclidean absolute distance reflects more the absolute differences in individual numerical features and is not suitable for datasets of IoT device information.

The improved constrained seed K-means recognition algorithm of the present invention uses the cosine similarity of the comparison multidimensional feature vectors to measure the similarity between two vectors by measuring the cosine value of the angle between them projected into a multidimensional space. The cosine similarity cosθ of vector X and vector Y is calculated using the following formula:

When the angle between vector X and vector Y is 0°, the value of cosine similarity cosθ is 1; when the angle between vector X and vector Y is 90°, the value of cosine similarity cosθ is 0; when vector X and vector Y point in completely opposite directions, the value of cosine similarity cosθ is -1. Without normalization, the range of cosine similarity values is between [-1,1]. The closer the value is to 1, the closer the directions of the two vectors are; the closer the value is to -1, the opposite their directions; the closer the value is to 0, the more orthogonal the two vectors are.

Euclidean distance and cosine similarity have different calculation methods and measurement characteristics. Two similar texts may be far apart in Euclidean distance due to the difference in the amount of data they contain, but they have a small angle between them and thus have a high cosine similarity. Euclidean distance reflects more of the absolute difference in individual numerical features, while cosine similarity focuses more on the difference in the direction of the vector and is insensitive to the absolute value. This feature makes up for the problem of inconsistent metric standards that may exist between samples from different devices.

Therefore, the improved constrained seed K-means recognition algorithm of the present invention selects cosine similarity to perform similarity analysis on the multi-dimensional feature vectors corresponding to two IoT devices to identify the devices.

For example, the multidimensional feature vectors corresponding to IoT device 1 and IoT device 2 are vector A and vector B, A = (7, 0, 5, 3, 10, 0, 1, 0, 0) and B = (3, 0, 2, 1, 4, 0, 0, 0, 1), then:

A·B＝7×3+0×0+5×2+3×1+……0×1＝74

It can be seen that the cosine similarity is used to perform similarity analysis on the multi-dimensional feature vectors corresponding to IoT device 1 and IoT device 2, and it is found that the two are highly similar, which is consistent with the actual situation. Therefore, cosine similarity is selected as a measure of the similarity between unknown devices and known types of marked IoT devices in the device feature library, and unknown devices are classified based on cosine similarity.

(a) When the cosine similarity value between the multidimensional feature vector corresponding to the unknown device and the multidimensional feature vector of the cluster center corresponding to any type of labeled IoT device in the device feature library is greater than a given threshold ε (in this embodiment, ε = 0.9), the unknown device is classified into the cluster, the type of the current unknown device is the type corresponding to the cluster, and the corresponding device and communication features are generated based on the multidimensional feature vector of the current unknown device and stored in the designated area.

(b) When the cosine similarity value between the multidimensional feature vector corresponding to the unknown device and the multidimensional feature vectors of the cluster centers of all clusters corresponding to all types of labeled IoT devices in the device feature library is not greater than a given threshold ε, it indicates that the current unknown device belongs to a new device category. After automatically creating a new device type based on the multidimensional feature vector of the current unknown device, the current unknown device is classified into the new device type, and then the corresponding device and communication features are generated based on the multidimensional feature vector of the current unknown device and stored in the designated area;

Step 6: Manually read the device and communication features stored in the designated area, obtain the new device type, manually check the manufacturer, device category, model, communication features and other information of the unknown device, and manually confirm whether the classification of the unknown device and the new device type are correct. After manual intervention and confirmation, the unknown device is identified, and the new device type, the device and communication features of the identified unknown device are automatically updated to the device feature library, thereby realizing the semi-automatic expansion of the device feature library, greatly reducing the workload of on-site personnel.

Claims (4)

1. A method for automatic device identification and expansion based on a feature library, characterized in that it comprises the following steps: Step 1: Abstract the characteristic message of the IoT device into a word frequency vector composed of characteristic words. After feature engineering processing, the IoT device information is converted into a multi-dimensional feature vector. Based on the accumulation of connected device services, a device feature library is established in the cloud. Step 2: When a new IoT device comes online, the HTTP response packet of the IoT device is obtained as the original sample through the sample collection module; Step 3: The feature extraction module extracts the sample features of the original sample: The feature extraction module extracts information that can reflect IoT devices from the HTTP response package, and then uses feature engineering to obtain the corresponding vectorized word vector information as sample features; Step 4: The data preprocessing module preprocesses the sample features extracted by the feature extraction module, converts the sample features of the text type into the sample features of the numerical type, and thus converts the IoT device information into a multi-dimensional feature vector; Step 5. The algorithm recognition module uses the multidimensional feature vector of the current online IoT device as input, and performs feature matching on the multidimensional feature vector with the multidimensional feature vector of the IoT device of known type in the device feature library. If the multidimensional feature vector of the current online IoT device is consistent with the multidimensional feature vector of any IoT device of known type in the device feature library, the current online IoT device belongs to a known device, and the current online IoT device is recognized. Otherwise, the current online IoT device belongs to an unknown device. The algorithm recognition module uses the improved constrained seed K-means recognition algorithm to calculate vector similarity and recognize and classify the current online IoT device. The improved constrained seed K-means recognition algorithm uses the cosine similarity of two multidimensional feature vectors to measure the similarity, and based on the cosine similarity Clustering is performed using the K-means recognition algorithm; during clustering, when the cosine similarity value between the multidimensional feature vector corresponding to the unknown device and the cluster center of a cluster of a known device type is greater than a given threshold ε, the unknown device is classified into the cluster, the device type of the current unknown device is the device type corresponding to the cluster, and the corresponding device and communication features are generated based on the multidimensional feature vector of the current unknown device and stored in a designated area; when the cosine similarity value between the multidimensional feature vector corresponding to the unknown device and the cluster centers of all clusters is not greater than a given threshold ε, it indicates that the current unknown device belongs to a new device category, and a new device type is automatically created based on the multidimensional feature vector of the current unknown device, and the current unknown device is classified into the new device type, and then the corresponding device and communication features are generated based on the multidimensional feature vector of the current unknown device and stored in a designated area; Step 6: Manually read the device and communication characteristics stored in the designated area, obtain the new device type, manually check the manufacturer, device category, model, and communication characteristic information of the unknown device, and manually confirm whether the classification of the unknown device and the new device type are correct; after manual intervention and confirmation, the unknown device is identified, and then the new device type and the device and communication characteristics of the identified unknown device are automatically updated to the device feature library, thereby realizing semi-automatic expansion of the device feature library. 2. The method for automatic device identification and capacity expansion based on a feature library according to claim 1, wherein step 2 comprises the following steps: Step 201: The sample collection module performs port scanning in the entire IP address space to obtain the IP addresses of untagged unknown IoT devices, and adds the IP addresses of tagged IoT devices of known types in the device feature library to form a device IP address set; Step 202: The sample collection module sends a request to all IP addresses in the device IP address set to obtain a complete HTTP response packet header as an original sample of the corresponding online IoT device. 3. The method for automatic device identification and capacity expansion based on a feature library according to claim 1, wherein step 3 comprises the following steps: Step 301, the feature extraction module counts the total number of header fields in the HTTP response packet and removes redundant information; Step 302: The feature extraction module selects the field with the highest frequency of occurrence from all header fields as device feature information, and then obtains the corresponding vectorized word vector information through feature engineering processing. The vectorized word vector information is the sample feature of the original sample. 4. The method for automatic device identification and expansion based on a feature library according to claim 1, wherein in step 5, the cosine similarity cosθ between vector X and vector Y is calculated using the following formula: Where x _i and y _i are the i-th elements in vector X and vector Y respectively.