CN116821053B - Data reporting methods, devices, computer equipment and storage media - Google Patents

Abstract

The application relates to a data reporting method, a device, computer equipment and a storage medium, wherein the data to be reported of a file is obtained, the characteristic value of the data to be reported is extracted, the data to be reported is classified into different barrel files according to the characteristic value and stored, the data to be reported is clustered in the barrel files according to the similarity to obtain a plurality of groups of data clusters to be reported, the data clusters to be reported are scored according to the ratio of normal data samples to malicious data samples under each group of data clusters to be reported, a plurality of groups of data clusters to be reported are selected according to the score to report, repeated or similar useless data reporting is reduced through clustering, the reported data is filtered through scoring, the problem that the reporting efficiency of the file data in the related technology is lower is solved, the space required for storing the data is reduced, and the reporting efficiency of the file data is improved.

Description

Data reporting methods, devices, computer equipment and storage media

Technical field

This application relates to the technical field of data reporting, and in particular to a data reporting method, device, computer equipment and storage medium.

Background technique

In the cloud scenario, there are many services that need to report the file data of the cloud server. However, the cost of reporting all the massive file data in the cloud server is high, and reporting the full amount requires more resources of the detection engine. Therefore, file data needs to be reported selectively according to different application scenarios.

In the existing technology, for reporting file data in cloud scenarios, file data is usually selected based on historical experience data, thereby completing the selection of reported file data. However, the reporting efficiency of this method is low, and important file data is easily overlooked, so the applicability is low.

At present, no effective solution has been proposed for the problem of low file data reporting efficiency in related technologies.

Contents of the invention

Based on this, it is necessary to address the above technical problems and provide a data reporting method, device, computer equipment and computer-readable storage medium that can improve the efficiency of file data reporting.

In the first aspect, this application provides a data reporting method. The methods include:

Get the data to be reported of the file;

Extract the characteristic values of the data to be reported;

According to the characteristic value, the data to be reported is classified into different bucket files for storage;

Perform similarity calculation on the data to be reported under the same bucket file, cluster the data to be reported according to the similarity, and generate multiple groups of data clusters to be reported, wherein the data clusters to be reported are Contains normal data samples and malicious data samples;

According to the proportion of the normal data samples and the malicious data samples in each group of the data clusters to be reported, each group of the data clusters to be reported is scored, and multiple groups of the data clusters to be reported are selected for reporting based on the scores. .

In one embodiment, the data to be reported includes: file reporting path data and malicious file path data, and the data fields of the file reporting data and the malicious file path data include the first identification code of the file, the file path data, and the first identification code of the file. name, file path, directory and generation time.

In one embodiment, extracting the feature values of the data to be reported includes:

Using the first identification code as the primary key, extract the directory of the file;

Split the directory to obtain multiple byte fragments;

Calculate the second identification codes of the plurality of byte fragments, combine the plurality of second identification codes, and generate the first characteristic value of the data to be reported.

In one embodiment, extracting the feature values of the data to be reported further includes:

Randomly shuffle each row of the first feature value multiple times;

Map the first set corresponding to the first feature value obtained after each shuffling to a second set, wherein each mapped value in the second set does not overlap with each other;

Search the mapping values in the second set in order from small to large until the first characteristic value corresponding to the found mapping value is the first preset value;

Obtain the number of digits corresponding to the found mapping value, and combine multiple of the number of digits to obtain the second feature value;

Classify the data to be reported according to the second characteristic value.

In one embodiment, clustering the data to be reported in the bucket file includes:

Select a piece of unclustered data to be reported, and calculate the similarity between the selected data to be reported and multiple groups of clustered data clusters to be reported in the same bucket file;

When the similarity is greater than the first threshold, merge the selected data to be reported into similar data clusters to be reported;

When the similarity is less than the first threshold, the selected data to be reported is newly created into a data cluster to be reported.

In one embodiment, selecting multiple groups of data clusters to be reported for reporting based on scores includes:

The first N data clusters to be reported with the highest scores are selected for reporting, or the data clusters to be reported with scores exceeding the second threshold are selected for reporting.

In one embodiment, reporting the data cluster to be reported includes:

Divide the data to be reported into multiple directory names according to the path of the data to be reported in the data cluster to be reported;

According to the directory name, perform regular substitution on the path, and calculate the degree of merging of the path after substitution;

When the degree of merging is lower than the third threshold, continue regular replacement of the path until the degree of merging of the path after substitution is higher than the third threshold;

Extract the regular expression of the merge path, calculate the coverage of the regular expression for the corresponding data to be reported, and the global coverage of all the data to be reported in the data cluster to be reported;

According to the coverage rate and the global coverage rate, the regular expression is selected, and the data cluster to be reported is reported according to the selected regular expression.

In a second aspect, this application also provides a data reporting device. The device includes:

The acquisition module is used to obtain the data to be reported in the file;

An extraction module, used to extract the characteristic values of the data to be reported;

Bucketing module: used to classify the data to be reported into different bucket files for storage according to the characteristic value;

A clustering module is used to calculate the similarity of the data to be reported under the same bucket file, cluster the data to be reported according to the similarity, and generate multiple groups of data clusters to be reported, where all The data cluster to be reported contains normal data samples and malicious data samples;

A scoring module is used to score each group of the data clusters to be reported based on the proportion of the normal data samples and the malicious data samples in each group of the data clusters to be reported, and select multiple groups of the data clusters to be reported based on the scores. Report data clusters for reporting.

In a third aspect, this application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Get the data to be reported of the file;

Extract the characteristic values of the data to be reported;

According to the characteristic value, the data to be reported is classified into different bucket files for storage;

Perform similarity calculation on the data to be reported under the same bucket file, cluster the data to be reported according to the similarity, and generate multiple groups of data clusters to be reported, wherein the data clusters to be reported are Contains normal data samples and malicious data samples;

According to the proportion of the normal data samples and the malicious data samples in each group of the data clusters to be reported, each group of the data clusters to be reported is scored, and multiple groups of the data clusters to be reported are selected for reporting based on the scores. .

In a fourth aspect, this application also provides a computer-readable storage medium. The computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by the processor, the following steps are implemented:

Get the data to be reported of the file;

Extract the characteristic values of the data to be reported;

According to the characteristic value, the data to be reported is classified into different bucket files for storage;

Perform similarity calculation on the data to be reported under the same bucket file, cluster the data to be reported according to the similarity, and generate multiple groups of data clusters to be reported, wherein the data clusters to be reported are Contains normal data samples and malicious data samples;

According to the proportion of the normal data samples and the malicious data samples in each group of the data clusters to be reported, each group of the data clusters to be reported is scored, and multiple groups of the data clusters to be reported are selected for reporting based on the scores. .

The above data reporting method, device, computer equipment and storage medium obtain the data to be reported from the file, extract the characteristic values of the data to be reported, and classify the data to be reported into different bucket files for storage according to the characteristic values. In the bucket file The data to be reported are clustered according to the similarity to obtain multiple groups of data clusters to be reported. According to the proportion of normal data samples and malicious data samples in each group of data clusters to be reported, each group of data clusters to be reported is scored and selected based on the scores. Multiple groups of data clusters to be reported are reported, and clustering is used to reduce the reporting of duplicate or similar useless data. The reported data is filtered through scoring, which solves the problem of low file data reporting efficiency in related technologies and reduces the storage of data. The required space improves the efficiency of file data reporting.

Description of the drawings

Figure 1 is an application environment diagram of the data reporting method in one embodiment;

Figure 2 is a schematic flow chart of a data reporting method in one embodiment;

Figure 3 is a flow chart of the first feature value calculation of the data reporting method in one embodiment;

Figure 4 is a flow chart of the second characteristic value calculation of the data reporting method in one embodiment;

Figure 5 is a bucket clustering flow chart of the data reporting method in one embodiment;

Figure 6 is a regular expression extraction diagram of the data reporting method in one embodiment;

Figure 7 is an overall flow chart of a data reporting method in one embodiment;

Figure 8 is a structural block diagram of a data reporting device in one embodiment;

Figure 9 is an internal structure diagram of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

The data reporting method provided by the embodiment of this application can be applied in the application environment as shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system may store data that server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The terminal 102 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, etc. The server 104 can be implemented as an independent server or a server cluster composed of multiple servers.

In one embodiment, as shown in Figure 2, a data reporting method is provided. This method is explained by taking the method applied to the terminal in Figure 1 as an example, and includes the following steps:

Step S202: Obtain the data to be reported of the file.

Among them, the data to be reported of the file includes the file name, file path, whether it is a malicious file, file generation time, etc. After obtaining the data to be reported in the file, the data to be reported will also be cleaned to remove dirty data in the data. Dirty data includes but is not limited to empty data, incomplete path data, garbled data and other types of abnormal data.

Step S204: Extract feature values of the data to be reported.

Among them, features are extracted based on the obtained data to be reported. Features include file directories, the number of files generated globally in the directory within a set period, the number of malicious and normal files, and their proportion to the total number of files. Among them, the cycle setting can be adjusted according to the actual situation, including but not limited to weeks, days, and hours.

Step S206: Classify the data to be reported into different bucket files for storage according to the characteristic values.

Among them, taking the cloud scenario as an example, the amount of file data reported every day is on the order of 10 billion. Therefore, reporting all file data will result in very low reporting efficiency and waste a lot of resources. Therefore, it is necessary to filter the data to be reported and select non-duplicate and appropriate data for reporting. Therefore, in this embodiment of the present application, the data to be reported is first divided into buckets based on the characteristic values of the data to be reported, and possibly similar data is put into the same bucket file.

Step S208: Calculate the similarity of the data to be reported under the same bucket file, cluster the data to be reported according to the similarity, and generate multiple groups of data clusters to be reported, where the data clusters to be reported include normal data samples and malicious data samples. .

Among them, only after the reported data has been divided into buckets, the data in the same bucket file will have the opportunity to calculate whether it is similar, and the data in different bucket files are considered to be dissimilar data. In a distributed environment, data to be reported in the same bucket file will be allocated to the same distributed computing node. In the bucket file, all data are processed one by one for clustering, and multiple groups of data clusters to be reported are obtained. After clustering is completed, the directory structure of the samples under each group of data clusters to be reported is similar, including normal data samples and malicious data samples.

Step S210: Score each group of data clusters to be reported based on the proportion of normal data samples and malicious data samples in each group of data clusters to be reported, and select multiple groups of data clusters to be reported for reporting based on the scores.

Among them, after obtaining multiple groups of data clusters to be reported, the score of the current data cluster to be reported is calculated based on the number, proportion, and number of directories of normal data samples and malicious data samples in each group of data clusters to be reported. This score determines The ranking of the corresponding data cluster to be reported among all data clusters. The higher the score, the higher the ranking, which means that the corresponding data cluster to be reported should be reported.

For example, in a malicious file detection scenario, the reporting principle is that the more samples the current data cluster to be reported represents, and the more malicious data samples there are in the data cluster to be reported, the more worthy the data cluster is of being reported, and vice versa. It is not worth reporting. This is because in the malicious file detection scenario, the background data needs to pay more attention to the directories with reported malicious files, and minimize the reporting of directories without reported malicious files to save resources and improve detection service performance. At this time, the scoring formula for the data cluster to be reported is: number of malicious data samples × M – number of normal data samples × 0.1/(number of directories × 10). Among them, M is a hyperparameter, which can be adjusted according to the actual situation. The default hyperparameter is 100.

In the above data reporting method, when clustering, all the data to be reported are not directly clustered. Instead, the characteristic values of the data to be reported are first extracted, and a preliminary screening of the data to be reported is performed based on the characteristic values, so as to make it possible to Similar data to be reported is put into the same bucket file, and then similarity calculation and clustering are performed on the data to be reported in the same bucket file, which improves the calculation efficiency of clustering. Calculate the similarity of the data to be reported under the same bucket file, cluster the data to be reported based on the similarity, and generate multiple groups of data clusters to be reported. According to the proportion and number of malicious data samples and normal data samples in the data clusters to be reported, Scoring is performed, and data clusters are selected for reporting based on the scores, which greatly reduces the reporting of duplicate or similar useless data. The reported data is filtered through scoring, which solves the problem of low data reporting efficiency in the existing technology and improves data efficiency. The efficiency of reporting reduces the space required to store data.

In one embodiment, the data to be reported includes: file reporting path data and malicious file path data. The data fields of the file reporting data and malicious file path data include the first identification code, file name, file path, directory and generation time of the file. .

Among them, the data to be reported includes normal file reporting path data and malicious file path data. The first identification code includes the md5 of normal files and malicious files. The md5 of the file refers to the hash generated by processing the file through the md5 encryption algorithm. The value is the unique identification code of the file. For different files, if the values of the first identification codes are the same, it means that the files are the same, and vice versa. If the file is modified, the value of its first identification code also changes accordingly.

In this embodiment, by obtaining the data to be reported, the obtained data to be reported is stored in the data platform as original data to provide data support for subsequent data processing and reporting.

In one embodiment, extracting the characteristic value of the data to be reported includes: using the first identification code as the main key to extract the directory of the file. Split the directory to obtain multiple byte fragments. Calculate the second identification codes of the multiple byte fragments, combine the multiple second identification codes, and generate the first characteristic value of the data to be reported.

Among them, the first identification code is used as the primary key to preprocess the data to be reported of the file, including segmenting the file path. In the embodiment of this application, the path segmentation methods used are segmentation by path separators and n garm segmentation. ngarm is an algorithm based on a statistical language model. Its basic idea is to divide the content in the text into bytes with a size of n. The sliding window operation forms a byte fragment sequence of length n. Among them, the size of n can be adjusted according to the actual situation. Calculate the second identification code for the divided file path, that is, the md5 of the divided file path, and combine the generated N second identification codes to generate simhash, which is the first feature value. Simhash is a fingerprint generation algorithm that can perform dimensionality reduction on text to obtain a simhash value. By comparing the simhash values of different texts, the similarity between texts can be determined.

Exemplarily, Figure 3 is a first feature value calculation flow chart of the data reporting method according to the embodiment of the present application. As shown in Figure 3, taking 2 garm as an example, the first identification code of the file is used as the primary key to obtain the directory of the file. , split the directory and obtain multiple byte fragments as the split results, such as /u, us, le, et, etc. Calculate md5 for the byte fragments obtained by the above division, and obtain N second identification codes. Combine N second identification codes bit by bit to generate simhash, which is the first feature value.

In this embodiment, by segmenting the file path and calculating the identification code, the second feature value of the data is obtained, which facilitates subsequent similarity calculation. Using the simhash method, the sensitivity to the order of file data is lower, and the classification accuracy of file data is higher.

In one embodiment, extracting the feature values of the data to be reported further includes: randomly shuffling each row of the first feature value multiple times. The first set corresponding to the first feature value obtained after each shuffling is mapped to the second set, where each mapped value in the second set does not overlap with each other. The mapping values in the second set are searched in ascending order until the first characteristic value corresponding to the mapping value is found to be the first preset value. Obtain the digit number corresponding to the found mapping value, and combine multiple digit numbers to obtain the second feature value. Classify the data to be reported according to the second characteristic value.

Among them, the second eigenvalue is the minhash value, and the Minhash algorithm is the minimum hash function algorithm. A column vector is randomly arranged by row, and the row number of the first non-zero element after rearrangement is the minimum hash function value. In the embodiment of the present application, after the first eigenvalue is calculated, each row in the column vector of the first eigenvalue is randomly scrambled and randomly arranged. The first set corresponding to the first eigenvalue obtained by random arrangement is mapped to the second set. The mapping relationship conforms to the perfect hash function. The perfect hash function refers to mapping each element of the set S to another series without conflict. The hash function of the set, for example, the set {0,1,2,3,4,5,6} is mapped to the new set {3,2,5,1,0,6,4}, and each new set If the number is not repeated, it means that the function is a complete hash function. In this application, each mapping value in the obtained second set does not repeat each other. Each mapping value in the second set is searched in order from small to large until the column of the first feature value corresponding to the found mapping value. The value in the row corresponding to the vector is the first preset value. In the embodiment of this application, the first preset value is 1. After finding the first first preset value appearing in the first set in order from small to large, the binary digit number corresponding to the position of the first preset value is obtained. Since multiple randomizations are performed, the binary digit numbers obtained by the mapping search after multiple randomizations are merged, and the merged result is the second eigenvalue. The above-mentioned process of mapping and searching after randomization and obtaining a binary number is the minhash process. The number of times minhash is obtained determines the size of the bucket. The more times, the longer the second feature value obtained by merging, the greater the number of bucket files, and the less data to be reported in each bucket file, but the clustering The recall rate will then decrease. The more times, the shorter the second feature value obtained by merging, and the higher the recall rate of clustering. Under normal circumstances, taking 2-3 minhash times can meet the clustering requirements of the order of 1 billion. According to the second characteristic value, the data to be reported can be classified, and the data to be reported can be classified into different bucket files. Bucketing is based on the ID calculated from the second feature value. For example, the format can be 12_73_51. All data with ID 12_73_51 will be divided into the same bucket file.

Exemplarily, Figure 4 is a second feature value calculation flow chart of the data reporting method according to an embodiment of the present application. As shown in Figure 4, column A is the binary representation of the first feature value, and column B is the binary representation of the first feature value. Binary digit number. Column C is the binary digit number mapped by the perfect hash function. Take two minhash as an example. In the first time, search according to the number in column C from small to large. The first 1 appears in column A. When , the corresponding binary digit number in column B is 5, then the result this time is 5. In the second time, the search is performed according to the numbers in column C from small to large. When the first 1 appears in column A, the corresponding binary number in column B is 3, so the result this time is 3. Combine the two results to get the minhash, and the final second eigenvalue is 5-1.

In this embodiment, the second eigenvalue is obtained by performing a perfect hash function mapping on the first eigenvalue and then taking a minhash. For a large amount of data, the minhash algorithm is more efficient in calculation, and there is no mandatory requirement for the order of the data. , has a wider scope of application. Bucketing preprocessing of data can improve subsequent clustering efficiency.

In one embodiment, clustering the data to be reported in the bucket file includes: selecting a piece of unclustered data to be reported, and calculating the difference between the selected data to be reported and multiple groups of clustered data to be reported in the same bucket file. similarity. When the similarity is greater than the first threshold, the selected data to be reported are merged into similar data clusters to be reported. When the similarity is less than the first threshold, the selected data to be reported is newly created into a data cluster to be reported.

Among them, only after the data to be reported is divided into buckets according to the second characteristic value, the data in the same bucket file has the opportunity to calculate whether it is similar, and the data in different bucket files are considered to be dissimilar data. In a distributed environment, data in the same bucket file will be distributed to the same distributed computing node (worker). Process all the data to be reported one by one in the same bucket file. When a certain piece of data to be reported is ready for clustering, first calculate whether there are similar data clusters in the clustered data clusters to be reported. If so, , then merge the piece of data to be reported into a similar data cluster to be reported; if it does not exist, create it as a separate data cluster. After processing the data to be reported one by one, the clustering of the data to be reported in the bucket file is completed.

Exemplarily, Figure 5 is a bucket clustering flow chart of a data reporting method according to an embodiment of the present application. As shown in Figure 5, the first feature value is generated by dividing the file directory path, and is calculated through the first feature value. Second eigenvalue, the data to be reported is divided into buckets according to the second eigenvalue, divided into bucket 1 to bucket N, clustering is performed in each bucket file, and finally the results obtained by clustering in each bucket file are merged to obtain The final clustering result.

In this embodiment, data in the same bucket is assigned to the same distributed computing node for clustering calculation, and a hierarchical clustering method is used to merge and aggregate the data based on whether the data is similar, thereby improving the efficiency of clustering.

In one embodiment, selecting multiple groups of data clusters to be reported based on scores includes: selecting the top N data clusters with the highest scores to be reported, or selecting data clusters with scores exceeding a second threshold to be reported.

Among them, after clustering is completed, each data cluster to be reported represents a class. After the score of each data cluster to be reported is calculated, two mechanisms can be used to report the data cluster to be reported based on the score. One is to use the score TopN The sorting method selects the top N data clusters with the highest scores to be reported for reporting; the other method is to set a second threshold, and all data clusters to be reported with scores exceeding the second threshold will be reported.

In this embodiment, scoring is performed based on the proportion of malicious data to determine whether to report the data. Two selection mechanisms are used to select the data to be reported, which is conducive to filtering malicious file data, saves computing resources, and improves the efficiency of the detection service. performance.

In one embodiment, reporting the data cluster to be reported includes: dividing the data to be reported into multiple directory names according to the path of the data to be reported in the data cluster to be reported. Based on the directory name, perform regular substitution on the path, and calculate the degree of merging of the paths after substitution. When the degree of merging is lower than the third threshold, regular replacement of the path is continued until the degree of merging of the replaced path is higher than the third threshold. Extract the regular expression of the merge path, calculate the coverage of the regular expression for the corresponding data to be reported, and the global coverage of all the data to be reported in the data cluster to be reported. According to the coverage rate and global coverage rate, a regular expression is selected, and the data cluster to be reported is reported based on the selected regular expression.

Among them, regular expressions are extracted from the paths of all samples in the data cluster that is decided to be reported. First, divide the directory into N directory names according to the path separator (such as "/"), then calculate the regular pattern based on the directory name, replace the numbers in the directory with the regular pattern \d+, and calculate the degree of merging of the replaced full paths. If After the replacement is completed, if the directory merging degree of the samples under this data cluster is lower than the third threshold, you need to further use regular replacement, for example, replace the "_" and "-" that exist in the N directories after segmentation with \w+. Then further merge the complete catalog of each sample under the cluster. After extracting the regular expression, calculate the coverage of the regular expression on the samples in the current data cluster and the global coverage, select the regular expression with high coverage within the cluster and low global coverage, and treat the reported data according to the selected regular expression Make a report.

Exemplarily, Figure 6 is a regular expression extraction diagram of a data reporting method according to an embodiment of the present application. As shown in Figure 6, the directory is first divided into N directory names according to path separators (such as "/"). Taking /var/www/sites/exam.com/up/16/propic_0/1209923724 as an example, the segmentation results are var, www, sites, exam.com, up, 16, propic_0, 1209923724. Calculate the regular pattern based on the directory name, replace the numbers in the directory with the regular pattern \d+, replace 16 with \d+, and replace 120992372 with \d+. Calculate the degree of merging of the replaced full paths. The directory after the first round of merging is: /var/www/sites/exam.com/up/\d+/propic_0/\d+. If after replacing the directories, the directory merging program of the samples under the cluster is lower than the third threshold, you need to further use regular replacement, for example, replace the "_" and "-" existing in the N directories after splitting with \w+. Then further merge the complete catalog of each sample under the cluster. The final merged directory regular expression is: /var/www/sites/exam.com/up/\d+/ propic_\w+/\d+.

Figure 7 is an overall flow chart of a data reporting method according to an embodiment of the present application. As shown in Figure 7, first collect data from existing files to be reported, and then perform data cleaning and feature extraction after collection. According to the extracted features and file paths, Cluster the data to be reported, calculate and analyze the clustering results, select the file data that needs to be reported based on the clustering results, when the file needs to be reported, extract the regular expression of the file reporting path, and report based on the regular expression. Through the above process, the reporting of duplicate or similar useless logs can be greatly reduced, the efficiency of data reporting can be improved, and the space required for storing reported data can be reduced, making the applicability higher. In addition, the data reporting method in the embodiment of the present application can be used not only for reporting file log data, but also for reporting system process and other user and operating system log data. This application is not limited to this.

It should be understood that although the steps in the flowcharts involved in the above-mentioned embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above embodiments may include multiple steps or stages. These steps or stages are not necessarily executed at the same time, but may be completed at different times. The execution order of these steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

Based on the same inventive concept, embodiments of the present application also provide a data reporting device for implementing the above-mentioned data reporting method. The solution to the problem provided by this device is similar to the solution recorded in the above method. Therefore, for the specific limitations in one or more embodiments of the data reporting device provided below, please refer to the limitations on the data reporting method mentioned above. I won’t go into details here.

In one embodiment, as shown in Figure 8, a data reporting device is provided, including:

The acquisition module 81 is used to obtain the data to be reported of the file;

The extraction module 82 is used to extract the feature values of the data to be reported;

The bucketing module 83 is used to classify the data to be reported into different bucket files for storage according to the characteristic values;

The clustering module 84 is used to calculate the similarity of the data to be reported under the same bucket file, cluster the data to be reported according to the similarity, and generate multiple groups of data clusters to be reported, where the data clusters to be reported contain normal data samples. with malicious data samples;

The scoring module 85 is used to score each group of data clusters to be reported based on the proportion of normal data samples and malicious data samples in each group of data clusters to be reported, and select multiple groups of data clusters to be reported for reporting based on the scores.

Each module in the above-mentioned data reporting device can be implemented in whole or in part by software, hardware and combinations thereof. Each of the above modules may be embedded in or independent of the processor of the computer device in the form of hardware, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in Figure 9. The computer device includes a processor, a memory, an input/output interface (Input/Output, referred to as I/O), and a communication interface. Among them, the processor, memory and input/output interface are connected through the system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage medium stores operating systems, computer programs and databases. This internal memory provides an environment for the execution of operating systems and computer programs in non-volatile storage media. The database of the computer device is used to store data to be reported. The input/output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. The computer program implements a data reporting method when executed by the processor.

Those skilled in the art can understand that the structure shown in Figure 9 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Specific computer equipment can May include more or fewer parts than shown, or combine certain parts, or have a different arrangement of parts.

In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, it implements the following steps:

Obtain the data to be reported from the file; extract the characteristic values of the data to be reported; classify the data to be reported into different bucket files for storage based on the characteristic values; calculate the similarity of the data to be reported under the same bucket file, and calculate the similarity based on the similarity The data to be reported is clustered to generate multiple groups of data clusters to be reported, in which the data clusters to be reported include normal data samples and malicious data samples; according to the proportion of normal data samples and malicious data samples in each group of data clusters to be reported, Score each group of data clusters to be reported, and select multiple groups of data clusters to be reported based on the scores.

In one embodiment, the processor also implements the following steps when executing the computer program:

Obtain the data to be reported of the file. The data to be reported includes: file reporting path data and malicious file path data. The data fields of the file reporting data and malicious file path data include the first identification code, file name, file path, directory and generation of the file. time.

In one embodiment, the processor also implements the following steps when executing the computer program:

Using the first identification code as the main key, extract the directory of the file; split the directory to obtain multiple byte fragments; calculate the second identification codes of the multiple byte fragments, merge the multiple second identification codes, and generate a report to be reported The first eigenvalue of the data.

In one embodiment, the processor also implements the following steps when executing the computer program:

Randomly scramble each row of the first feature value multiple times; map the first set corresponding to the first feature value obtained after each scramble to a second set, where each mapped value in the second set do not repeat each other; search the mapping values in the second set in order from small to large until the first feature value corresponding to the found mapping value is the first preset value; obtain the corresponding value of the found mapping value Digit number, multiple digit numbers are combined to obtain the second characteristic value; according to the second characteristic value, the data to be reported is classified.

In one embodiment, the processor also implements the following steps when executing the computer program:

Select a piece of unclustered data to be reported, and calculate the similarity between the selected data to be reported and multiple groups of clustered data clusters to be reported in the same bucket file; when the similarity is greater than the first threshold, the selected data to be reported will be The data is merged into similar data clusters to be reported; when the similarity is less than the first threshold, the selected data to be reported is newly created into a data cluster to be reported.

In one embodiment, the processor also implements the following steps when executing the computer program:

The top N data clusters with the highest scores to be reported are selected for reporting, or the data clusters with scores exceeding the second threshold are selected for reporting.

In one embodiment, the processor also implements the following steps when executing the computer program:

According to the path of the data to be reported in the data cluster to be reported, the data to be reported is divided into multiple directory names; according to the directory name, the path is replaced by regularization, and the degree of merging of the replaced path is calculated; when the degree of merging is lower than the When the third threshold is reached, continue regular replacement of the path until the degree of merging of the replaced path is higher than the third threshold; extract the regular expression of the merged path, calculate the coverage of the regular expression for the corresponding data to be reported, and the coverage of the corresponding data to be reported. The global coverage of all data to be reported in the data cluster; based on the coverage and global coverage, select a regular expression, and report the data cluster to be reported based on the selected regular expression.

In one embodiment, a computer-readable storage medium is provided with a computer program stored thereon. When the computer program is executed by a processor, the following steps are implemented:

Obtain the data to be reported from the file; extract the characteristic values of the data to be reported; classify the data to be reported into different bucket files for storage based on the characteristic values; calculate the similarity of the data to be reported under the same bucket file, and calculate the similarity based on the similarity The data to be reported is clustered to generate multiple groups of data clusters to be reported, in which the data clusters to be reported include normal data samples and malicious data samples; according to the proportion of normal data samples and malicious data samples in each group of data clusters to be reported, Score each group of data clusters to be reported, and select multiple groups of data clusters to be reported based on the scores.

In one embodiment, the processor also implements the following steps when executing the computer program:

Obtain the data to be reported of the file. The data to be reported includes: file reporting path data and malicious file path data. The data fields of the file reporting data and malicious file path data include the first identification code, file name, file path, directory and generation of the file. time.

In one embodiment, the processor also performs the following steps when executing the computer program:

Using the first identification code as the main key, extract the directory of the file; split the directory to obtain multiple byte fragments; calculate the second identification codes of the multiple byte fragments, merge the multiple second identification codes, and generate a report to be reported The first eigenvalue of the data.

In one embodiment, the processor also implements the following steps when executing the computer program:

Randomly scramble each row of the first feature value multiple times; map the first set corresponding to the first feature value obtained after each scramble to a second set, where each mapped value in the second set do not repeat each other; search the mapping values in the second set in order from small to large until the first feature value corresponding to the found mapping value is the first preset value; obtain the corresponding value of the found mapping value Digit number, multiple digit numbers are combined to obtain the second characteristic value; according to the second characteristic value, the data to be reported is classified.

In one embodiment, the processor also implements the following steps when executing the computer program:

Select a piece of unclustered data to be reported, and calculate the similarity between the selected data to be reported and multiple groups of clustered data clusters to be reported in the same bucket file; when the similarity is greater than the first threshold, the selected data to be reported will be The data is merged into similar data clusters to be reported; when the similarity is less than the first threshold, the selected data to be reported is newly created into a data cluster to be reported.

In one embodiment, the processor also implements the following steps when executing the computer program:

The top N data clusters with the highest scores to be reported are selected for reporting, or the data clusters with scores exceeding the second threshold are selected for reporting.

In one embodiment, the processor also implements the following steps when executing the computer program:

According to the path of the data to be reported in the data cluster to be reported, the data to be reported is divided into multiple directory names; according to the directory name, the path is replaced by regularization, and the degree of merging of the replaced path is calculated; when the degree of merging is lower than the When the third threshold is reached, continue regular replacement of the path until the degree of merging of the replaced path is higher than the third threshold; extract the regular expression of the merged path, calculate the coverage of the regular expression for the corresponding data to be reported, and the coverage of the corresponding data to be reported. The global coverage of all data to be reported in the data cluster; based on the coverage and global coverage, select a regular expression, and report the data cluster to be reported based on the selected regular expression.

It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all It is information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage. In the medium, when the computer program is executed, it may include the processes of the embodiments of the above methods. Any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetic variable memory (Magnetoresistive Random) Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene memory, etc. Volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration and not a limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM). The databases involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include blockchain-based distributed databases, etc., but are not limited thereto. The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to this.

The technical features of the above embodiments can be combined in any way. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, all possible combinations should be used. It is considered to be within the scope of this manual.

The above-described embodiments only express several implementation modes of the present application, and their descriptions are relatively specific and detailed, but should not be construed as limiting the patent scope of the present application. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all fall within the protection scope of the present application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims (10)

1. A data reporting method, characterized by including: Get the data to be reported of the file; Extract the characteristic values of the data to be reported; According to the characteristic value, the data to be reported is classified into different bucket files for storage; Perform similarity calculation on the data to be reported under the same bucket file, cluster the data to be reported according to the similarity, and generate multiple groups of data clusters to be reported, wherein the data clusters to be reported are Contains normal data samples and malicious data samples; According to the proportion of the normal data samples and the malicious data samples in each group of the data clusters to be reported, each group of the data clusters to be reported is scored, and multiple groups of the data clusters to be reported are selected for reporting based on the scores. . 2. The data reporting method according to claim 1, characterized in that the data to be reported includes: file reporting path data and malicious file path data, and the data fields of the file reporting data and the malicious file path data include The first identification code, file name, file path, directory and generation time of the file. 3. The data reporting method according to claim 1, wherein extracting the feature values of the data to be reported includes: Using the first identification code as the primary key, extract the directory of the file; Split the directory to obtain multiple byte fragments; Calculate the second identification codes of the plurality of byte fragments, combine the plurality of second identification codes, and generate the first characteristic value of the data to be reported. 4. The data reporting method according to claim 3, wherein extracting the characteristic value of the data to be reported further includes: Randomly shuffle each row of the first feature value multiple times; Map the first set corresponding to the first feature value obtained after each shuffling to a second set, wherein each mapped value in the second set does not overlap with each other; Search the mapping values in the second set in order from small to large until the first characteristic value corresponding to the found mapping value is the first preset value; Obtain the digit number corresponding to the found mapping value, and combine multiple of the digit numbers to obtain the second feature value; Classify the data to be reported according to the second characteristic value. 5. The data reporting method according to claim 1, wherein clustering the data to be reported in the bucket file includes: Select a piece of unclustered data to be reported, and calculate the similarity between the selected data to be reported and multiple groups of clustered data clusters to be reported in the same bucket file; When the similarity is greater than the first threshold, merge the selected data to be reported into similar data clusters to be reported; When the similarity is less than the first threshold, the selected data to be reported is newly created into a data cluster to be reported. 6. The data reporting method according to claim 1, characterized in that selecting multiple groups of data clusters to be reported according to scores for reporting includes: The first N data clusters to be reported with the highest scores are selected for reporting, or the data clusters to be reported with scores exceeding the second threshold are selected for reporting. 7. The data reporting method according to claim 1, characterized in that reporting the data cluster to be reported includes: Divide the data to be reported into multiple directory names according to the path of the data to be reported in the data cluster to be reported; According to the directory name, perform regular substitution on the path, and calculate the degree of merging of the path after substitution; When the degree of merging is lower than the third threshold, continue regular replacement of the path until the degree of merging of the path after substitution is higher than the third threshold; Extract the regular expression of the merge path, calculate the coverage of the regular expression for the corresponding data to be reported, and the global coverage of all the data to be reported in the data cluster to be reported; According to the coverage rate and the global coverage rate, the regular expression is selected, and the data cluster to be reported is reported according to the selected regular expression. 8. A data reporting device, characterized in that it includes: The acquisition module is used to obtain the data to be reported in the file; An extraction module, used to extract the characteristic values of the data to be reported; Bucketing module: used to classify the data to be reported into different bucket files for storage according to the characteristic value; A clustering module is used to calculate the similarity of the data to be reported under the same bucket file, cluster the data to be reported according to the similarity, and generate multiple groups of data clusters to be reported, where all The data cluster to be reported contains normal data samples and malicious data samples; A scoring module is used to score each group of the data clusters to be reported based on the proportion of the normal data samples and the malicious data samples in each group of the data clusters to be reported, and select multiple groups of the data clusters to be reported based on the scores. Report data clusters for reporting. 9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that when the processor executes the computer program, the processor implements the claims as claimed in The steps of the data reporting method according to any one of claims 1 to 7. 10. A computer-readable storage medium with a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the data reporting method according to any one of claims 1 to 7 are implemented.