CN115567412A - Traffic deduplication method, device, electronic equipment and storage medium - Google Patents

Abstract

The present application provides a traffic deduplication method, device, electronic equipment, and storage medium. The method includes: acquiring traffic data to be deduplicated, the traffic data to be deduplicated includes multiple pieces of traffic data; wherein, each piece of traffic data includes a first identifier, a second identifier and recorded traffic, and the first identifier is Generated according to the recording interface corresponding to the recorded traffic, the second identifier is generated according to the sub-call set and service label corresponding to the recorded traffic; according to the first identifier and second identifier corresponding to each piece of traffic data, the The traffic data to be deduplicated is deduplicated. This application improves the efficiency and accuracy of traffic deduplication by using the first identifier corresponding to the recording interface and the second identifier generated by using the service label and the sub-call set to deduplicate traffic.

Description

Traffic deduplication method, device, electronic equipment and storage medium

technical field

The present application relates to the technical field of software development and testing, and specifically relates to a traffic deduplication method, device, electronic equipment and storage medium.

Background technique

After the business service is updated, it is necessary to test the business service through traffic recording and playback. During the test, the test results generated by comparing the traffic data recorded on the live network with the playback traffic data in the playback environment, because there will be a lot of repeated traffic in the recorded traffic, resulting in low test efficiency.

In order to solve this problem, the method of manually labeling the traffic is currently used to deduplicate the recorded traffic. This traffic deduplication method is inefficient.

Contents of the invention

The purpose of the embodiments of the present application is to provide a traffic deduplication method, device, electronic equipment, and storage medium, so as to improve the efficiency of traffic deduplication.

In the first aspect, the embodiment of the present application provides a traffic deduplication method, including:

Obtain traffic data to be deduplicated, the traffic data to be deduplicated includes multiple pieces of traffic data; wherein, each piece of traffic data includes a first identifier, a second identifier, and recorded traffic, and the first identifier is based on the recorded The recording interface corresponding to the traffic is generated, and the second identification is generated according to the sub-call set and the service label corresponding to the recorded traffic;

Deduplication is performed on the traffic data to be deduplicated according to the first identifier and the second identifier corresponding to each piece of traffic data.

In this embodiment of the present application, traffic deduplication is performed by using the first identifier corresponding to the recording interface and the second identifier generated by using the service label and the sub-call set, thereby improving the efficiency and accuracy of traffic deduplication.

In any embodiment, the deduplication of the traffic data to be deduplicated according to the first identifier and the second identifier corresponding to each piece of traffic data includes:

determining the same traffic data in the recorded traffic according to the first identifier and the second identifier;

Part of the same traffic data is removed, and only one piece of traffic data is kept, so as to realize the deduplication of the traffic data to be deduplicated.

In the embodiment of this application, the first identifier can be used to represent the uniqueness of the recording interface, and the second identifier can be used to represent the uniqueness of the sub-call. Therefore, deduplication of the recording traffic through the first identifier and the second identifier can improve Deduplication accuracy.

In any embodiment, before obtaining the traffic data to be deduplicated, the method further includes:

Obtain the recording traffic through the jvm sandbox repeater tool, and the recording traffic includes a set of sub-calls;

extracting the recording interface of the recorded traffic and generating a first identifier according to the recording interface;

Generate a service label corresponding to the recorded traffic, and generate a second identifier according to the service label and the sub-call set.

In this embodiment of the present application, traffic deduplication is performed by using the first identifier corresponding to the recording interface and the second identifier generated by using the service label and the sub-call set, thereby improving the efficiency and accuracy of traffic deduplication.

In any embodiment, the sub-call set includes at least one sub-call; and the call type corresponding to at least one sub-call is determined by the service server in advance according to the hash value of the thread corresponding to the sub-call; wherein, the call type includes a main thread sub-call and asynchronous thread subcalls.

In the embodiment of the present application, by dividing sub-calls into main thread sub-calls and asynchronous thread sub-calls, many unnecessary scenarios can be avoided, and the accuracy of deduplication of recorded traffic can be improved.

In any embodiment, the second identifier is generated according to the service label and the sub-call set of recorded traffic, including:

Generate a main thread subcall set according to the subcall of the main thread subcall according to the call type;

Generate a collection of asynchronous thread subcalls for subcalls of asynchronous thread subcalls according to the call type;

According to the preset format, the main thread sub-call set, the asynchronous thread sub-call set and the business label are generated into target strings;

The target character string is calculated by using a preset algorithm to obtain the second identifier.

In the embodiment of the present application, by dividing sub-calls into main thread sub-calls and asynchronous thread sub-calls, many unnecessary scenarios can be avoided, combined with business tags, the accuracy of deduplication of recorded traffic is improved.

In any embodiment, after the second identification is generated, the method further includes:

storing the first identifier, the second identifier, and the traffic data corresponding to each piece of traffic data in the search server as a piece of traffic data;

Correspondingly, the acquisition of the traffic data to be deduplicated includes:

Obtain the traffic data to be deduplicated within a preset time period from the search server.

Due to the large amount of recorded traffic data, in order to reduce the pressure on the recording platform, store the preprocessed traffic data in the search server, and then read the deduplicated recorded traffic data from the search server to reduce the recording platform purpose of pressure.

In any embodiment, generating the service label corresponding to the recorded traffic includes:

Extract preset fields from each flow data;

The business tag matching the preset field is determined from the pre-stored field tag correspondence.

In the embodiment of the present application, a corresponding service label is generated for each piece of traffic data, and whether the scenarios corresponding to the traffic data are reflected through the service label is the same, so as to improve the accuracy of deduplication of recorded traffic.

In the second aspect, the embodiment of the present application provides a traffic deduplication device, including:

A data acquisition module, configured to acquire traffic data to be deduplicated, where the traffic data to be deduplicated includes multiple pieces of traffic data; wherein, each piece of traffic data includes a first identifier, a second identifier, and recorded traffic, and the first The identification is generated according to the recording interface corresponding to the recording traffic, and the second identification is generated according to the sub-call set and service label corresponding to the recording traffic;

The deduplication module is configured to deduplicate the traffic data to be deduplicated according to the first identification and the second identification corresponding to each piece of traffic data.

In a third aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory, and a bus, wherein,

The processor and the memory communicate with each other through the bus;

The memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the method of the first aspect.

In a fourth aspect, the embodiment of the present application provides a non-transitory computer-readable storage medium, including:

The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the method of the first aspect.

Other features and advantages of the present application will be set forth in the ensuing description and, in part, will be apparent from the description, or can be learned by practicing the embodiments of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings that need to be used in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application, so It should not be regarded as a limitation on the scope, and those skilled in the art can also obtain other related drawings according to these drawings without creative work.

FIG. 1 is a schematic flow chart of a traffic deduplication method provided in an embodiment of the present application;

FIG. 2 is a schematic flowchart of another traffic deduplication method provided in the embodiment of the present application;

FIG. 3 is a schematic structural diagram of a flow deduplication device provided in an embodiment of the present application;

FIG. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

detailed description

Embodiments of the technical solutions of the present application will be described in detail below in conjunction with the accompanying drawings. The following examples are only used to illustrate the technical solution of the present application more clearly, and therefore are only examples, rather than limiting the protection scope of the present application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of the application; the terms used herein are only for the purpose of describing specific embodiments, and are not intended to To limit this application; the terms "comprising" and "having" and any variations thereof in the specification and claims of this application and the description of the above drawings are intended to cover a non-exclusive inclusion.

In the description of the embodiments of the present application, technical terms such as "first" and "second" are only used to distinguish different objects, and should not be understood as indicating or implying relative importance or implicitly indicating the number, specificity, or specificity of the indicated technical features. Sequence or primary-secondary relationship. In the description of the embodiments of the present application, "plurality" means two or more, unless otherwise specifically defined.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The occurrences of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is understood explicitly and implicitly by those skilled in the art that the embodiments described herein can be combined with other embodiments.

In the description of the embodiment of the present application, the term "and/or" is only a kind of association relationship describing associated objects, which means that there may be three kinds of relationships, such as A and/or B, which may mean: A exists alone, and A exists at the same time and B, there are three cases of B alone. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship.

In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two), similarly, "multiple groups" refers to more than two groups (including two), and "multiple pieces" refers to More than two pieces (including two pieces).

In the description of the embodiments of the present application, the technical terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical" "Horizontal", "Top", "Bottom", "Inner", "Outer", "Clockwise", "Counterclockwise", "Axial", "Radial", "Circumferential", etc. indicate the orientation or positional relationship based on the drawings Orientation or positional relationship is only for the convenience of describing the embodiment of the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as an implementation of the present application. Example limitations.

In the description of the embodiments of this application, unless otherwise clearly specified and limited, technical terms such as "installation", "connection", "connection" and "fixation" should be interpreted in a broad sense, for example, it can be a fixed connection or a fixed connection. Disassembled connection, or integration; it can also be a mechanical connection, or an electrical connection; it can be a direct connection, or an indirect connection through an intermediary, and it can be the internal communication of two components or the interaction relationship between two components. Those of ordinary skill in the art can understand the specific meanings of the above terms in the embodiments of the present application according to specific situations.

The daily traffic generated in the production environment is massive and there are a lot of repetitions. A large number of repeated business requests cannot realize the effective testing of the business system under test. Therefore, a solution that can deduplicate a large number of system business requests in the production environment is needed.

At present, traffic can be controlled by manually labeling traffic data, that is, manually setting a corresponding label for each traffic, and removing traffic belonging to the same label, so that only one traffic with the same label is kept, realizing traffic control. Deduplication operation. However, in the face of a large amount of traffic, the method of manual tagging will lead to a slower efficiency in traffic deduplication.

In order to solve the above technical problems, the inventor of the present application provides a traffic deduplication method, that is, to distinguish the similarities and differences of the traffic data according to the recording interface, the service label corresponding to each traffic data, and the sub-call set corresponding to each traffic data , to achieve the purpose of deduplication of recorded traffic.

Before introducing the specific scheme of this application, in order to better understand this application, the relevant concepts involved are introduced:

Entry call: The call formed by an external request is called an entry call. Currently, the types supported by the traffic playback platform include http, dubbo, mq (rocketMQ), etc.

Sub-call: a sub-call formed by an out-of-process or java (requires enhancement) method, such as an out-of-process call (redis, mybatis), java method enhancement, custom sub-call, etc. Among them, sub-calls can be divided into main thread sub-calls and asynchronous thread sub-calls.

jvm sandbox repeater: It is a general solution for server-side recording/playback based on Jvm-Sandbox.

Recording: The process of serializing and storing the input parameters, output parameters, downstream RPC, DB, cache, etc. of a request.

Playback: the process of restoring the recorded data, re-initiating one or N times of requests, and performing MOCK on specific downstream nodes.

MOCK: During playback, the intercepted sub-call will not actually be called, and the return value during recording will be returned directly by using the process intervention capability of Sandbox.

Elasticsearch: search server, is a distributed, highly scalable, high real-time search and data analysis engine. It can easily enable large amounts of data to be searched, analyzed and explored. Taking full advantage of the horizontal scalability of Elasticsearch can make data more valuable in the production environment. The implementation principle of Elasticsearch is mainly divided into the following steps. First, the user submits the data to the Elasticsearch database, and then uses the word segmentation controller to segment the corresponding sentence, and stores its weight and word segmentation results into the data. When the user searches for data At that time, the results will be ranked and scored according to the weight, and then the returned results will be presented to the user.

It can be understood that the traffic deduplication method provided in the embodiment of the present application can be applied to an electronic device, and the electronic device can be a terminal or a server; the terminal can specifically be a smart phone, a tablet computer, a computer, or a personal digital assistant (PDA). , PDA), etc.; the server can specifically be an application server or a Web server. It can be understood that a traffic recording platform runs in the electronic device, and the traffic recording platform can communicate with the service server to receive the recorded traffic sent by the service server.

Figure 1 is a schematic flow chart of a traffic deduplication method provided in the embodiment of the present application. As shown in Figure 1, the method includes:

Step 101: Obtain traffic data to be deduplicated, the traffic data to be deduplicated includes multiple pieces of traffic data; wherein, each piece of traffic data includes a first identifier, a second identifier, and recorded traffic, and the first identifier is based on The recording interface corresponding to the recording traffic is generated, and the second identification is generated according to the sub-call set and service label corresponding to the recording traffic;

Step 102: Deduplicate the traffic data to be deduplicated according to the first identifier and the second identifier corresponding to each piece of traffic data.

In step 101, the traffic data to be deduplicated may be pre-stored in a preset location, for example, may be pre-stored in a search server or a database. The electronic device may acquire traffic data within a preset time period from the search server according to a preset period to form traffic data to be deduplicated. For example: the preset cycle can be one day, and the electronic device can obtain all the traffic data of the previous day at 10 am every day.

Since there is a lot of traffic data generated in a day, and there is also a lot of duplicate traffic data, it is necessary to deduplicate the duplicate traffic data. It can be understood that the so-called repeated traffic data refers to the traffic data corresponding to the same scene under the same recording interface. Take online shopping as an example: two users submit orders in the same event on the same online shopping platform, and the traffic data corresponding to the two orders are duplicate traffic data.

The first identifier is used to represent the globally unique identifier of the recording interface corresponding to the recording traffic. It can be obtained by calculating the recording interface by using the MD5 algorithm, or by calculating the recording interface by using the hash algorithm, or by using other algorithms. The embodiment of the application does not specifically limit this.

Recorded traffic is the business request sent by the user terminal to the business server, and the business server responds to the business traffic generated in the process of the business request, and sends the business traffic to the electronic device through the preset request method, and is recorded by the traffic in the electronic device It is recorded by the platform. It can be understood that one service request can correspond to one recording traffic. The service server can send the service flow to the electronic device through an http request. The http request includes a recording interface, so the recording interface can be extracted from the http request.

The second identifier is used to characterize the globally unique identifier of the recorded traffic, which is obtained by calculating the sub-call set and service label corresponding to the recorded traffic according to a preset algorithm. The preset algorithm can be a one-way function algorithm, and the one-way function It is difficult for the algorithm to have the result of the function output, and reverse the input specific data. Wherein, the one-way function algorithm may be a cryptographic hash function, specifically an MD5 algorithm, a hash algorithm, or the like. Wherein, the recording traffic includes multiple sub-call sets, wherein the multiple sub-call sets may include main thread sub-calls, and may also include asynchronous thread sub-calls.

The service label is used to represent the type of recorded traffic, which can be determined by some specific fields in the recorded traffic. It can be understood that the recorded traffic may correspond to one service label, or may correspond to multiple service labels, or there may be no corresponding service label. For the case of no service tag, the electronic device may generate the second identifier according to the sub-call set.

In step 102, since the same recording interface may include traffic data of multiple scenarios, and the same scenario may include multiple pieces of traffic data, it is necessary to deduplicate the multiple pieces of traffic data belonging to the same scenario in the same recording interface. Moreover, even if the traffic data contained in different recording interfaces is the same, they are considered as different traffic. Therefore, in order to improve the accuracy of traffic deduplication, the first identifier can be used to distinguish the recording interface, and the second identifier can be used to distinguish the traffic data, so that deduplication is performed on the data to be deduplicated according to the first identifier and the second identifier.

In this embodiment of the present application, traffic deduplication is performed by using the first identifier corresponding to the recording interface and the second identifier generated by using the service label and the sub-call set, thereby improving the efficiency and accuracy of traffic deduplication.

On the basis of the above embodiments, the deduplication of the traffic data to be deduplicated according to the first identifier and the second identifier corresponding to each piece of traffic data includes:

determining the same traffic data in the recorded traffic according to the first identifier and the second identifier;

Part of the same traffic data is removed, and only one piece of traffic data is kept, so as to realize the deduplication of the traffic data to be deduplicated.

In the specific implementation process, the deduplication method of spark can be used for deduplication. Among them, spark includes a variety of deduplication methods, such as: distinct deduplication, group by operation, row_number window operation, etc. Among them, distinct deduplication is to use one or more fields as the basis for deduplication. When the values corresponding to the fields in two pieces of data are the same, the two pieces of data are considered to be the same data, and only one of the pieces of data is kept. The group by operation is to use the deduplication column as an aggregation field to achieve the purpose of deduplication through aggregation. For example: use the first identifier and the second identifier as the aggregation field, and aggregate the traffic data according to the first identifier and the second identifier, and only keep A piece of traffic data after aggregation. In this way, the deduplication operation of the traffic data to be deduplicated is realized.

In the embodiment of this application, the first identifier can be used to represent the uniqueness of the recording interface, and the second identifier can be used to represent the uniqueness of the sub-call. Therefore, deduplication of the recording traffic through the first identifier and the second identifier can improve Deduplication accuracy.

On the basis of the above embodiments, before obtaining the traffic data to be deduplicated, the method further includes:

Obtain the recording traffic through the jvm sandbox repeater tool, and the recording traffic includes a set of sub-calls;

extracting the recording interface of the recorded traffic and generating a first identifier according to the recording interface;

Generate a service label corresponding to the recorded traffic, and generate a second identifier according to the service label and the sub-call set.

In the specific implementation process, the recorded traffic is obtained by recording the service request by the jvm sandbox repeater tool running in the electronic device. For the relevant explanation of the recorded traffic, refer to the above-mentioned embodiments, and will not be repeated here. When the service server responds to the service request sent by the client, it may make multiple calls. Therefore, the recorded traffic includes traffic data corresponding to multiple sub-calls, and the traffic data corresponding to multiple sub-calls constitutes a sub-call set.

When the service server reports the recording traffic to the electronic device, the reporting request will carry the recording interface, for example, the recording interface can be extracted from the http request through the ingress call. After the recording interface is obtained, a first identifier corresponding to the recording interface may be generated according to a preset algorithm, and details may be referred to the foregoing embodiments.

After acquiring the recorded traffic, the electronic device determines the service label corresponding to the recorded traffic according to a preset field in the recorded traffic. It is understandable that the corresponding relationship between fields and business tags can be stored in the electronic device in advance. After the recording volume is obtained, each field in the recorded traffic is matched with the preset field. If the matching is successful, the recorded traffic The field that matches successfully in is the default field. The business label of the preset field is determined from the pre-stored corresponding relationship.

After the electronic device obtains the service tag, it uses a preset algorithm to generate a second identifier for the service tag and the sub-call set. For the specific preset algorithm, refer to the above-mentioned embodiments, which will not be repeated here.

In this embodiment of the present application, traffic deduplication is performed by using the first identifier corresponding to the recording interface and the second identifier generated by using the service label and the sub-call set, thereby improving the efficiency and accuracy of traffic deduplication.

On the basis of the foregoing embodiments, the invocation type of the sub-invocation may be determined in the following manner:

A hash value corresponding to the main thread is pre-stored in the electronic device, and the electronic device calculates the hash value of the thread corresponding to the sub-call, compares the calculated hash value with the stored hash value of the main thread, and if they are consistent, It is then determined that the sub-call is a main thread sub-call, and if not consistent, then it is determined that the sub-call is an asynchronous thread sub-call.

On the basis of the above-mentioned embodiments, the generation of the second identifier according to the service label and the sub-call set of the recorded traffic includes:

Generate a main thread subcall set according to the subcall of the main thread subcall according to the call type;

Generate an asynchronous thread subcall set for subcalls of asynchronous thread subcalls according to the call type;

generating a target string from the main thread sub-call set, the asynchronous thread sub-call set, and the service tag according to a preset format;

The target character string is calculated by using a preset algorithm to obtain the second identifier.

In the specific implementation process, after the electronic device determines the call type of each sub-call, it can form the sub-calls belonging to the main thread sub-call to form the main thread sub-call set, and the sub-calls belonging to the asynchronous thread sub-call to form the asynchronous thread sub-call set. Call collection. Then combine the sub-call set of the main thread, the sub-call set of the asynchronous thread and the business label to form a large set, that is, the target string. Specifically, the main thread sub-call set, the asynchronous thread sub-call set, and the business label can be separated by commas in English to obtain the target string, that is, the target string = main thread sub-call set, asynchronous thread sub-call set, business Label.

After the target character string is obtained, a preset algorithm such as MD5 algorithm or hash algorithm can be used to calculate the target character string to obtain the second identifier.

In the embodiment of the present application, by dividing sub-calls into main thread sub-calls and asynchronous thread sub-calls, many unnecessary scenarios can be avoided, combined with business tags, the accuracy of deduplication of recorded traffic is improved.

On the basis of the above embodiments, after generating the second identifier, the method further includes:

storing the first identifier, the second identifier and the recorded traffic as a piece of traffic data in the search server;

Correspondingly, the acquisition of the traffic data to be deduplicated includes:

Obtain the traffic data to be deduplicated within a preset time period from the search server.

In a specific implementation process, since the service server can generate a large amount of service traffic in a short period of time, the electronic device can obtain a large amount of recording traffic in real time. If the recording traffic is deduplicated in real time, it will increase the load pressure on the electronic equipment. In order to reduce the load pressure on electronic equipment, in the embodiment of the present application, after obtaining the first identifier and the second identifier corresponding to the recorded traffic, the recorded traffic and the first identifier and the second identifier corresponding to the recorded traffic are generated into a piece of traffic data, and the The generated traffic data is stored in the search server. It can be understood that the traffic data may also include information such as a time stamp corresponding to the recorded traffic. It can be understood that the time stamp of the recorded traffic may be the generation time of the recorded traffic, the time when the recorded traffic is stored in the search server, or the time when the electronic device receives the recorded traffic, etc.

The electronic device may acquire traffic data within a preset time period from the search server, and refer to the acquired traffic data as traffic data to be deduplicated. Among them, the preset time period can be the time period input by the tester, for example, it can be the traffic data from October 1 to 2, 2022; the preset time period can also be a time period determined according to the current time, for example: it can be Traffic data of the day before the date corresponding to the current time.

It can be understood that, in order to reduce the storage pressure of the search server, the traffic data in the search server can be cleared periodically, for example, the traffic data two weeks before the current time can be cleared.

Due to the large amount of recorded traffic data, in order to reduce the pressure on the recording platform, store the preprocessed traffic data in the search server, and then read the deduplicated recorded traffic data from the search server to reduce the recording platform purpose of pressure.

Fig. 2 is a schematic flow chart of another traffic deduplication method provided in the embodiment of the present application. As shown in Fig. 2, the method includes:

The business server judges whether the sub-call is an asynchronous thread: After the business server generates business traffic according to the business request sent by the client, it judges whether each sub-call contained in the business traffic is an asynchronous thread, and marks the sub-call. It can be understood that, the method for judging whether the thread corresponding to the sub-call is an asynchronous thread refers to the above-mentioned embodiment, and will not be repeated here. And "0" may be used to indicate that the thread corresponding to the sub-call is the main thread, and "1" may be used to indicate that the thread corresponding to the sub-call is an asynchronous thread. It can be understood that other marking methods can also be used for marking, which is not specifically limited in this embodiment of the present application.

The service server sends the service flow to the flow recording platform (jvm sandbox repeater) of the electronic device, specifically, the service flow may be sent through an http request (jvm sandbox repeater module). A flow recording platform controller (jvm sandbox repeater console) in the electronic device records the service flow after receiving the service flow to obtain the recorded flow.

The electronic device obtains the recording interface from the recording traffic sent by the service server, and generates an ingress MD5 according to the recording interface.

The electronic device adds the sub-call set of the main thread, the sub-call set of the asynchronous thread and the business label into a large set, and connects them with English commas to form a string (ie, the above-mentioned target string), and performs MD5 calculation on the string , to obtain sub-MD5.

The ingress MD5, sub-MD5 and recorded traffic form a traffic data and store it in elastic search.

Take out multiple traffic data from elastic search, and use spark to deduplicate the recorded traffic according to the ingress MD5 and sub-MD5.

Store the deduplicated traffic data in a preset database, for example: mysql database.

The embodiment of the present application controls the flow by using the business label, and combines the sub-call set of the main thread and the sub-call set of the asynchronous thread to ensure the basic skeleton of the recorded business request, on the basis of improving the efficiency of deduplication of the recorded traffic In addition, the accuracy of deduplication is improved.

Fig. 3 is a schematic structural diagram of a traffic deduplication device provided in an embodiment of the present application, and the device may be a module, program segment or code on an electronic device. It should be understood that the device corresponds to the above-mentioned method embodiment in FIG. 1 , and can perform various steps involved in the method embodiment in FIG. 1 . The specific functions of the device can refer to the description above. To avoid repetition, detailed descriptions are appropriately omitted here. The device includes: a data acquisition module 301 and a deduplication module 302; wherein:

The data acquisition module 301 is used to obtain traffic data to be deduplicated, and the traffic data to be deduplicated includes multiple pieces of traffic data; wherein, each piece of traffic data includes a first identifier, a second identifier, and recorded traffic, and the first The identification is generated according to the recording interface corresponding to the recording traffic, and the second identification is generated according to the sub-call set and service label corresponding to the recording traffic;

The deduplication module 302 is configured to deduplicate the traffic data to be deduplicated according to the first identifier and the second identifier corresponding to each piece of traffic data.

On the basis of the foregoing embodiments, the deduplication module 302 is specifically used for:

determining the same traffic data in the recorded traffic according to the first identifier and the second identifier;

Part of the same traffic data is removed, and only one piece of traffic data is kept, so as to realize the deduplication of the traffic data to be deduplicated.

On the basis of the above-mentioned embodiments, the device also includes a flow preprocessing module for:

Obtain the recording traffic through the jvm sandbox repeater tool, and the recording traffic includes a set of sub-calls;

extracting the recording interface of the recorded traffic and generating a first identifier according to the recording interface;

Generate a service label corresponding to the recorded traffic, and generate a second identifier according to the service label and the sub-call set.

On the basis of the above embodiments, the set of sub-calls includes at least one sub-call; and the call type corresponding to the at least one sub-call is determined by the service server in advance according to the hash value of the thread corresponding to the sub-call; wherein, The call types include main thread sub-call and asynchronous thread sub-call.

On the basis of the above embodiments, the traffic preprocessing module is specifically used for:

Generate a main thread subcall set according to the subcall of the main thread subcall according to the call type;

Generate an asynchronous thread subcall set for subcalls of asynchronous thread subcalls according to the call type;

generating a target string from the main thread sub-call set, the asynchronous thread sub-call set, and the service tag according to a preset format;

The target character string is calculated by using a preset algorithm to obtain the second identifier.

On the basis of the foregoing embodiments, the device also includes a data storage module for:

storing the first identifier, the second identifier and the recorded traffic as a piece of traffic data in the search server;

Correspondingly, the traffic preprocessing module is specifically used for:

Obtain the traffic data to be deduplicated within a preset time period from the search server.

On the basis of the above embodiments, the traffic preprocessing module is specifically used for:

Extract preset fields from each flow data;

The service tag matching the preset field is determined from the pre-stored field tag correspondence.

FIG. 4 is a schematic diagram of the physical structure of the electronic device provided by the embodiment of the present application. As shown in FIG. 4, the electronic device includes: a processor (processor) 401, a memory (memory) 402, and a bus 403; wherein,

The processor 401 and the memory 402 communicate with each other through the bus 403;

The processor 401 is used to call the program instructions in the memory 402 to execute the methods provided by the above method embodiments, for example, including: acquiring traffic data to be deduplicated, the traffic data to be deduplicated includes multiple traffic data; wherein, each piece of traffic data includes a first identification, a second identification and recording traffic, the first identification is generated according to the recording interface corresponding to the recording traffic, and the second identification is based on the recording traffic The corresponding sub-call set and service label are generated; according to the first identifier and the second identifier corresponding to each piece of traffic data, deduplication is performed on the traffic data to be deduplicated.

The processor 401 may be an integrated circuit chip with signal processing capability. Above-mentioned processor 401 can be general-purpose processor, comprises central processing unit (Central Processing Unit, CPU), network processor (NetworkProcessor, NP) etc.; Can also be digital signal processor (DSP), application-specific integrated circuit (ASIC), Field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can realize or execute various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

Memory 402 may include but not limited to random access memory (Random Access Memory, RAM), read-only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable read-only memory (Erasable Programmable Read-Only Memory, EPROM), Electrically Erasable Programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), etc.

This embodiment discloses a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by the computer, the computer The methods provided by the above method embodiments can be executed, for example, including: acquiring traffic data to be deduplicated, the traffic data to be deduplicated includes multiple pieces of traffic data; wherein, each piece of traffic data includes a first identifier, a second Identifying and recording traffic, the first identification is generated according to the recording interface corresponding to the recording traffic, the second identification is generated according to the sub-call set and service label corresponding to the recording traffic; according to the corresponding The first identifier and the second identifier deduplicate the traffic data to be deduplicated.

This embodiment provides a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the methods provided in the foregoing method embodiments, for example including : Obtain traffic data to be deduplicated, the traffic data to be deduplicated includes multiple pieces of traffic data; wherein, each piece of traffic data includes a first identifier, a second identifier, and recorded traffic, and the first identifier is based on the The recording interface corresponding to the recording traffic is generated, and the second identification is generated according to the sub-call set and the service label corresponding to the recording traffic; according to the first identification and the second identification corresponding to each piece of traffic data, the to-be-deduplicated Traffic data is deduplicated.

In the embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

In addition, a unit described as a separate component may or may not be physically separated, and a component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Furthermore, each functional module in each embodiment of the present application may be integrated to form an independent part, each module may exist independently, or two or more modules may be integrated to form an independent part.

In this document, relational terms such as first and second etc. are used only to distinguish one entity or operation from another without necessarily requiring or implying any such relationship between these entities or operations. Actual relationship or sequence.

The above descriptions are only examples of the present application, and are not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and changes may be made to the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims (10)

1. A method for traffic deduplication, comprising:

acquiring data of a flow to be deduplicated, wherein the data of the flow to be deduplicated comprises a plurality of pieces of flow data; each piece of traffic data comprises a first identifier, a second identifier and recording traffic, wherein the first identifier is generated according to a recording interface corresponding to the recording traffic, and the second identifier is generated according to a sub-call set and a service label corresponding to the recording traffic;

and according to the first identifier and the second identifier corresponding to each piece of flow data, carrying out deduplication on the flow data to be deduplicated.

2. The method according to claim 1, wherein the performing deduplication on the to-be-deduplicated flow data according to the first identifier and the second identifier corresponding to each flow data includes:

determining the same stream data in the recording stream according to the first identifier and the second identifier;

and performing partial elimination on the same flow data, and only reserving one flow data to realize the deduplication of the flow data to be deduplicated.

3. The method of claim 1, wherein prior to obtaining the data of the stream to be deduplicated, the method further comprises:

acquiring a recording flow through a jvm sandbox repeater tool, wherein the recording flow comprises a sub-call set;

extracting a recording interface for recording the flow and generating a first identifier according to the recording interface;

and generating a service label corresponding to the recording flow, and generating a second identifier according to the service label and the sub-call set.

4. The method of claim 3, wherein the set of sub-calls comprises at least one sub-call; the calling type corresponding to the at least one sub-call is determined by the service server in advance according to the hash value of the thread corresponding to the sub-call; the calling types comprise a main thread sub-call and an asynchronous thread sub-call.

5. The method of claim 4, wherein generating the second identifier according to the service label and the set of sub-calls of the recording traffic comprises:

generating a main thread sub-call set for the sub-call of the main thread sub-call according to the call type;

generating an asynchronous thread sub-call set for the sub-call of the asynchronous thread sub-call according to the call type;

generating a target character string from the main thread sub-call set, the asynchronous thread sub-call set and the service label according to a preset format;

and calculating the target character string by using a preset algorithm to obtain the second identifier.

6. The method according to claim 3, wherein the generating the service label corresponding to the recording traffic includes:

extracting a preset field from each piece of flow data;

and determining the service label matched with the preset field from the corresponding relation of the prestored field labels.

7. The method according to any of claims 3-6, wherein after generating the second identity, the method further comprises:

storing the first identifier, the second identifier and the recording flow as a piece of flow data in a search server;

correspondingly, the acquiring the data of the to-be-deduplicated flow includes:

and acquiring the data of the flow to be deduplicated within a preset time period from the search server.

8. A flow de-weighting device, comprising:

the data acquisition module is used for acquiring to-be-deduplicated flow data, and the to-be-deduplicated flow data comprises a plurality of pieces of flow data; each piece of flow data comprises a first identifier, a second identifier and a recording flow, wherein the first identifier is generated according to a recording interface corresponding to the recording flow, and the second identifier is generated according to a sub-call set and a service label corresponding to the recording flow;

and the duplication removing module is used for carrying out duplication removal on the to-be-duplicated flow data according to the first identification and the second identification corresponding to each flow data.

9. An electronic device, comprising: a processor, a memory, and a bus, wherein,

the processor and the memory are communicated with each other through the bus;

the memory stores program instructions executable by the processor, the program instructions being invoked by the processor to perform the method of any of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1-7.

Images