CN112468389B - Imap mail reduction method based on tcp recombination - Google Patents

Abstract

The invention discloses an imap mail reduction method based on tcp recombination, which is characterized in that scheme design is carried out by combining characteristics in an imap mail protocol and enumeration conditions of tcp recombination, a development platform is adopted to carry out imap mail reduction processing for a linux platform of an operating system centros 7.4, and maximum reduction mail materials of sub-mails are successfully realized.

Description

Imap mail reduction method based on tcp recombination

Technical Field

The invention relates to the field of data processing, in particular to an imap mail reduction method of tcp recombination.

Background

An imap mail protocol analysis transmission layer adopts a tcp mode to carry out data transmission, tcp sub-packets and tcp recombination are involved in the tcp transmission process, packet loss exists in the transmission process, and the arrival sequence of messages is inconsistent, so that in the transmission process of mail content data packets, the inconsistency of the data packet sequence can cause discontinuous encoding data to cause analysis errors, and meanwhile, the arrival sequence of packet loss retransmission data packets can be later than the end frame of a mail, and identification needs to be carried out in the analysis of sub-mails. The imap mail supports a plurality of commands to be carried out simultaneously, supports the same command to transmit a plurality of mails, and is divided by mail division identifiers.

The imap protocol communication process is mainly characterized in that a plurality of commands are supported to be carried out simultaneously, different commands need to be identified in the communication process, and the server side carries out corresponding response according to different commands. An ambiguous command refers to an ambiguous command that has more than one command that returns a result. Under the condition that multiple commands can be simultaneously carried out, the client sends another command when one command response is not completely received, because the commands are not completely independent, and the execution of one command can affect the execution results of other commands. At this time, if the command is an ambiguous command, the client sends other commands before the response reaches the client, which causes an exception to the execution result. The client is required to have to continue sending the next command after the result of one ambiguous command is successfully reached. For example, the UID command is sent to acquire the content, other UID commands can not be sent until the UID content is acquired, otherwise, the command corresponding to which UID the response is can not be distinguished.

Disclosure of Invention

In view of the current situation and existing problems of the prior art, the invention provides a tcp restructured imap mail reduction method, which enumerates the possible situations of sequence and next sequence numbers corresponding to newly arrived data frames by researching the details of an imap mail protocol and combining the data test result and a tcp restructuring principle, successfully realizes the full coverage of the possible situations, and simultaneously changes the condition of stopping mail storage by combining the characteristic that the arrival sequence of the data frames is inconsistent in the tcp restructuring process, successfully corresponds the analyzed data packets to the storage of corresponding mails, realizes the maximum reduction of mail contents, and the result is basically consistent with the result of tracing tcp data streams by wireshark through test verification.

The technical scheme adopted by the invention is as follows: an imap mail reduction method based on tcp recombination is characterized in that a development platform is adopted to carry out imap mail reduction processing for a linux platform of an operating system centros 7.4, and the reduction processing flow executes the following operations:

firstly, judging whether a communication port is a 143 port, and if the communication port is the 143 port, performing a second step; if the communication port is not 143 ports, the procedure ends.

And step two, performing instruction identification and setting an instruction zone bit.

Step three, judging whether the mail identifier and the command identifier are included, and if the mail identifier and the command identifier are included, performing step four; if the mail identifier and the command identifier are not included, the process goes to step eleven.

Step four, judging whether the mail header flag bit is 0, and if the mail header flag bit is 0, performing step five; and if the mail header flag bit is not 0, performing step eight.

And step five, judging whether the current file descriptor is not empty, if not, executing the step six and then executing the step seven, otherwise, directly executing the step seven.

And step six, closing the current mail storage file and updating the name of the next opened file.

Step seven, opening the mail, storing the related content of the mail, and then performing step eight.

And step eight, updating the identifier tag and the ending identifier, and then performing step nine.

Step nine, judging whether a mail head identifier is included, and if the mail head identifier is included, performing step ten; if the mail header identifier is not included, the process ends directly.

Step ten, marking the position 1 by the mail head.

And eleventh, judging whether the instruction zone bit is 1 and does not comprise a FETCH (UID character string, if yes, directly ending the program, and if not, performing the twelfth step.

Step twelve, judging whether the load initial position and the length of actual writing exist or not, and if so, performing step thirteen; if not, the routine ends.

And thirteen step, carrying out tcp recombinant reduction by using a tcp recombinant reduction module.

In the thirteenth step, the tcp recombinant reduction module performs the following operations in the tcp recombinant reduction process:

firstly, judging whether the frame count of the current mail storage is greater than 0, if not, directly storing the first frame, and after the structure body initialization is carried out, ending the program; if the sequence is larger than 0, judging whether the current frame sequence is smaller than the maximum frame next sequence, if not, sequentially storing the frames, and ending the program; if the current frame next sequence is smaller than the maximum frame next sequence, judging which enumeration condition belongs to, calculating intercepted data corresponding to various enumeration condition calculation formulas, reading data which should be stored after the current frame from a file, writing the intercepted data of the current frame and reading the data, and ending the program.

The beneficial effects produced by the invention are as follows: the invention mainly realizes the full coverage of tcp recombination condition and the effective storage of partial mails, avoids the storage of repeated frames, and realizes the effective interception of partial load repetition condition, so that the data received by a receiver can be restored as much as possible according to the data condition sent by a sender. The invention is an imap mail reduction method based on tcp recombination, which successfully realizes the reduction of the same mail materials by taking the analysis result of wireshark as a reference basis, and simultaneously, the analysis result of wireshark is limited in the whole communication process, namely, the tcp flow is tracked. The invention supports two modes of mail reduction and reading storage data packets to carry out mail material reduction in the process of sending the mails in real time, can realize the storage and reduction of the sub-mails of the mails, and simultaneously, the tcp recombination reduction module can be applied to the communication and reduction of the mails of other tcp/ip communication.

Drawings

FIG. 1 is a flow chart of the reduction process of the present invention;

FIG. 2 is a flow chart of tcp recombinant reduction performed by the tcp recombinant reduction module of the present invention;

FIG. 3 is an enumeration diagram of tcp reassembly according to an embodiment of the present invention.

Detailed Description

The invention is further illustrated by the following figures and examples.

As shown in fig. 1, the reduction process flow of this embodiment operates according to the following steps:

step one, judging whether the port is 143, namely judging a tcp communication port of the data frame, if the port is 143, performing step two, otherwise, identifying that the data packet is not the imap protocol related data packet, and ending the program.

Step two, carrying out instruction identification, setting an instruction zone bit, and specifically operating as follows: analyzing the position of the first space character, if the space position is the first character or the last character, instructing to mark the position 0, otherwise, continuously judging the character after the space, if the space position is the number, searching the next space position and judging the character after the space position, if the character is still the number, instructing to mark the position 0, otherwise, comparing the specific instruction with the content of the current data frame. Instructions commonly used in imap communication include the following: ok, id, no, uid, capability, startls, status, logic, fetch, flags, check, store, authentication, namespace, lsub, list, noop, idle, select, exists, and ap, etc., with the instruction identifying position 1 if the data frame contents include the instruction.

Step three, judging whether the mail identifier and the command identifier UID FETCH are included, and the two identifiers are included at the same time, which means that the step four needs to be carried out when the data packet related to the initial storage of the mail is detected, otherwise, the step eleven needs to be carried out.

In combination with the modification of different versions of the imap protocol, the imap mail includes the following identifiers: RFC822, RFC822.HEADER, RFC822.HEADER.FIELDS, RFC822.HEADER.FIELDS.NOT, RFC822.MIME, RFC822.TEXT, BODY [ ], BODY [ HEADER ], BODY [ HEADER.FIELDS, BODY [ HEADER.FIELDS.NOT, BODY [ MIME ], BODY [ TEXT ], BODY.PEEK [ HEADER ], BODY.PEEK [ HEADER.FIELDS, BODY.PEEK [ MIMD.TEXT, BODY.TEXT.NOT, BODY.PEEK [ TEXT ], the above-mentioned main identifiers relate to two categories, one category is from the old protocol RFC822, one category is a new protocol that uses BODY to replace RFC822, and functions to identify the entire mail BODY identifier for efficient content identification and storage.

And step four, judging whether the mail header flag bit is 0 or not. In the mail transmission process, two transmission modes exist, one mode is that the information of the whole mail is transmitted together and only corresponds to a data packet containing UID FETCH, and the other mode is that the mail header and the mail body are transmitted separately and respectively correspond to a data packet containing UID FETCH. Therefore, it is necessary to detect whether the last stored information is the mail header or the mail body information, i.e., to determine whether the flag bit of the mail header is 0. And if the mail header information is the mail header information, only the step eight is needed, otherwise, the mail body information is the mail body information, the fact that the data packet containing the UID FETCH at present corresponds to the starting position of a new mail is proved, and the step five is needed.

It can be seen that the design process of the scheme adopts a mode of identifying a new mail and stopping the storage of the previous mail, and the main reasons are as follows: identifying the beginning and the end of a command in the imap communication process, simultaneously, a plurality of mails exist in a tcp data packet in the communication process, a mail division is needed, and the identification identifiers of the mail division are r \ n \ r \ n and r \ n \ r \ nReceived from in combination with data query, test data and the like; the identifier of the beginning of the mail content is UID FETCH, and the identifier of the end is identification number OK FETCH COMPLETED \ r \ n. In the process of analyzing and storing the mails, the sub-mail identifiers and the end identifiers both correspond to the end of storage of one mail and correspond to the end frames, but due to the characteristics of tcp communication, data before the end frames may lose retransmission and reach the client end in a sequence after the end frames, so that the file descriptors stored this time cannot be closed immediately after the end frames are identified, but the file descriptors are closed only after the start identifiers of the next mail are waited, and the data packets in the communication process can be stored in the corresponding mails.

And step five, judging that the current file descriptor is not null. And if not, the empty file corresponds to the current storage file. If the file is not empty, the file is required to be closed first and then a new file is opened, namely the step six is executed and then the step seven is executed, otherwise, only the step seven is directly executed.

And step six, closing the current mail storage file and updating the name of the next opened file. The file name is named according to the ip quintuple information, and meanwhile, the number of mails related to the same ip quintuple is recorded, counting needs to be updated, and files are prevented from being stored before being covered.

And step seven, opening the mail, namely newly opening a file to store the related content of the mail. Step eight is required after opening the new mail.

And step eight, updating the identifier tag and the ending identifier. When the mail storage start data packet is identified, the label of the current mail needs to be updated, and the end identifier also contains label information, so that the mail storage start data packet can be correspondingly identified and also needs to be updated. And step eight, recording whether the current data packet is the initial data packet of the mail head, namely performing step nine.

And step nine, judging whether the HEADER identifier HEADER is included or not, wherein the HEADER identifier HEADER is used for judging whether the HEADER identifier HEADER is a HEADER start data packet or not. If yes, the step ten needs to be carried out, and if not, the process is ended directly.

Step ten, marking the position 1 by the mail head. The mail header flag bit is used for assisting in judging whether a new mail storage file needs to be opened, and is represented as a mail header at 1.

And eleventh, judging whether the instruction flag bit is 1 and does not include the FETCH (UID character string, wherein the tenth is the next step which is required to be carried out when the third step judges whether the mail identifier and the command identifier UID FETCH are included and are not established, the eleventh step is used for avoiding the storage of irrelevant data, if the judgment is carried out in the eleventh step, the instruction is other instructions related to the mail communication, the storage is not required, and the process is directly finished, otherwise, the twelfth step is carried out.

Step twelve, calculating the load initial position and length of actual writing. The part is provided with four different labels for identification, and the length of a data packet to be intercepted is calculated at the same time, wherein the four labels are as follows: beginllag (e.g., "FETCH (UID"), startLable (e.g., "r \ n"), endLable (e.g., "r \ n"), finalLable (e.g., "FETCH completed \ r \ n"). The beginllab is an instruction initial data packet identifier, the data packet is divided into an initial data packet and a truncated data packet according to the identifier, the startlab corresponds to an actual truncation position, if the current data packet is the initial packet, the initial position of the data packet is the byte position of the first startlab identified behind the beginllab detected by a load, if the data packet is the truncated data packet, the initial position of the data packet is the initial position of the load, and meanwhile, the endlab and the finalllab respectively correspond to a mail receiving stop mark and a current instruction end mark, so that the length is calculated according to the existence conditions of the two labels in the load, specifically, the two conditions exist simultaneously, only one of the conditions and none of the conditions exist, and redundant characters are intercepted to calculate the data length actually written into the mail storage file. Because the imap adopts endLable ("\\ \ r \ n) \ r \ n" and "\ r \ nReceived: from") as the identifiers of the mail distribution, the identifiers need to be circularly detected in the detection program, the identifiers need to be retreated after the identifiers are identified, and the number of bytes corresponding to the character string Received: from needs to be returned. After the load initial position and length are identified by each judgment, step thirteen is carried out, and tcp recombination reduction is carried out. If the detection does not exist, the routine ends.

Thirteen steps, a tcp recombination reduction module. The data structure adopted by the module is as follows:

struct{

uint32_t tcpSeq;

uint32_t packetLen;

uint32_t realLen;

} tcpSeqlist [150000 ]// seq order used to order tcp

Wherein tcpSeq represents the sequence corresponding to the current load, packetLen represents the length of the current load, realLen represents the length of the current frame which is really stored, nxtSeq represents the next sequence corresponding to the current load frame, and the relationship exists: tcpSeq + packetLen = nxtSeq. The maximum subscript currently stored in the structure is recorded in the program and corresponds to tcpSeqNum. The data structure is stored in a way that tcpSeq is increased when subscript is increased, so that the maximum next sequence is the next sequence corresponding to tcpSeqlist [ tcpSeqNum ], if the tcpSeq of the current frame is not smaller than the value, the next sequence is directly stored, and if not, whether the current storage frame needs to be stored between two adjacent frames of data which are stored before needs to be judged.

As shown in fig. 2, it is a flowchart of the tcp recombinant reduction module, and the specific work flow is as follows: firstly, judging whether the frame count stored in the current mail is greater than 0, if not, directly storing the first frame, and initializing a structural body, otherwise, indicating that the current frame is not the first data frame stored in the current mail, needing to judge the storage position of the current data frame, namely, calculating the maximum next sequence stored currently, comparing the sequence of the current frame with the value, if not, only sequentially storing the current frame, otherwise, indicating that the current frame needs to search for an insertion position between the data frames stored previously, and judging which enumeration condition the current frame belongs to, wherein the specific enumeration condition is introduced as shown in fig. 3. And then calculating the intercepted data according to the enumeration condition obtained by judgment and a calculation formula corresponding to the enumeration condition. And then reading data which should be stored after the current frame from the file, and finally writing the intercepted data of the current frame and reading the data.

To facilitate showing the relationship between the current frame, the i-1 th frame, and the i-th frame data, each frame is identified by two data sequences and a next sequence. The implementation and considerations of tcp recombination are described below.

As shown in FIG. 3, the tcp reassembly cases are enumerated, where seq, seq _ i-1, seq _ i identify the sequences of the current, i-1, and i-th frames, respectively, nxtseq _ i-1, nxtseq _ i identify the next sequences of the current, i-1, and i-th frames, respectively, seq _0, nxtseq _0 represent the sequence and next sequence of the 0-th frame, and seq _ tcpSeNum, xtsq _ tcpSeNum represent the sequence and next sequence of the current largest frame. Seven cases are listed in fig. 3:

case 1 satisfies the condition: seq > = nxtseq _ i-1& & nxtseq < = seq _ i

Case 2 satisfies the condition: i = =0& & nxtseq <= seq _ i

Case 3 satisfies the condition: i = =0& & seq < seq _ i & & nxtseq > = seq _ i & & nxtseq < = nxtseq _ i = xtseq = n

The condition 4 satisfies the condition: seq < = nxtseq _ i _1& & nxtseq > = nxtseq _ i _1& & nxtseq < = seq _ i _1& & nxtseq <= seq _ i < =

Case 5 satisfies the condition: seq < = nxtseq _ i _1& & nxtseq > = seq _ i & & & nxtseq >

Case 6 satisfies the condition: seq > = nxtseq _ i _1& & seq & < = seq _ i & & nxtseq > = seq _ i

Case 7 satisfies the condition:

i==tcpSeqNum-1&&seq>=seq_i&&seq<=nxtseq_i&&nxtseq>=nxtseq_i

wherein case 1 and case 2 do not require further load interception, and the remaining five cases require further load interception.

The following explains how to calculate the length of the load intercept, taking case 3 as an example: three values, lenJudge, tcpSeqNew and offset, are recorded, wherein lenJudge represents the length of the initial calculation, and is calculated according to the sequence and the nextsequence of the current frame, and if the length of the initial calculation is 0, the data of the current frame is stored before and is directly returned. tcpSeqNew denotes the updated sequence, and offset denotes the offset of the actual intercept position of the load relative to the start position. Corresponding to the situation in fig. 3, it is only necessary to intercept the data from seq to seq _0 in the current load, so there are:

lenJudge=seq_0-seq;

tcpSeqNew=seq;

offset=0;

the load length and offset of the clipping start position for the other cases can be obtained by analogy. The length of the data which needs to be intercepted really needs to be calculated next, because the command identifier, the line interactive character, the mail distributing character and the like are included in the mail transmission process in the imap communication process. the tcp reassembly and restoration module will transmit the actual intercepted load position as a parameter, calculate the offset from the load start position as offset, and calculate by combining the magnitude relationship between the offset and the offset:

if offset compare is greater than offset, then there is realLenJudge = lenJudge- (offset compare-offset);

if offset match is not greater than offset, then there is realLenJudge = lenJudge.

And the realLenJudge represents the finally obtained load length, if the load length is more than 0, the storage is continued, and if the load length is not more than 0, the storage is returned. In these seven cases, data exceeding the current seq sequence arrives first and is stored, and it needs to be read out to be spliced with the current frame truncated data for storage. And the data length required to be read can be obtained by performing length calculation according to the realLen in the previously stored record. Meanwhile, the values in the current frame corresponding list need to be updated as follows:

tcpSeq=tcpSeqs;

packetLen=lenJudge;

realLen=realLenJudge;

through the processing, the tcp recombination reduction module successfully realizes the complete sequential storage of the mail materials, thereby realizing the effective reduction of the mail materials.

The development platform is a linux platform with a centros 7.4 operating system, if the communication port is judged not to be 143, the protocol is judged not to be an imap protocol, the program is finished, and otherwise, the content of the data packet needs to be further judged. And in the restoration process, according to the mail identifier used in imap communication, identifying the sub-mail identifier, transmitting the intercepted load specific position into a tcp recombination restoration module, and changing the condition that the mail stops storing into the condition that the start of the next mail is identified by combining the characteristics that a data packet is transmitted and the arrival sequence is inconsistent in the tcp communication process, thereby successfully realizing the restoration of the maximum mail storage.

The invention realizes the full coverage of various possible communication conditions by researching the characteristics in the imap mail protocol communication process, and simultaneously realizes enumeration of tcp recombination conditions, thereby successfully ensuring the reliability of stored mail data.

Claims (1)

1. An imap mail reduction method based on tcp recombination is characterized in that a development platform is adopted to carry out imap mail reduction processing for a linux platform of an operating system centros 7.4, and the reduction processing flow executes the following operations:

firstly, judging whether a communication port is a 143 port, and if the communication port is the 143 port, performing a second step; if the communication port is not 143 ports, the procedure is ended;

step two, carrying out instruction identification and setting an instruction zone bit;

step three, judging whether the mail identifier and the command identifier are included, and if the mail identifier and the command identifier are included, performing step four; if the mail identifier and the command identifier are not included, performing the step eleven;

step four, judging whether the mail header flag bit is 0, and if the mail header flag bit is 0, performing step five; if the mail header flag bit is not 0, performing step eight;

step five, judging whether the current file descriptor is not empty, if not, executing the step six and then executing the step seven, otherwise, directly executing the step seven;

step six, closing the current mail storage file and updating the name of the next opened file;

step seven, opening the mail, storing the related content of the mail, and then performing step eight;

step eight, updating the identifier tag and the ending identifier, and then performing step nine;

step nine, judging whether a mail head identifier is included, and if the mail head identifier is included, performing step ten; if the mail head identifier is not included, the program is directly ended;

step ten, marking the position 1 by a mail head;

step eleven, judging whether the instruction zone bit is 1 and does not include FETCH (UID character string, if yes, directly ending the program, if not, performing step twelve;

step twelve, judging whether the load initial position and the length of actual writing exist or not, and if so, performing step thirteen; if not, the program is ended;

thirteen, carrying out tcp recombinant reduction by a tcp recombinant reduction module;

the tcp recombination reduction module performs the following operation in the tcp recombination reduction process:

firstly, judging whether the frame count of the current mail storage is greater than 0, if not, directly storing the first frame, and after the structure body initialization is carried out, ending the program; if the sequence is larger than 0, judging whether the current frame sequence is smaller than the maximum frame next sequence, if not, sequentially storing the frames, and ending the program; if the next sequence of the maximum frame is less than the next sequence of the maximum frame, judging which enumeration condition belongs to, calculating the length of the load interception corresponding to various enumeration condition calculation formulas, wherein the calculation comprises recording the length of the initial interception, the updated sequence and the offset of the actual interception position of the load relative to the initial position; the tcp restructuring and restoring module takes the transmitted actual intercepted load position as a parameter, calculates the offset relative to the load initial position and the finally obtained load length by combining the offset of the actual intercepted position of the load relative to the initial position, calculates the data length required to be read according to the final load length stored in the previous frame, reads the data which is stored after the current frame from the file, writes the intercepted data and the read data of the current frame, and ends the program.

Images