CN106446028B - Indexing system of network session packet - Google Patents
Indexing system of network session packet Download PDFInfo
- Publication number
- CN106446028B CN106446028B CN201610773444.8A CN201610773444A CN106446028B CN 106446028 B CN106446028 B CN 106446028B CN 201610773444 A CN201610773444 A CN 201610773444A CN 106446028 B CN106446028 B CN 106446028B
- Authority
- CN
- China
- Prior art keywords
- information
- network session
- retrieval
- time
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000006835 compression Effects 0.000 claims abstract description 22
- 238000007906 compression Methods 0.000 claims abstract description 22
- 238000013507 mapping Methods 0.000 claims abstract description 5
- 238000000034 method Methods 0.000 claims description 20
- 238000010586 diagram Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9014—Indexing; Data structures therefor; Storage structures hash tables
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
- H04L67/5651—Reducing the amount or size of exchanged application data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Computer And Data Communications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides an index system of a network session packet, which comprises at least one index space for storing retrieval information of certain information of the network session packet, an information compression module, a retrieval information projection module and a reading module. The index space is internally provided with a plurality of storage bits, one storage bit corresponds to one retrieval information, and all the storage bits have the same initial value; the information compression module is used for carrying out compression operation on some or several kinds of information of the specified network session packet to obtain retrieval information corresponding to each piece of information; the retrieval information projection module is used for projecting and mapping the related information of the network session packet to corresponding storage positions of an index space for storing retrieval information of the information, and the projected storage positions are marked as projected when being projected for the first time; the reading module is used for finding out the corresponding network session packet. The invention can greatly improve the system performance.
Description
Technical Field
The invention belongs to the field of data indexing, and particularly relates to an indexing system of a network session packet.
Background
The current index is established by taking four tuples as KEY, and under the condition of mass data flow, the processing performance and capacity of a HASH table cannot be met. At present, the mainstream HASH processing limit scale is about 1000 ten thousand, the size of the memory occupied by the index block is not fixed, the memory size is increased along with the increase of the data volume, the fixed memory cannot be allocated in advance, and frequent and invalid memory application release needs to be borne.
Disclosure of Invention
In order to solve the above problem, the present invention provides an index system for network session packets, which comprises the following modules:
the system comprises at least one index space for storing retrieval information of certain information of a network session packet, an information compression module, a retrieval information projection module and a reading module.
The index space is internally provided with a plurality of storage bits, one storage bit corresponds to one retrieval information, and all the storage bits have the same initial value.
The information compression module is used for carrying out compression operation on some or several kinds of information of the specified network session packet to obtain retrieval information corresponding to each piece of information.
The retrieval information projection module is used for projecting and mapping the related information of the network session packet to corresponding storage bits of an index space for storing the retrieval information of the information, and the projected storage bits are marked as projected when being projected for the first time; the related information of the network session packets with the same retrieval information is projected and mapped to the same storage bit.
The reading module is used for finding out the corresponding storage bit according to the retrieval information corresponding to one or more information of the network session packet to be read, and further finding out the corresponding network session packet.
Further, the retrieval information projection module divides time into a plurality of time period layers, each time period layer divides the time into a plurality of time units which are arranged in time sequence and have the same length, each time unit is divided into a plurality of time units which are arranged in time sequence and have the same length to form the next time period layer, the time units are divided in a circulating mode until the preset minimum time unit is reached, the retrieval information projection module circularly records the storage value condition of all storage bits in each index space according to the preset period, and each recording moment corresponds to the time unit corresponding to each time period layer.
Further, the information type of the network session packet includes IP address information and/or port number information and/or network protocol ID number information.
Further, the method for compressing certain information of the specified network session packet by the information compression module to obtain the retrieval information comprises the following steps:
the method comprises the following steps: and taking the parameters of the information as input parameters of a Hash function to carry out Hash operation to obtain a Hash value.
Step two: and dividing the hash value by the size of the index space corresponding to the information to carry out remainder, wherein the obtained remainder is the retrieval information.
Furthermore, the retrieval information projection module projects and maps the information subjected to Hash operation in the information compression module to the storage bit with the bit number as the rest value in the corresponding index space.
The working method of the index system of the network session packet is characterized in that in the method for compressing the information to obtain the retrieval information, if the compression operation is carried out on the IP address, each byte of the IP address is independently used as an input parameter of the Hash.
Furthermore, in the method for compressing information to obtain the retrieval information, if the port number is compressed, each byte of the port number is independently used as an input parameter of the Hash.
Furthermore, in the process of compressing the information to obtain the retrieval information, the number of bits corresponding to the size of the index space is the denominator of the remainder operation formula.
Further, the original value of each memory bit is 0, and is marked as being 1 after being projected.
Further, the length of the time unit is 24 hours at the maximum.
The invention has the beneficial effects that:
the invention can promote the processing scale of the index to infinity, write and inquire at high speed by defining a novel indexing technology, namely a projection index, and almost has the cost of 0 when two or more indexes are combined into a higher-level index. The projection index of the invention can reside in the memory because of small volume, the cost of hard disk IO is saved, and the cost of writing and reading is basically close to 0 by bit operation, but the performance can be greatly improved.
Drawings
FIG. 1 is a schematic diagram of the present invention.
FIG. 2 is an exemplary diagram of time period layers.
Detailed Description
The invention provides an index system of a network session packet, which comprises at least one index space for storing retrieval information of certain information of the network session packet, an information compression module, a retrieval information projection module and a reading module.
The index space is internally provided with a plurality of storage bits, one storage bit corresponds to one retrieval information, and all the storage bits have the same initial value.
The information compression module is used for carrying out compression operation on some or several kinds of information of the specified network session packet to obtain retrieval information corresponding to each piece of information.
The retrieval information projection module is used for projecting and mapping the related information of the network session packet to corresponding storage bits of an index space for storing the retrieval information of the information, and the projected storage bits are marked as projected when being projected for the first time; the related information of the network session packets with the same retrieval information is projected and mapped to the same storage bit.
The reading module is used for finding out the corresponding storage bit according to the retrieval information corresponding to one or more information of the network session packet to be read, and further finding out the corresponding network session packet.
Further, the retrieval information projection module divides time into a plurality of time period layers, each time period layer divides the time into a plurality of time units which are arranged in time sequence and have the same length, each time unit is divided into a plurality of time units which are arranged in time sequence and have the same length to form the next time period layer, the time units are divided in a circulating mode until the preset minimum time unit is reached, the retrieval information projection module circularly records the storage value condition of all storage bits in each index space according to the preset period, and each recording moment corresponds to the time unit corresponding to each time period layer. An example of time division is shown in fig. 2. In the figure, the time unit length of the first layer time period layer is 1 day, the 1 day is divided into time units with the length of an integer number of hours, each hour is divided into time units with the length of an integer number of minutes, and the time units with the length of each minute are divided into time units with the length of an integer number of seconds. It should be understood that the value of the specific time unit length is determined by itself according to the actual situation. The present embodiment preferably takes 24 hours as the longest time unit to enable the query to be accurate to the date.
The information type of the network session packet comprises IP address information and/or port number information and/or network protocol ID number information. Of course, the information type is not limited to this, and the information type may be expanded.
The method for compressing certain information of the appointed network session packet by the information compression module to obtain the retrieval information comprises the following steps:
the method comprises the following steps: and taking the parameters of the information as input parameters of a Hash function to carry out Hash operation to obtain a Hash value.
Step two: and dividing the hash value by the size of the index space corresponding to the information to carry out remainder, wherein the obtained remainder is the retrieval information.
For example, assume that the index space storing the IP address information has a space of N MB. Assume that the IP addresses are IP1, IP2, IP3, IP4(1 byte 8 bits). If the retrieval information is POS, the calculation steps are as follows:
HashKey＝HASH(IP1,IP2,IP3,IP4)；
POS＝HashKey％(N*1024*1024*8)；
the origins of N × 1024 × 8 are: in the process of compressing information to obtain retrieval information, the number of bits corresponding to the size of the index space is the denominator of the remainder operation formula.
It is emphasized that the retrieved information projection module projects and maps the information subjected to the Hash operation in the information compression module to the storage bit with the bit number of the corresponding index space as the rest value. The storage bits possibly calculated by the same information of different network session packets are the same, so that the information belongs to the same information and is projected and mapped to the same storage bits, and the size of the index space is greatly saved. If the index space is larger, one storage bit corresponds to less same information, the information classification is finer, and if the index space is smaller, the storage bit corresponds to more same information, and the information classification is coarser. In general, this design can accommodate any size of index space.
The working method of the present invention is explained below. The working method can be divided into a flow with time divided into a plurality of time period layers, a storage flow, a storage value condition recording flow and a reading flow.
The process of time-dividing into a plurality of time period layers specifically comprises the following steps: each time period layer divides time into a plurality of time units which are arranged in time sequence and have the same length, each time unit is divided into a plurality of time units which are arranged in time sequence and have the same length to form a next time period layer, the division is circulated until a preset minimum time unit is reached, the retrieval information projection module circularly records the storage value condition of all storage bits of each index space according to a preset period, and each recording moment corresponds to the time unit corresponding to each time period layer.
The storage process comprises the following steps:
step 1: at least one index space used for storing retrieval information of certain information of a network session packet is pre-designated, a plurality of storage bits are arranged in the index space, one storage bit corresponds to one retrieval information, and all the storage bits have the same initial value.
Step 2: extracting some or several kinds of information of the appointed network session packet, and determining the index space and the index space size corresponding to various kinds of information.
And step 3: and respectively carrying out Hash operation on each information of each network session packet and then carrying out projection mapping.
The Hash operation of each message comprises the following steps:
step 3.1: and extracting the parameters of the information.
Step 3.2: and taking the parameters as the input value of the Hash to carry out operation to obtain a Hash value.
Step 3.3: and dividing the hash value by the size of the index space corresponding to the information to carry out remainder, thereby obtaining a remainder.
Step 3.4: the retrieval information projection module maps the information to the storage bit with the bit number as the residual value in the corresponding index space, the residual value is used as the index information of the information, and the mapped storage bit is marked as projected.
The storage value condition recording flow specifically comprises the following steps: and circularly recording the storage value conditions of all storage bits in each index space according to a preset period, and corresponding each recording time to the time unit corresponding to each time period layer.
The reading process comprises the following steps:
step 1: a certain information or several information of the network session stream to be read is predefined.
Step 2: and (3) the information compression module performs compression operation on the information in the step (1) one by one to obtain corresponding retrieval information.
And step 3: and the reading module finds the corresponding storage bit according to the retrieval information, and further searches the corresponding network session packet.
The step 3 specifically comprises the following steps:
step 3.1: and taking the time period layer with the longest time unit length as a first layer time period layer, starting to read whether storage bits corresponding to all retrieval information are marked as projected or not by the system from a time unit selected in advance in the first layer time period layer, if so, locking the time unit, executing the next step, otherwise, traversing and reading the storage bits corresponding to all the retrieval information in other time units in the first layer time period layer until the storage bits can be locked and the storage bits are marked as the projected time units, and if not, ending the reading process.
Step 3.2: the system starts to search the time unit which is marked as the projected storage position corresponding to each retrieval information in each time unit of the next time period layer divided by the locked time unit in a traversing way, and the time unit is locked.
And 3.3, repeating the step 3.2 until the corresponding time unit of the time period layer with the shortest locking time unit is obtained.
Step 3.4: and reading the network session flow in the last locked time unit, and extracting the network session flow meeting the requirement.
Claims (9)
1. An indexing system for network session packets, the system comprising the following modules: the system comprises at least one index space, an information compression module, a retrieval information projection module and a reading module, wherein the index space is used for storing retrieval information of certain information of a network session packet; wherein,
the index space is internally provided with a plurality of storage bits, one storage bit corresponds to one retrieval information, and all the storage bits have the same initial value;
the information compression module is used for carrying out compression operation on some or several kinds of information of the specified network session packet to obtain retrieval information corresponding to each piece of information;
the retrieval information projection module is used for projecting and mapping the related information of the network session packet to corresponding storage bits of an index space for storing the retrieval information of the information, and the projected storage bits are marked as projected when being projected for the first time; the related information of the network session packets with the same retrieval information is projected and mapped to the same storage bit;
the reading module is used for finding out the corresponding storage bit according to the retrieval information corresponding to one or more information of the network session packet to be read, and further finding out the corresponding network session packet;
the retrieval information projection module also divides time into a plurality of time period layers, each time period layer divides the time into a plurality of time units which are arranged in time sequence and have the same length, each time unit is divided into a plurality of time units which are arranged in time sequence and have the same length to form the next time period layer, the time units are divided in a circulating mode until the preset minimum time unit is reached, the retrieval information projection module circularly records the storage value condition of all storage bits in each index space according to the preset period, and each recording moment corresponds to the time unit corresponding to each time period layer.
2. The system for indexing network session packets as claimed in claim 1, wherein the information type of the network session packet includes IP address information and/or port number information and/or network protocol ID number information.
3. The system for indexing network session packets according to claim 1, wherein the method for compressing the information of the designated network session packet by the information compression module to obtain the retrieval information comprises:
the method comprises the following steps: taking the parameters of the information as input parameters of a Hash function to carry out Hash operation to obtain a Hash value;
step two: and dividing the hash value by the size of the index space corresponding to the information to carry out remainder, wherein the obtained remainder is the retrieval information.
4. The system for indexing network session packets as claimed in claim 3, wherein the retrieved information projection module projects and maps the information subjected to the Hash operation in the information compression module to the storage bits with the bit number as the rest value in the corresponding index space.
5. The system for indexing network session packets as claimed in claim 3, wherein in the method for obtaining the search information by compressing the information, if the compression operation is performed on the IP address, each byte of the IP address is independently used as the input parameter of the Hash.
6. The system for indexing network session packets as claimed in claim 3, wherein in the method for compressing information to obtain the retrieved information, if the port number is compressed, each byte is used as the input parameter of the Hash independently.
7. The system for indexing network session packets as claimed in claim 3, wherein the number of bits corresponding to the size of the index space is the denominator of the remainder operation formula in the process of compressing the information to obtain the retrieval information.
8. The system for indexing network session packets of claim 1, wherein the original value of each stored bit is 0, marked as projected to become 1.
9. The system for indexing network session packets of claim 1, wherein the time unit length is up to 24 hours.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610773444.8A CN106446028B (en) | 2016-08-31 | 2016-08-31 | Indexing system of network session packet |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610773444.8A CN106446028B (en) | 2016-08-31 | 2016-08-31 | Indexing system of network session packet |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106446028A CN106446028A (en) | 2017-02-22 |
| CN106446028B true CN106446028B (en) | 2019-12-27 |
Family
ID=58091506
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610773444.8A Active CN106446028B (en) | 2016-08-31 | 2016-08-31 | Indexing system of network session packet |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106446028B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112650756B (en) * | 2020-12-29 | 2023-05-02 | 科来网络技术股份有限公司 | Time projection indexing method and system based on time sequence data |
| CN113596098B (en) * | 2021-07-01 | 2023-04-25 | 杭州迪普科技股份有限公司 | Session retrieval method, apparatus, device and computer readable storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101501626A (en) * | 2005-03-10 | 2009-08-05 | 全球定位有限公司 | Method and apparatus for managing network elements in a satellite navigation data distribution system |
| CN104618361A (en) * | 2015-01-22 | 2015-05-13 | 中国科学院计算技术研究所 | Network stream data reordering method |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101546342B (en) * | 2009-05-08 | 2012-07-04 | 阿里巴巴集团控股有限公司 | Method and system for implementing search service |
| CN103442293B (en) * | 2013-08-30 | 2016-10-05 | 武汉眸博科技有限公司 | Network multimedia file quickly starts player method and system |
| CN103488709B (en) * | 2013-09-09 | 2017-06-16 | 东软集团股份有限公司 | A kind of index establishing method and system, search method and system |
| KR20150071500A (en) * | 2013-12-18 | 2015-06-26 | 삼성전자주식회사 | Method and Apparatus for Managing Data |
-
2016
- 2016-08-31 CN CN201610773444.8A patent/CN106446028B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101501626A (en) * | 2005-03-10 | 2009-08-05 | 全球定位有限公司 | Method and apparatus for managing network elements in a satellite navigation data distribution system |
| CN104618361A (en) * | 2015-01-22 | 2015-05-13 | 中国科学院计算技术研究所 | Network stream data reordering method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106446028A (en) | 2017-02-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104346357B (en) | The file access method and system of a kind of built-in terminal | |
| JP5996088B2 (en) | Cryptographic hash database | |
| WO2018148151A1 (en) | Merge tree garbage metrics | |
| CN102622434B (en) | Data storage method, data searching method and device | |
| WO2013152678A1 (en) | Method and device for metadata query | |
| CN107368527B (en) | Multi-attribute index method based on data stream | |
| JP2013037517A (en) | Memory system having key-value store method | |
| US20110040761A1 (en) | Estimation of postings list length in a search system using an approximation table | |
| CN108399175B (en) | Data storage and query method and device | |
| CN103049355B (en) | Method and equipment for database system recovery | |
| US20110216774A1 (en) | Packet file system | |
| CN102244758A (en) | Video-recording-file-based data acquisition method and equipment | |
| WO2012174906A1 (en) | Data storage and search method and apparatus | |
| CN112262379B (en) | Storing data items and identifying stored data items | |
| US20180364938A1 (en) | Extent-based data location table management | |
| CN106446028B (en) | Indexing system of network session packet | |
| CN116450656A (en) | Data processing method, device, equipment and storage medium | |
| JP2020529166A (en) | Packet processing | |
| CN114721594A (en) | Distributed storage method, device, equipment and machine readable storage medium | |
| CN107391769B (en) | Index query method and device | |
| EP3093772B1 (en) | Data processing method, data writing method, and related apparatus | |
| CN106326464B (en) | A kind of network session packet indexing means based on retrieval information projection | |
| CN107423425A (en) | A kind of data quick storage and querying method to K/V forms | |
| CN109522242A (en) | A kind of method and apparatus for searching for Cache data | |
| CN107609143B (en) | Fragment information storage method of distributed real-time memory database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200804 Address after: 41401-41406, unit 1, building 4, No. 966, north section of Tianfu Avenue, Chengdu hi tech Zone, Sichuan 610041 Patentee after: Chengdu Kelai Network Technology Co., Ltd Address before: China high tech Zone of Chengdu City, Sichuan province 610041 Road No. 99 Tianfu Software Park B6-7 Patentee before: COLASOFT Co.,Ltd. |
|
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 610041 12th, 13th and 14th floors, unit 1, building 4, No. 966, north section of Tianfu Avenue, high tech Zone, Chengdu, Sichuan Patentee after: Kelai Network Technology Co.,Ltd. Address before: 41401-41406, 14th floor, unit 1, building 4, No. 966, north section of Tianfu Avenue, Chengdu hi tech Zone, China (Sichuan) pilot Free Trade Zone, Chengdu hi tech Zone, Sichuan 610041 Patentee before: Chengdu Kelai Network Technology Co.,Ltd. |