CN111694992A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN111694992A CN111694992A CN201910198443.9A CN201910198443A CN111694992A CN 111694992 A CN111694992 A CN 111694992A CN 201910198443 A CN201910198443 A CN 201910198443A CN 111694992 A CN111694992 A CN 111694992A
- Authority
- CN
- China
- Prior art keywords
- data
- pointer
- target
- data processing
- target data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9027—Trees
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure relates to a data processing method and apparatus. The method is applied to a data storage system based on an LSM-Tree structure, and comprises the following steps: receiving a data processing request containing a target keyword, wherein the data processing request is used for requesting to execute data processing operation on target data corresponding to the target keyword; and executing data processing operation on the target data through a handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data. The access speed of the LSM-Tree structure can be increased, and the data processing efficiency can be effectively improved.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
Background
A Log Structured Merge Tree (LSM-Tree) is a structure widely used in the field of data storage. At least an upper storage space (e.g., MemTable) having a read-write characteristic and a lower storage space (e.g., BaseTable) having a read-only characteristic are included in the LSM structure. When the amount of data in the MemTable exceeds a threshold, the LSM-Tree structure needs to merge MemTable and BaseTable to generate a new BaseTable, and to generate a new empty MemTable. The merge between MemTable and BaseTable may cause a change in the memory resources used to store data corresponding to the key. At present, when an LSM-Tree structure is accessed, resource positioning is carried out in the whole data storage system through keywords in each access, memory resources used for storing data corresponding to the keywords are determined, then data processing is carried out on the data in the memory resources, and the data processing efficiency is low.
Disclosure of Invention
In view of this, the present disclosure provides a data processing method and apparatus, so that the access speed of the LSM-Tree structure can be increased, and further the data processing efficiency can be effectively increased.
According to a first aspect of the present disclosure, there is provided a data processing method applied to a data storage system based on an LSM-Tree structure, the method including: receiving a data processing request containing a target keyword, wherein the data processing request is used for requesting to execute data processing operation on target data corresponding to the target keyword; and executing data processing operation on the target data through a handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data.
In one possible implementation, the performing, by a handle corresponding to the target keyword, a data processing operation on the target data includes: judging whether the pointer is valid; when the pointer is determined to be valid, performing a data processing operation on the target data through the pointer.
In one possible implementation, the method further includes: when the pointer is determined to be invalid, carrying out resource positioning in the data storage system according to the target keyword, and searching the target data; reassigning the pointer according to the search result; and executing data processing operation on the target data through the assigned finger.
In one possible implementation, the LSM-Tree structure includes at least a first hierarchy and a second hierarchy, data in the first hierarchy having a read-write characteristic, data in the second hierarchy having a read-only characteristic; the method further comprises the following steps: when the data amount of the data table in the first hierarchy is larger than a threshold value, determining the hierarchy of the data table as a third hierarchy, and generating a new empty data table and setting the hierarchy of the empty data table as the first hierarchy, wherein the data in the third hierarchy has a read-only characteristic; merging data in the third hierarchy to the second hierarchy.
In one possible implementation, the pointers include a read pointer and a write pointer; the method further comprises the following steps: setting the write pointer to a null pointer when a data table in the first level is null and data in the third level is not merged to the second level; when the data in the third level is merged to the second level and the write pointer is a null pointer, setting the read pointer to be a null pointer.
In a possible implementation manner, the determining whether the pointer is valid includes at least one of the following steps: when the read pointer is a non-null pointer, determining that the read pointer is valid; and when the write pointer is a non-null pointer, determining that the write pointer is valid.
In a possible implementation manner, the handle further includes a first identifier, the LSM-Tree structure includes a second identifier, and the first identifier and the second identifier are the same before the data in the third hierarchy is merged into the second hierarchy; the method further comprises the following steps: and after the data in the third hierarchy is merged into the second hierarchy, updating the second identifier, wherein the updated second identifier is different from the first identifier.
In one possible implementation, the pointers include a read pointer and a write pointer; judging whether the pointer is valid, including: and when the first identifier and the second identifier are the same, determining that the read pointer and the write pointer are valid.
In one possible implementation, the method further includes: and after resource positioning is carried out in the data storage system according to the target keyword, updating the first identifier, wherein the updated first identifier is the same as the second identifier.
In one possible implementation, when it is determined that the pointer is valid, performing a data processing operation on the target data by the pointer includes at least one of: reading the target data pointed by the read pointer; and modifying the target data pointed by the write pointer.
In a possible implementation manner, reassigning the pointer according to the search result includes: and if the target data is found, assigning the memory address of the target data to the read pointer.
In one possible implementation, the performing, by the reassigned pointer, a data processing operation on the target data includes: and reading the target data pointed by the read pointer after reassignment.
In a possible implementation manner, reassigning the pointer according to the search result includes: if the target data is found, copying the target data to the first level; and assigning the memory address of the target data in the first level to the write pointer.
In one possible implementation, the performing, by the reassigned pointer, a data processing operation on the target data includes: and modifying the target data pointed by the write pointer after reassignment.
In a possible implementation manner, reassigning the pointer according to the search result includes: if the target data is not found, creating an empty memory resource in the first level; and assigning the memory address of the empty memory resource to the write pointer.
In one possible implementation, the performing, by the reassigned pointer, a data processing operation on the target data includes: and writing the target data into the empty memory resource pointed by the write pointer after reassignment.
According to a second aspect of the present disclosure, there is provided a data processing apparatus applied to a data storage system based on an LSM-Tree structure, the apparatus comprising: a receiving module, configured to receive a data processing request including a target keyword, where the data processing request is used to request to perform a data processing operation on target data corresponding to the target keyword; and the data processing module is used for executing data processing operation on the target data through a handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data.
According to a third aspect of the present disclosure, there is provided a data processing apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the data processing method of the first aspect.
According to a fourth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer program instructions, wherein the computer program instructions, when executed by a processor, implement the data processing method of the first aspect described above.
In a data storage system based on an LSM-Tree structure, receiving a data processing request containing a target keyword, wherein the data processing request is used for requesting to execute data processing operation on target data corresponding to the target keyword; and executing data processing operation on the target data through a handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data. And setting a handle corresponding to the keyword in the LSM-Tree structure, and directly accessing the data corresponding to the keyword in the LSM-Tree structure through the handle corresponding to the keyword to execute data processing operation, so that the access speed of the LSM-Tree structure can be increased, and the data processing efficiency can be effectively improved.
Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.
FIG. 1 illustrates a schematic diagram of a data storage system based on an LSM-Tree architecture;
FIG. 2 shows a flow diagram of a data processing method of an embodiment of the present disclosure;
FIG. 3 shows a schematic diagram of an LSM-Tree structure of an embodiment of the present disclosure;
FIG. 4 shows a schematic diagram of data merging in an LSM-Tree structure according to an embodiment of the present disclosure;
FIG. 5 is a diagram illustrating updating a pointer in a handle according to one embodiment of the present disclosure;
fig. 6 shows a schematic diagram of updating a second sequence number in an LSM-Tree structure according to an embodiment of the disclosure;
FIG. 7 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;
fig. 8 shows a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.
Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.
In the data storage system based on the LSM-Tree structure, the storage format of the data is a key-value pair (key-value) structure, the value is a data value of the data, and the key value (key) is an index parameter of the data. FIG. 1 illustrates a schematic diagram of a data storage system based on an LSM-Tree structure. In the data storage system shown in fig. 1, the LSM-Tree structure includes a data table memtable (memory table) for storing data in a first hierarchy (high hierarchy), and a data table BaseTable for storing data in a second hierarchy (low hierarchy). Wherein, MemTabel has the characteristics of reading and writing, and BaseTable has the characteristics of reading and writing. When key-value data is written, MemTabel in the first level is written first. When the data volume of MemTable in the first level exceeds a threshold value, combining the MemTable with BaseTable in the second level to obtain a new BaseTable in the second level, and generating a new empty MemTale in the first level to continue data writing.
The data written into the LSM-Tree structure is stored hierarchically, the data with the same key can be stored in data tables of different hierarchies, and the data stored in the data table of a higher hierarchy is new data relative to the data stored in the data table of a lower hierarchy. That is, if there is data in MemTable and BaseTable that is identical to a key, the data in MemTable corresponding to the key is new data with respect to the data in BaseTable corresponding to the key, and when MemTable and BaseTable are merged, the data in MemTable corresponding to the key overwrites the data in BaseTable corresponding to the key.
As stated in the background section, since the merging of MemTable and BaseTable may cause a change in the memory address for storing data corresponding to a keyword, when accessing the LSM-Tree structure, it is necessary to perform resource location according to the keyword each time, determine the memory address for storing the data corresponding to the keyword, and then release the memory address after performing data processing operations such as read operation or write operation according to the memory address. The access mode of resource positioning according to the keywords each time is limited by the scale of the data storage system, and when the scale of the data storage system is large, the access speed of the LSM-Tree structure is low, so that the data processing efficiency is reduced.
The data processing method provided by the disclosure can be applied to the data storage system based on the LSM-Tree structure shown in FIG. 1, so as to improve the access speed of the LSM-Tree structure and further effectively improve the data processing efficiency. The data processing method provided by the present disclosure is described in detail below.
Fig. 2 shows a schematic flow chart of a data processing method according to an embodiment of the present disclosure. The method shown in fig. 2 may be applied to a data storage system based on an LSM-Tree structure, and the method may include:
step 21, receiving a data processing request including a target keyword, where the data processing request is used to request to perform a data processing operation on target data corresponding to the target keyword.
And step 22, executing data processing operation on the target data through the handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data.
And creating a global handle linked list in the LSM-Tree structure, wherein all handles created in the LSM-Tree structure are stored in the global handle linked list. Each handle comprises a key word and a pointer for pointing to data corresponding to the key word, so that data processing operation can be directly executed on the data corresponding to the key word through the handle corresponding to the key word without resource positioning.
In one possible implementation, the pointers in the handle include a read pointer and a write pointer.
Fig. 3 shows a schematic diagram of an LSM-Tree structure of an embodiment of the present disclosure. As shown in fig. 3, the LSM-Tree structure includes a first level (high level) of memtab, a second level (low level) of BaseTable, and a global handle linked list. The global handle linked list comprises a handle A corresponding to the first key and a handle B corresponding to the second key. Memabael includes: data a corresponding to the first keyword and data b corresponding to the second keyword. The BaseTable includes: data c corresponding to the first keyword and data d corresponding to the third keyword. The handle A includes: the data processing method comprises a first key, a read pointer and a write pointer, wherein the read pointer and the write pointer in a handle A both point to data a corresponding to the first key in MemTabel, namely the data a can be directly accessed through the read pointer and the write pointer in the handle A, and then data processing operation is performed on the data a. The handle B comprises: the data processing method comprises a second key, a read pointer and a write pointer, wherein the read pointer and the write pointer in the handle B both point to data B corresponding to the second key in MemTabel, namely the data B can be directly accessed through the read pointer and the write pointer in the handle B, and then data processing operation is performed on the data B.
In a possible implementation manner, for hot data with a higher access rate in the LSM-Tree structure, a handle corresponding to the hot data, that is, a handle corresponding to a keyword of the hot data, may be created to improve the access speed of the hot data.
In addition to creating handles for hot data in the LSM-Tree structure, handles may be created for other data in the LSM-Tree structure according to actual needs, which is not specifically limited by the present disclosure.
In one possible implementation, the LSM-Tree structure includes at least a first hierarchy and a second hierarchy, data in the first hierarchy having a read-write characteristic, and data in the second hierarchy having a read-only characteristic; when the data amount of the data table in the first hierarchy exceeds a threshold value, determining the hierarchy of the data table as a third hierarchy, and generating a new empty data table and setting the hierarchy of the empty data table as the first hierarchy, wherein the data in the third hierarchy has a read-only characteristic; merging the data in the third hierarchy to the second hierarchy.
The threshold value may be set according to practical situations, and the present disclosure is not particularly limited thereto.
Also taking the above-mentioned fig. 3 as an example, if the amount of data in memtab of fig. 3 exceeds a threshold, the data merging shown in fig. 4 needs to be performed. Fig. 4 shows a schematic diagram of data merging in an LSM-Tree structure according to an embodiment of the present disclosure. As shown in fig. 4, the level of MemTable with the data amount exceeding the threshold is set to the third level from the first level, that is, the MemTable with the data amount exceeding the threshold is changed to the data table immutable (immutable table) in the third level, and a new empty MemTable is generated in the first level for the subsequent data writing, wherein the third level is located between the first level and the second level, and the data in the third level has a read-only characteristic.
And carrying out data combination on the ImmuTable and the BaseTable to obtain a new BaseTable in the second level. Because the data a corresponding to the first keyword in the ImmuTable is new data relative to the data c corresponding to the first keyword in the BaseTable, in the data merging process, the data a corresponding to the first keyword in the ImmuTable covers the data c corresponding to the first keyword in the BaseTable. In the new BaseTable obtained, include: data a corresponding to the first keyword, data b corresponding to the second keyword, and data d corresponding to the third keyword.
In one possible implementation, the performing a data processing operation on the target data through a handle corresponding to the target keyword includes: judging whether the pointer is valid; when the pointer is determined to be valid, a data processing operation is performed on the target data by the pointer.
In one possible implementation manner, the method further includes: when the pointer is determined to be invalid, carrying out resource positioning in the data storage system according to the target key words, and searching target data; reassigning the pointer according to the search result; and executing data processing operation on the target data through the reassigned pointer.
When the pointer in the handle corresponding to the target keyword is valid, the LSM-Tree structure indicates that data merging does not occur, namely, the target data can be directly accessed and data processing operation can be executed through the pointer; when the pointer in the handle corresponding to the target keyword is invalid, it indicates that data merging has occurred in the LSM-Tree structure, the memory address of the data corresponding to the target keyword has changed, and the pointer cannot accurately point to the target data.
The manner of determining whether the pointer is valid includes, but is not limited to, the following two manners.
The first method comprises the following steps:
in one possible implementation, when the data table in the first level is empty and the data in the third level is not merged to the second level, the write pointer is set to be a null pointer; and when the data in the third level is merged to the second level and the write pointer is a null pointer, setting the read pointer to be a null pointer.
In one possible implementation, the determining whether the pointer is valid includes at least one of the following steps: when the read pointer is a non-null pointer, determining that the read pointer is valid; and when the write pointer is a non-null pointer, determining that the write pointer is valid.
And in the process of executing data combination of ImmuTable and BaseTable in the LSM-Tree structure, synchronously updating the pointer in each handle in the global handle linked list.
After the MemTabel with the data volume exceeding the threshold is changed into the ImmuTable and a new empty MemTable is generated and before the ImmuTable and the BaseTable are subjected to data combination, because the ImuTable has read-only performance, at the moment, the global handle linked list is traversed, the write pointer in each handle in the global handle linked list is set as an empty pointer, and the read pointer is unchanged. That is, for any handle, data access and data processing operations can only be directly performed through the read pointer in the handle, and data access and data processing operations cannot be directly performed through the write pointer in the handle.
After data merging is performed on the ImmuTable and the BaseTable, the memory address of the data changes, at this time, the global handle linked list is traversed, and for any handle in the global handle linked list: if the write pointer in the handle is still a null pointer, the LSM-Tree structure indicates that resource location is not performed through the keyword in the handle to perform reassignment on the write pointer, and at this time, the read pointer in the handle is set as the null pointer, that is, data access and data processing operations cannot be directly performed through the read pointer and the write pointer in the handle; if the write pointer in the handle is a non-null pointer, it indicates that resource location has been performed on the write pointer through the key in the handle in the LSM-Tree structure, and the write pointer is reassigned, at this time, the read pointer and the write pointer in the handle do not need to be updated, that is, data access and data processing operations can be directly performed through the read pointer and the write pointer in the handle.
Still taking the above fig. 4 as an example, in the process of performing the data merging shown in fig. 4, fig. 5 is a schematic diagram illustrating a pointer in an update handle according to an embodiment of the present disclosure. As shown in fig. 5, before the MemTable whose data amount exceeds the threshold is changed to ImmuTable and a new empty MemTable is generated and the ImmuTable and the BaseTable are subjected to data merging, the global handle linked list is traversed, the write pointers in the handle a and the handle B are set as empty pointers, and the read pointer is unchanged.
The write pointer in the handle a is set as a null pointer, which indicates that the write pointer is invalid, that is, the data a in the ImmuTable cannot be accessed and the data processing operation cannot be performed through the write pointer in the handle a, and at this time, the data a in the ImmuTable can still be accessed and the data processing operation cannot be performed through the read pointer in the handle a. The write pointer in the handle B is set to be a null pointer, which indicates that the write pointer is invalid, that is, the data B in the ImmuTable cannot be accessed and the data processing operation cannot be performed through the write pointer in the handle B, and at this time, the data B in the ImmuTable can still be accessed and the data processing operation cannot be performed through the read pointer in the handle B.
After data combination is carried out on the ImmuTable and the BaseTable, the global handle linked list is traversed, and the read pointers in the handle A and the handle B are set as null pointers.
The read pointer and the write pointer in the handle a are both set as null pointers, that is, the read pointer and the write pointer in the handle a are both invalid, and at this time, the read pointer and the write pointer in the handle a cannot directly access the data a in the BaseTable and perform data processing operations. The read pointer and the write pointer in the handle B are both set to null pointers, that is, the read pointer and the write pointer in the handle B are both invalid, and at this time, the read pointer and the write pointer in the handle B cannot directly access the data B in the BaseTable and perform data processing operations.
In the process of executing data merging of ImmuTable and BaseTable in the LSM-Tree structure, the pointer in each handle in the global handle linked list is synchronously updated, so that after a data processing request containing a target keyword is received, whether the pointer in the handle corresponding to the target keyword is a non-empty pointer can be determined, whether the pointer in the handle corresponding to the target keyword is effective can be determined, and when the pointer in the handle corresponding to the target keyword is determined to be effective, the target data is directly accessed and data processing operation is executed through the pointer in the handle corresponding to the target keyword, and target data does not need to be searched in the LSM-Tree structure through resource positioning, so that the access speed of the LSM-Tree structure is increased, and the data processing efficiency is improved.
And the second method comprises the following steps:
in a possible implementation manner, the handle further includes a first identifier, the LSM-Tree structure includes a second identifier, and the first identifier and the second identifier are the same before the data in the third hierarchy is merged into the second hierarchy; and after the data in the third level is merged into the second level, updating the second identifier, wherein the updated second identifier is different from the first identifier.
In a possible implementation manner, after the resource location is performed in the data storage system according to the target keyword, the first identifier is updated, and the updated first identifier is the same as the second identifier.
In one possible implementation, the pointers include a read pointer and a write pointer; judging whether the pointer is valid, including: and when the first identification and the second identification are the same, determining that the read pointer and the write pointer are valid.
In one possible implementation, the first identifier is a first sequence number and the second identifier is a second sequence number.
And in the process of executing data combination of ImmuTable and BaseTable in the LSM-Tree structure, synchronously updating the second sequence number in the LSM-Tree structure.
When the data volume in MemTabel is smaller than the threshold value, or after MemTabel with the data volume exceeding the threshold value is changed into ImmuTable and new empty MemTable is generated and before data merging of ImmuTable and BaseTable is carried out, the first sequence number in the handle is the same as the second sequence number in the LSM-Tree structure.
After the ImmuTable and the BaseTable are merged, the memory address of the data is changed, and the second sequence number is incremented, for example, the second sequence number is incremented by 1. At this time, the first sequence number is different from the second sequence number. After resource location is carried out in the data storage system according to the key in the handle and the pointer in the handle is reassigned, the first sequence number in the handle is updated, so that the updated first sequence number in the handle is the same as the second sequence number in the LSM-Tree, namely the pointer in the handle becomes valid.
Still taking the above fig. 4 as an example, in the process of performing the data merging shown in fig. 4, fig. 6 shows a schematic diagram of updating the second sequence number in the LSM-Tree according to an embodiment of the present disclosure. As shown in fig. 6, before MemTable whose data amount exceeds the threshold is changed to ImmuTable and new empty MemTable is generated and data merging is performed between ImmuTable and BaseTable, the first sequence number in handle a, the first sequence number in handle B and the second sequence number in LSM-Tree are all 003.
After the ImmuTable is data merged with the BaseTable, the second sequence number is incremented, becoming 004.
The first sequence number 003 in the handle a is different from the second sequence number 004 in the LSM-Tree, that is, the read pointer and the write pointer in the handle a are both invalid, and at this time, both the read pointer and the write pointer in the handle a cannot directly access the data a in the BaseTable and perform data processing operations. The first sequence number 003 in the handle B is different from the second sequence number 004 in the LSM-Tree, that is, the read pointer and the write pointer in the handle B are both invalid, and at this time, both the read pointer and the write pointer in the handle B cannot directly access the data B in the BaseTable and perform data processing operations.
The first identifier and the second identifier may be serial numbers, but may be other identifiers, and the disclosure is not limited thereto.
Because in the process of executing data merging of ImmuTable and BaseTable in the LSM-Tree structure, the second identifier in the LSM-Tree structure is updated synchronously so that after receiving a data processing request containing a target key, it may be determined whether the pointer within the handle corresponding to the target key is valid by determining whether the first identifier within the handle corresponding to the target key is the same as the second identifier in the LSM-Tree structure, when the pointer in the handle corresponding to the target keyword is determined to be valid, the target data is directly accessed and the data processing operation is executed through the pointer in the handle corresponding to the target keyword without searching the target data corresponding to the target keyword in the LSM-Tree structure through resource positioning, therefore, the access speed of the LSM-Tree structure is improved, and the data processing efficiency is further improved.
In one possible implementation, when the pointer is determined to be valid, performing a data processing operation on the target data by the pointer, including at least one of the following steps: reading target data pointed by the reading pointer; the target data pointed to by the write pointer is modified.
When a read pointer in a handle corresponding to the target keyword is valid, target data can be directly read through the read pointer; when the write pointer in the handle corresponding to the target key is valid, the target data can be directly modified through the write pointer.
When the pointer in the handle corresponding to the target keyword is determined to be invalid, resource positioning is carried out in the data storage system according to the target keyword, target data are searched, the pointer in the handle corresponding to the target keyword is reassigned according to the search result, and then data processing operation is carried out on the target data through the reassigned pointer.
Carrying out resource positioning in the data storage system according to the target key words, wherein the process of searching the target data comprises the following steps: step 1, searching target data in MemTable according to the target keyword, if the target data are searched, determining the memory address of the target data, and if the target data are not searched, continuing to execute step 2; step 2, if ImmuTable exists in the LSM-Tree structure, executing step 3, otherwise executing step 4; step 3, searching target data in ImmuTable according to the target keyword, if the target data is searched, determining the memory address of the target data, and if the target data is not searched, continuing to execute the step 4; and 4, searching target data in the BaseTable according to the target keyword, if the target data is searched, determining the memory address of the target data, and if the target data is not searched, determining that the target data is not stored in the LSM-Tree structure.
In one possible implementation, reassigning the pointer according to the search result includes: and if the target data is found, assigning the memory address of the target data to the read pointer.
In one possible implementation, the data processing operation is performed on the target data by the reassigned pointer, including: and reading the target data pointed by the reassigned read pointer.
After resource positioning is carried out in the data storage system according to the target keyword to find the target data, the memory address of the target data is assigned to the read pointer in the handle corresponding to the target keyword, so that the target data can be directly read through the read pointer in the handle corresponding to the target keyword before the next data combination of ImmuTable and BaseTable is carried out in the LSM-Tree structure, resource positioning is not needed again, and the access speed and the processing efficiency of the target data are effectively improved.
In one possible implementation, reassigning the pointer according to the search result includes: if the target data is found, copying the target data to a first level; and assigning the memory address of the target data in the first level to a write pointer.
In one possible implementation, the data processing operation is performed on the target data by the reassigned pointer, including: and modifying the target data pointed by the write pointer after reassignment.
Because only the MemTable has the writable characteristic, the target data can be copied into the MemTable after the target data is found by performing resource positioning in the data storage system according to the target keyword, and then the memory address of the target data in the MemTable is assigned to the write pointer in the handle corresponding to the target keyword, so that the target data can be directly modified through the write pointer in the handle corresponding to the target keyword before the data combination of the ImmuTable and the BaseTable is executed next time in the LSM-Tree structure, resource positioning is not required to be performed again, and the access speed and the processing efficiency of the target data are effectively improved.
In one possible implementation, reassigning the pointer according to the search result includes: if the target data is not found, creating an empty memory resource in the first level; and assigning the memory address of the empty memory resource to the read pointer and the write pointer.
After resource positioning is carried out in the data storage system according to the target keyword, if target data is not found, the data storage system does not store the target data, at this time, an empty memory resource is created in MemTable, and then the memory address of the empty memory resource is assigned to the read pointer and the write pointer in the handle corresponding to the target keyword, so that the target data which is not stored in the data storage system can be directly determined according to the read pointer in the handle corresponding to the target keyword subsequently without carrying out resource positioning again, or the target data can be directly written into the memory resource according to the write pointer in the handle corresponding to the target keyword.
In one possible implementation, the data processing operation is performed on the target data by the reassigned pointer, including: and writing the target data into the empty memory resource pointed by the write pointer after reassignment.
In a data storage system based on an LSM-Tree structure, receiving a data processing request containing a target keyword, wherein the data processing request is used for requesting to execute data processing operation on target data corresponding to the target keyword; and executing data processing operation on the target data through a handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data. And setting a handle corresponding to the keyword in the LSM-Tree structure, and directly accessing the data corresponding to the keyword in the LSM-Tree structure through the handle corresponding to the keyword to execute data processing operation, so that the access speed of the LSM-Tree structure can be increased, and the data processing efficiency can be effectively improved.
Fig. 7 shows a schematic structural diagram of a data processing apparatus according to an embodiment of the present disclosure. The apparatus 70 shown in fig. 7 may be applied to a data storage system based on an LSM-Tree structure, and may be used to perform the steps of the above-described embodiment of the method shown in fig. 2, where the apparatus 70 includes:
a receiving module 71, configured to receive a data processing request including a target keyword, where the data processing request is used to request to perform a data processing operation on target data corresponding to the target keyword;
and a data processing module 72, configured to perform a data processing operation on the target data through a handle corresponding to the target keyword, where the handle includes a pointer for pointing to the target data.
In one possible implementation, the data processing module 72 includes:
the judgment submodule is used for judging whether the pointer is valid;
and the data processing submodule is used for executing data processing operation on the target data through the pointer when the pointer is determined to be valid.
In one possible implementation, the data processing module 72 further includes:
the searching submodule is used for carrying out resource positioning in the data storage system according to the target key word and searching target data when the pointer is determined to be invalid;
the pointer assignment submodule is used for reassigning the pointer according to the search result;
and the data processing submodule is also used for executing data processing operation on the target data through the reassigned pointer.
In one possible implementation, the LSM-Tree structure includes at least a first hierarchy and a second hierarchy, data in the first hierarchy having a read-write characteristic, and data in the second hierarchy having a read-only characteristic;
the apparatus 70 further comprises:
the generating module is used for determining the hierarchy of the data table as a third hierarchy when the data quantity of the data table in the first hierarchy exceeds a threshold value, generating a new empty data table and setting the hierarchy of the empty data table as the first hierarchy, wherein data in the third hierarchy has a read-only characteristic;
and the merging module is used for merging the data in the third level into the second level.
In one possible implementation, the pointers include a read pointer and a write pointer;
the apparatus 70 further comprises:
the pointer updating module is used for setting the write pointer as a null pointer when the first-level total data table is null and the data in the third level is not merged into the second level;
and the pointer updating module is also used for setting the read pointer as a null pointer when the data in the third level is merged to the second level and the write pointer is a null pointer.
In one possible implementation, the determining sub-module includes:
the reading pointer judging unit is used for determining that the reading pointer is valid when the reading pointer is a non-null pointer;
and the write pointer judging unit is used for determining that the write pointer is valid when the write pointer is a non-null pointer.
In a possible implementation manner, the handle further includes a first identifier, the LSM-Tree structure includes a second identifier, and the first identifier and the second identifier are the same before the data in the third hierarchy is merged into the second hierarchy;
the apparatus 70 further comprises:
and the identifier updating module is used for updating the second identifier after the data in the third hierarchy is merged into the second hierarchy, wherein the updated second identifier is different from the first identifier.
In one possible implementation, the pointers include a read pointer and a write pointer;
the judgment module is specifically configured to:
and when the first identification and the second identification are the same, determining that the read pointer and the write pointer are valid.
In a possible implementation manner, the identifier updating unit is further configured to update the first identifier after performing resource location in the data storage system according to the target keyword, where the updated first identifier is the same as the second identifier.
In one possible implementation, the data processing sub-module includes:
a reading unit, configured to read target data pointed by the read pointer;
and the modification unit is used for modifying the target data pointed by the write pointer.
In a possible implementation manner, the pointer assignment module submodule is specifically configured to:
and if the target data is found, assigning the memory address of the target data to the read pointer.
In one possible implementation, the reading unit is further configured to:
and reading the target data pointed by the reassigned read pointer.
In a possible implementation manner, the pointer assignment module submodule is specifically configured to:
if the target data is found, copying the target data to a first level;
and assigning the memory address of the target data in the first hierarchy to the write pointer.
In one possible implementation, the modification unit is further configured to:
and modifying the target data pointed by the write pointer after reassignment.
In a possible implementation manner, the pointer assignment module submodule is specifically configured to:
if the target data is not found, creating an empty memory resource in the first level;
and assigning the memory address of the empty memory resource to the write pointer.
In one possible implementation, the data processing sub-module further includes:
and the write-in unit is used for writing target data into the empty memory resource pointed by the write pointer after reassignment.
The apparatus 70 provided in the present disclosure can implement each step in the method embodiment shown in fig. 2, and implement the same technical effect, and is not described herein again to avoid repetition.
Fig. 8 shows a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. As shown in fig. 8, at the hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory, such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, the network interface, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (peripheral component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 8, but that does not indicate only one bus or one type of bus.
And a memory for storing the program. In particular, the program may include program code comprising computer operating instructions. The memory may include both memory and non-volatile storage and provides instructions and data to the processor.
The processor reads a corresponding computer program from the non-volatile memory into the memory and then runs the computer program, thereby forming the data processing device on a logic level. The processor executes the program stored in the memory and specifically executes: receiving a data processing request containing a target keyword, wherein the data processing request is used for requesting to execute data processing operation on target data corresponding to the target keyword; and executing data processing operation on the target data through a handle corresponding to the target key, wherein the handle comprises a pointer for pointing to the target data.
In one possible implementation, the processor is specifically configured to perform: judging whether the pointer is valid; when the pointer is determined to be valid, a data processing operation is performed on the target data by the pointer.
In one possible implementation, the processor is specifically configured to perform: when the pointer is determined to be invalid, carrying out resource positioning in the data storage system according to the target key words, and searching target data; reassigning the pointer according to the search result; and executing data processing operation on the target data through the reassigned pointer.
In one possible implementation, the LSM-Tree structure includes at least a first hierarchy and a second hierarchy, data in the first hierarchy having a read-write characteristic, and data in the second hierarchy having a read-only characteristic; the processor is specifically configured to perform: when the data amount of the data table in the first hierarchy exceeds a threshold value, determining the hierarchy of the data table as a third hierarchy, and generating a new empty data table and setting the hierarchy of the empty data table as the first hierarchy, wherein the data in the third hierarchy has a read-only characteristic; merging the data in the third hierarchy to the second hierarchy.
In one possible implementation, the pointers include a read pointer and a write pointer; the processor is specifically configured to perform: when the data table in the first level is empty and the data in the third level is not merged to the second level, setting the write pointer as a null pointer; and when the data in the third level is merged to the second level and the write pointer is a null pointer, setting the read pointer to be a null pointer.
In one possible implementation, the processor is specifically configured to perform: when the read pointer is a non-null pointer, determining that the read pointer is valid; and when the write pointer is a non-null pointer, determining that the write pointer is valid.
In a possible implementation manner, the handle further includes a first identifier, the LSM-Tree structure includes a second identifier, and the first identifier and the second identifier are the same before the data in the third hierarchy is merged into the second hierarchy; the processor is specifically configured to perform: and after the data in the third level is merged into the second level, updating the second identifier, wherein the updated second identifier is different from the first identifier.
In one possible implementation, the pointers include a read pointer and a write pointer; the processor is specifically configured to perform: and when the first identification and the second identification are the same, determining that the read pointer and the write pointer are valid.
In one possible implementation, the processor is specifically configured to perform: and after resource positioning is carried out in the data storage system according to the target keyword, updating the first identifier, wherein the updated first identifier is the same as the second identifier.
In one possible implementation, the processor is specifically configured to perform: reading target data pointed by the reading pointer; the target data pointed to by the write pointer is modified.
In one possible implementation, the processor is specifically configured to perform: and if the target data is found, assigning the memory address of the target data to the read pointer.
In one possible implementation, the processor is specifically configured to perform: and reading the target data pointed by the reassigned read pointer.
In one possible implementation, the processor is specifically configured to perform: if the target data is found, copying the target data to a first level; and assigning the memory address of the target data in the first hierarchy to the write pointer.
In one possible implementation, the processor is specifically configured to perform: and modifying the target data pointed by the write pointer after reassignment.
In one possible implementation, the processor is specifically configured to perform: if the target data is not found, creating an empty memory resource in the first level; and assigning the memory address of the empty memory resource to the write pointer.
In one possible implementation, the processor is specifically configured to perform: and writing the target data into the empty memory resource pointed by the write pointer after reassignment.
The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present specification may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present specification may be embodied directly in a hardware decoding processor, or in a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.
The electronic device may execute the method executed in the method embodiment shown in fig. 2, and implement the functions of the method embodiment shown in fig. 2, which are not described herein again in this specification.
Embodiments of the present specification also propose a computer-readable storage medium storing one or more programs, the one or more programs including instructions, which when executed by an electronic device including a plurality of application programs, enable the electronic device to perform the data processing method in the embodiment shown in fig. 2, and specifically perform the steps of the embodiment of the method shown in fig. 2.
The present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.
The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).
Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein were chosen in order to best explain the principles of the embodiments, the practical application, or technical improvements to the techniques in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (19)
1. A data processing method is applied to a data storage system based on a log structure merged Tree (LSM-Tree) structure, and comprises the following steps:
receiving a data processing request containing a target keyword, wherein the data processing request is used for requesting to execute data processing operation on target data corresponding to the target keyword;
and executing data processing operation on the target data through a handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data.
2. The method of claim 1, wherein performing data processing operations on the target data via a handle corresponding to the target key comprises:
judging whether the pointer is valid;
when the pointer is determined to be valid, performing a data processing operation on the target data through the pointer.
3. The method of claim 2, further comprising:
when the pointer is determined to be invalid, carrying out resource positioning in the data storage system according to the target keyword, and searching the target data;
reassigning the pointer according to the search result;
and executing data processing operation on the target data through the assigned finger.
4. The method of claim 3, wherein the LSM-Tree structure comprises at least a first level and a second level, the data in the first level having read-write characteristics and the data in the second level having read-only characteristics;
the method further comprises the following steps:
when the data amount of the data table in the first hierarchy exceeds a threshold value, determining the hierarchy of the data table as a third hierarchy, and generating a new empty data table and setting the hierarchy of the empty data table as the first hierarchy, wherein the data in the third hierarchy has a read-only characteristic;
merging data in the third hierarchy to the second hierarchy.
5. The method of claim 4, wherein the pointers comprise a read pointer and a write pointer;
the method further comprises the following steps:
setting the write pointer to a null pointer when a data table in the first level is null and data in the third level is not merged to the second level;
when the data in the third level is merged to the second level and the write pointer is a null pointer, setting the read pointer to be a null pointer.
6. The method of claim 5, wherein determining whether the pointer is valid comprises at least one of:
when the read pointer is a non-null pointer, determining that the read pointer is valid;
and when the write pointer is a non-null pointer, determining that the write pointer is valid.
7. The method of claim 4, further comprising a first identifier in the handle, and a second identifier in the LSM-Tree structure, wherein the first identifier and the second identifier are the same before merging the data in the third hierarchy into the second hierarchy;
the method further comprises the following steps:
and after the data in the third hierarchy is merged into the second hierarchy, updating the second identifier, wherein the updated second identifier is different from the first identifier.
8. The method of claim 7, wherein the pointers comprise a read pointer and a write pointer;
judging whether the pointer is valid, including:
and when the first identifier and the second identifier are the same, determining that the read pointer and the write pointer are valid.
9. The method of claim 8, further comprising:
and after resource positioning is carried out in the data storage system according to the target keyword, updating the first identifier, wherein the updated first identifier is the same as the second identifier.
10. The method according to claim 6 or 9, wherein when it is determined that the pointer is valid, performing a data processing operation on the target data by the pointer comprises at least one of the following steps:
reading the target data pointed by the read pointer;
and modifying the target data pointed by the write pointer.
11. The method according to claim 6 or 9, wherein reassigning the pointer according to the lookup result comprises:
and if the target data is found, assigning the memory address of the target data to the read pointer.
12. The method of claim 11, wherein performing data processing operations on the target data by the reassigned fingers comprises:
and reading the target data pointed by the read pointer after reassignment.
13. The method according to claim 6 or 9, wherein reassigning the pointer according to the lookup result comprises:
if the target data is found, copying the target data to the first level;
and assigning the memory address of the target data in the first level to the write pointer.
14. The method of claim 13, wherein performing data processing operations on the target data by the reassigned fingers comprises:
and modifying the target data pointed by the write pointer after reassignment.
15. The method according to claim 6 or 9, wherein reassigning the pointer according to the lookup result comprises:
if the target data is not found, creating an empty memory resource in the first level;
and assigning the memory address of the empty memory resource to the write pointer.
16. The method of claim 15, wherein performing data processing operations on the target data by the reassigned fingers comprises:
and writing the target data into the empty memory resource pointed by the write pointer after reassignment.
17. A data processing apparatus, wherein the apparatus is applied to a data storage system based on LSM-Tree structure, the apparatus comprising:
a receiving module, configured to receive a data processing request including a target keyword, where the data processing request is used to request to perform a data processing operation on target data corresponding to the target keyword;
and the data processing module is used for executing data processing operation on the target data through a handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data.
18. A data processing apparatus, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to perform the data processing method of any one of claims 1-16.
19. A non-transitory computer readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the data processing method of any one of claims 1 to 16.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910198443.9A CN111694992B (en) | 2019-03-15 | 2019-03-15 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910198443.9A CN111694992B (en) | 2019-03-15 | 2019-03-15 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111694992A true CN111694992A (en) | 2020-09-22 |
| CN111694992B CN111694992B (en) | 2023-05-26 |
Family
ID=72475364
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910198443.9A Active CN111694992B (en) | 2019-03-15 | 2019-03-15 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111694992B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114398378A (en) * | 2022-03-25 | 2022-04-26 | 北京奥星贝斯科技有限公司 | Method and device for determining index cost |
| WO2023029937A1 (en) * | 2021-08-31 | 2023-03-09 | 华为技术有限公司 | Data processing method, data processing apparatus, and related device |
| CN116431066A (en) * | 2023-03-21 | 2023-07-14 | 深圳市万翼数字技术有限公司 | Data storage method, device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105447059A (en) * | 2014-09-29 | 2016-03-30 | 华为技术有限公司 | Data processing method and device |
| CN105787093A (en) * | 2016-03-17 | 2016-07-20 | 清华大学 | Log file system construction method based on LSM-Tree structure |
| CN106844650A (en) * | 2017-01-20 | 2017-06-13 | 中国科学院计算技术研究所 | A kind of daily record merges the merging method and system of tree |
| US20170364414A1 (en) * | 2016-06-17 | 2017-12-21 | Acronis International Gmbh | System and method for data deduplication using log-structured merge trees |
| US20180107402A1 (en) * | 2016-10-19 | 2018-04-19 | Acronis International Gmbh | System and method for data storage using log-structured merge trees |
-
2019
- 2019-03-15 CN CN201910198443.9A patent/CN111694992B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105447059A (en) * | 2014-09-29 | 2016-03-30 | 华为技术有限公司 | Data processing method and device |
| CN105787093A (en) * | 2016-03-17 | 2016-07-20 | 清华大学 | Log file system construction method based on LSM-Tree structure |
| US20170364414A1 (en) * | 2016-06-17 | 2017-12-21 | Acronis International Gmbh | System and method for data deduplication using log-structured merge trees |
| US20180107402A1 (en) * | 2016-10-19 | 2018-04-19 | Acronis International Gmbh | System and method for data storage using log-structured merge trees |
| CN106844650A (en) * | 2017-01-20 | 2017-06-13 | 中国科学院计算技术研究所 | A kind of daily record merges the merging method and system of tree |
Non-Patent Citations (1)
| Title |
|---|
| 马文龙;朱妤晴;蒋德钧;熊劲;张立新;孟潇;包云岗;: "Key-Value型NoSQL本地存储系统研究" * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023029937A1 (en) * | 2021-08-31 | 2023-03-09 | 华为技术有限公司 | Data processing method, data processing apparatus, and related device |
| CN114398378A (en) * | 2022-03-25 | 2022-04-26 | 北京奥星贝斯科技有限公司 | Method and device for determining index cost |
| CN114398378B (en) * | 2022-03-25 | 2022-11-01 | 北京奥星贝斯科技有限公司 | Method and device for determining index cost |
| CN116431066A (en) * | 2023-03-21 | 2023-07-14 | 深圳市万翼数字技术有限公司 | Data storage method, device, electronic equipment and storage medium |
| CN116431066B (en) * | 2023-03-21 | 2024-04-26 | 深圳市万翼数字技术有限公司 | Data storage method, device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111694992B (en) | 2023-05-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111797073B (en) | Database management method, electronic device, and computer-readable storage medium | |
| CN109558525B (en) | Test data set generation method, device, equipment and storage medium | |
| CN107506181A (en) | Business processing, data processing method, device and electronic equipment | |
| CN111241040B (en) | Information acquisition method and device, electronic equipment and computer storage medium | |
| CN109284321B (en) | Data loading method, device, computing equipment and computer readable storage medium | |
| CN111125555B (en) | Enterprise information acquisition method and device | |
| CN111694992B (en) | Data processing method and device | |
| JP7254925B2 (en) | Transliteration of data records for improved data matching | |
| CN110866382A (en) | Document generation method, device, terminal equipment and medium | |
| CN110688111A (en) | Configuration method, device, server and storage medium of business process | |
| CN115934161A (en) | Code change influence analysis method, device and equipment | |
| CN111538667A (en) | Page testing method and device | |
| CN106156050B (en) | Data processing method and device | |
| US20140156708A1 (en) | Method and system for implementing an array using different data structures | |
| CN111831638A (en) | Data table creating method and device | |
| CN114880289A (en) | File grouping display method and computing device | |
| CN112148746A (en) | Method and device for generating database table structure document, electronic device and storage medium | |
| CN112486492A (en) | Page generation method and device, storage medium and electronic equipment | |
| CN116432185B (en) | Abnormality detection method and device, readable storage medium and electronic equipment | |
| CN111722785B (en) | Cache updating method and device | |
| CN114265846A (en) | Data operation method and device, electronic equipment and storage medium | |
| CN109460255B (en) | Memory address query method and device | |
| CN111967764A (en) | Service access verification method and device and electronic equipment | |
| CN117271440B (en) | File information storage method, reading method and related equipment based on freeRTOS | |
| CN111143232A (en) | Method, apparatus and computer program product for storing metadata |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |