CN111290710B - Cloud copy storage method and system based on dynamic adjustment of replication factors - Google Patents
Cloud copy storage method and system based on dynamic adjustment of replication factors Download PDFInfo
- Publication number
- CN111290710B CN111290710B CN202010063948.7A CN202010063948A CN111290710B CN 111290710 B CN111290710 B CN 111290710B CN 202010063948 A CN202010063948 A CN 202010063948A CN 111290710 B CN111290710 B CN 111290710B
- Authority
- CN
- China
- Prior art keywords
- replication
- data block
- subset
- nodes
- priority queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000010076 replication Effects 0.000 title claims abstract description 167
- 238000000034 method Methods 0.000 title claims abstract description 35
- 239000006185 dispersion Substances 0.000 claims abstract description 15
- 230000001351 cycling effect Effects 0.000 claims description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a cloud copy storage method and a cloud copy storage system based on a dynamic adjustment replication factor, wherein the cloud copy storage method comprises the following steps: generating a plurality of arrangements of all nodes according to a preset dispersion width and the number of nodes in the distributed storage system, and dividing a replication subset of replication factors according to the arrangements; the replication factors of the initial data blocks, adding all the data blocks into priority queues corresponding to the replication factors, and placing copies of the data blocks into corresponding replication subsets according to the priority queues; counting the access times of each data block, increasing the replication factor of the data block with the highest access times, adding the data block with the highest access times into a high-level priority queue, and reducing the replication factor of the data block with the lowest access times, so as to add the data block with the lowest access times into a low-level priority queue; and comparing the level of the priority queue of the last cycle of each data block with the level of the priority queue of the current cycle, and adjusting the copy subset of each data block according to the comparison result.
Description
Technical Field
The invention relates to a copy placement problem of a distributed storage system, in particular to a copy placement method designed for balancing the possibility of data loss and the number of copies, and belongs to the field of distributed computing.
Background
The current age is an information explosion age, data is rapidly increased, the traditional storage mode can not meet the needs of the age, distributed storage starts to rapidly develop to meet the needs, but a distributed storage system is mostly composed of cheap commercial machines, so that a certain node in the system is invalid and becomes a normal state, the data can not be lost when the node is invalid, and the reliability and the usability of the data are guaranteed by the existing distributed storage system in a mode of placing multiple copies.
The method of copy placement can affect the reliability and availability of data in the system and even the performance of the overall system. This makes it very interesting to study the replica placement technique. Existing distributed storage systems such as HDFS, GFS default to randomly select copies, but this approach has proven to almost certainly lose data when one percent of nodes fail at the same time in a large-scale cluster, and HDFS default to static replication factors, with a default value of 3, that is, each data block will store 3 copies in the system, so that although the reliability and availability of data are guaranteed, the heat of the data blocks is not differentiated, the load of the nodes storing the data blocks with high heat is improved, while some data blocks are hardly accessed or accessed for a low number of times, and not so many copies are needed, which may make the load of the whole cluster unbalanced. The concept of duplicating subsets is also proposed in the prior art, and using the concept to place duplicates can significantly reduce the probability of data loss, but the purpose of the concept is to improve the durability of the system, and the problem of load balancing is not considered.
Disclosure of Invention
The invention provides a method for placing copies, which aims to improve the storage efficiency of the whole system by reducing the probability of data loss in a distributed storage system and dynamically changing the number of data block copies. The method divides the data blocks into different priorities according to the access times of the data blocks in the period, adds the data blocks belonging to the different priorities into the copy subsets of different copy factors, and limits the copy placement quantity. In addition, the method also provides a method for converting the data blocks among the replication subsets of different replication factors, so as to achieve the purpose of dynamically changing the replication factors.
Aiming at the defects of the prior art, the invention provides a cloud copy storage method based on dynamic adjustment of replication factors, which comprises the following steps:
step 1, generating a plurality of arrangements of all nodes according to a preset dispersion width and the number of nodes in a distributed storage system, and dividing a replication subset of replication factors according to the arrangements;
initializing replication factors of data blocks in the distributed storage system, adding all the data blocks into priority queues corresponding to the replication factors, and placing copies of the data blocks into corresponding replication subsets according to the priority queues;
step 3, periodically counting the access times of each data block, increasing the replication factor of the data block with the highest access times, adding the data block with the highest access times into a high-level priority queue, and reducing the replication factor of the data block with the lowest access times, so as to add the data block with the lowest access times into a low-level priority queue;
and 4, comparing the level of the priority queue of the last cycle of each data block with the level of the priority queue of the current cycle, and adjusting the copy subset of each data block according to the comparison result to add or delete or keep copies.
And 5, cycling the step 3 and the step 4 to dynamically adjust the replication factors of the data blocks in the distributed storage system so as to dynamically change the number of the multiple copies of each data block.
The cloud copy storage method based on the dynamic adjustment replication factors comprises the following steps that in the step 1, constraint conditions are included in the arrangement process of generating a plurality of all nodes;
and judging whether the generated arrangement meets the constraint condition, if not, rearranging all nodes randomly to generate new arrangements until P arrangements meeting the constraint condition are generated.
The cloud copy storage method based on the dynamic adjustment replication factor, wherein the process of forming the arrangement number P of all the plurality of nodes in the step 1 comprises the following steps:
wherein S is the dispersion width; r is the smallest replication factor in the dynamic adjustment range.
The cloud replica storage method based on dynamically adjusting replication factors, wherein the constraint includes restricting nodes in the same replication subset to appear on the same rack of the distributed storage system.
According to the cloud copy storage method based on the dynamic adjustment replication factors, in the step 4, the replication subset where each data block is located is adjusted according to the comparison result, and copies are added, deleted or kept, and the method specifically comprises the following steps:
when the comparison result is that the level R' of the priority queue of the data block in the previous period is larger than the level R of the priority queue of the current period, checking whether the replication subset of the data block is a subset of replication subsets with replication factors equal to R, if so, adding and placing a copy in the replication subset with replication factors equal to R;
when R' is greater than R, checking whether all replication factors are equal to the replication subset of R, and if so, directly deleting the replicas on the nodes with the difference between the two sets.
The invention also provides a cloud copy storage system based on the dynamic adjustment of the replication factors, which comprises:
the method comprises the steps that a module 1, according to a preset dispersion width and the number of nodes in a distributed storage system, an arrangement of a plurality of all nodes is generated, and a replication subset of replication factors is divided according to the arrangement;
the module 2 initializes the replication factor of the data blocks in the distributed storage system, adds all the data blocks into a priority queue corresponding to the replication factor, and places the copies of the data blocks into corresponding replication subsets according to the priority queue;
the module 3 periodically counts the access times of all the data blocks, increases the replication factor of the data block with the highest access times, adds the data block with the highest access times into a high-level priority queue, and reduces the replication factor of the data block with the lowest access times, so as to add the data block with the lowest access times into a low-level priority queue;
and a module 4, comparing the level of the priority queue of the last cycle of each data block with the level of the priority queue of the current cycle, and adjusting the copy subset of each data block according to the comparison result, and adding or deleting or maintaining the copy.
Module 5, looping the module 3 and the module 4 dynamically adjusts the replication factor of the data blocks in the distributed storage system to dynamically change the number of multiple copies of each data block.
The cloud copy storage system based on the dynamic adjustment replication factors comprises constraint conditions in the arrangement process of generating a plurality of all nodes in the module 1;
and judging whether the generated arrangement meets the constraint condition, if not, rearranging all nodes randomly to generate new arrangements until P arrangements meeting the constraint condition are generated.
The cloud copy storage system based on the dynamic adjustment replication factor, wherein the process of arranging the number P of the plurality of all nodes in the module 1 comprises the following steps:
wherein S is the dispersion width; r is the smallest replication factor in the dynamic adjustment range.
The cloud replica storage system based on dynamically adjusting replication factors, wherein the constraint includes restricting nodes in the same replication subset to appear on the same rack of the distributed storage system.
The cloud copy storage system based on the dynamic adjustment of the replication factors, wherein the module 4 adjusts the replication subset of each data block according to the comparison result, and adds, deletes or maintains the copies, specifically includes:
when the comparison result is that the level R' of the priority queue of the data block in the previous period is larger than the level R of the priority queue of the current period, checking whether the replication subset of the data block is a subset of replication subsets with replication factors equal to R, if so, adding and placing a copy in the replication subset with replication factors equal to R;
when R' is greater than R, checking whether all replication factors are equal to the replication subset of R, and if so, directly deleting the replicas on the nodes with the difference between the two sets.
Drawings
FIG. 1 is a flow chart of a system of the present invention;
fig. 2 is a schematic diagram of the correspondence between priority queues and replicated subsets.
Detailed Description
Specifically, the invention comprises the following steps:
A. the replication subsets are partitioned according to a user-provided dispersion width S and a number of nodes N in the distributed storage system.
A1. And randomly generating P all N node arrangements according to S input by a user. Corresponding constraints may be added in generating the permutation (e.g., limiting nodes in the same replication subset to appear on the same shelf).
A2. The replication subsets of the different replication factors R are partitioned according to the arrangement described above.
B. The replicas are placed based on the partitioned replication subsets.
B1. All data blocks are added to the priority queue of r=3 by default with a replication factor of 3 initially.
B2. And placing copies of the data blocks into corresponding replication subsets according to the priority queues. In this embodiment, four queue objects are specifically created and are respectively used to store data blocks with different priorities, and each queue pair applies all replication subsets generated by one replication factor R (that is, a data block in a queue with a certain priority can only place a copy in the replication subset generated by a specific replication factor R). The correspondence between priority queues and replicated subsets is as follows: the data block in the highest priority queue places the replica using the replica subset of r=5, the next highest replica subset of r=4, and so on, as shown in fig. 2. When placing copies of a block of data, one is randomly selected from all corresponding copy subsets to place.
C. Periodically counting the access times of the data blocks and adjusting the priority of the data blocks.
C1. Periodically (e.g., in 10 minutes) count the number of times each data block is accessed, and order the data blocks by the number of times accessed in the period.
C2. The data block with the highest 10% of the accessed times is added to the priority queue with r=5.
C3. The next 10% of the data blocks are added to the priority queue with r=4.
C4. The 30% of the data blocks with the lowest number of accesses are added to the priority queue with r=2.
C5. The remaining data blocks are added to the priority queue with r=3.
D. The placement of the copies is dynamically adjusted according to priority.
D1. The duplicate factor of the priority queue for the last cycle of each data block is denoted as R'. And according to a corresponding R' to R (replication factor of the priority queue at present) strategy, adjusting a replication subset of the data block, and adding or deleting the replicas.
D2. And after the placement adjustment of the copy is completed, returning to C to count the access times of the data block in the next period.
The invention has the advantages that the invention provides a copy placement method capable of dynamically changing the replication factor and reducing the data loss probability, and the priority queue where the data block is located is changed by periodically counting the access times of the data block, and the replication subset where the data block is located is adjusted according to the priority. Finally, the aim of improving the storage efficiency is achieved.
In order to make the above features and effects of the present invention more clearly understood, the following specific examples are given with reference to the accompanying drawings.
The steps of the present invention are further described below with reference to fig. 1, as in fig. 1, the steps of the present invention include: A. dividing the replication subsets; B. placing copies based on the partitioned copy subsets; C. periodically counting the access times of the data blocks and adjusting the priority of the data blocks; D. the placement of the copies is dynamically adjusted according to priority. One specific embodiment is as follows:
A. the replication subsets are divided according to a user-provided dispersion width S and the number of nodes N in the system.
A1. And randomly generating P all N node arrangements according to S input by a user. Corresponding constraints may be added in generating the permutation (e.g., limiting nodes in the same replication subset to appear on the same shelf).
A11. Wherein the dispersion width S is defined as: the data of one data node may be uniformly dispersed over S other nodes. (if s=4, assuming that there are duplicate subsets { N1, N2, N3} and { N1, N4, N5}, then the data blocks on node N1 will be evenly distributed across nodes N2, N3, N4, N5 because when the duplicate is placed using the duplicate subsets, the duplicate of the same data block can only be placed within the same duplicate subset
A12. Wherein the number of permutations P is represented by the formula:generating, wherein S is a dispersion width; r is the smallest replication factor in the dynamic adjustment range, i.e. r=2. If P is not an integer, rounding up.
A13. Wherein the arrangement generated by constraint restriction can be added to meet the requirement, and if the generated arrangement does not meet the constraint restriction, all nodes are rearranged randomly to generate new arrangements until P arrangements meeting the constraint are generated.
A2. The replication subsets of the different replication factors R are partitioned according to the arrangement described above.
A21. The rules for generating duplicate subsets from the permutations are as follows: each R node in the arrangement is divided into a replicated subset in sequence. (e.g., total number of nodes in the system N=9, replication factor R=3, existing permutations of N1, N2, N3, N4, N5, N6, N7, N8, N9, the replication subsets { N1, N2, N3} { N4, N5, N6} { N7, N8, N9 })
A22. The meaning of the replication subsets that separate out the different replication factors R is: the replication subsets are generated separately for the same permutation using different replication factors R. (e.g., the existing arrangement N1, N2, N3, N4, N5, N6. uses replication factors r=2 and r=3 to generate replication subsets, resulting in replication subsets { N1, N2} { N3, N4} { N5, N6} { N1, N2, N3} { N4, N5, N6 }) replication factors used in the method have r=2, r=3, r=4, r=5.
B. The replicas are placed based on the partitioned replication subsets.
B1. All data blocks are added to the priority queue of r=3 by default with a replication factor of 3 initially.
B11. The method uses four priority queues, and the priorities are from high to low: a priority queue of r=5, a priority queue of r=4, a priority queue of r=3, and a priority queue of r=2. The data blocks in these four queues are duplicated using the duplicate subsets generated by r=5, r=4, r=3, r=2, respectively.
B12. The initial replication factor for all data blocks is 3, and the replication factor for all newly added data blocks is 3 as with the default HDFS.
B2. And placing copies of the data blocks into corresponding replication subsets according to the priority queues.
B21. The placement rules of the data blocks are as follows, the first replica randomly selects nodes in a system to place, and the rest of the replicas are placed in the same replication subset containing the first replica placement node, if a plurality of replication subsets contain the placement nodes of the first replica, one of the replication subsets is randomly selected to place the rest of the nodes. (e.g., when R=3, there are replication subsets { N1, N2, N3} and { N1, N4, N5}, when the first replica is placed on node N1, the remaining two replicas will be placed on N2 and N3, or N4 and N5.)
C. Periodically counting the access times of the data blocks and adjusting the priority of the data blocks.
C1. Periodically (e.g., in 10 minutes) count the number of times each data block is accessed, and order the data blocks by the number of times accessed in the period.
C2. The data block with the highest 10% of the accessed times is added to the priority queue with r=5.
C3. The next 10% of the data blocks are added to the priority queue with r=4.
C4. The 30% of the data blocks with the lowest number of accesses are added to the priority queue with r=2.
C5. The remaining data blocks are added to the priority queue with r=3.
D. The placement of the copies is dynamically adjusted according to priority.
D1. The duplicate factor of the priority queue for the last cycle of each data block is denoted as R'. And according to a corresponding R' to R (replication factor of the priority queue at present) strategy, adjusting a replication subset of the data block, and adding or deleting the replicas.
D11. Wherein the R' to R strategy is detailed as follows: when R' is smaller than R, that is, when the priority of the data block becomes high, checking whether the replication subset of the data block is a subset of replication subsets with a replication factor equal to R, and if so, adding and placing a copy in the replication subset with the replication factor equal to R. (e.g., an existing data block is added from the priority queue of r=2 to the priority queue of r=3. The copy subset of the original block is { N1, N2}, that is, the block is stored on nodes N1 and N2, there are two copies in total, there is one copy subset { N1, N2, N3}, the check finds that { N1, N2} is a subset of { N1, N2, N3}, so it is only necessary to add a copy to node N3.) if not, find the copy subset of the data block and the largest common subset of all copy subsets whose copy factor is equal to R, delete the data copy on the node of the difference between the copy subset of the data block and the largest common subset, and add the copy to the non-largest common subset node of the copy subset containing the copy factor of the largest common subset equal to R. ( For example, there is a block of data that goes from a priority queue of r=3 to a priority queue of r=4, where the duplicate subset where the block originally resides is { N4, N5, N6}, and there is a duplicate subset of r=4 { N1, N2, N3, N4} { N5, N6, N7, N8}. { N4, N5, N6} is not a subset of { N1, N2, N3, N4} and { N5, N6, N7, N8}, find { N4, N5, N6} and { N1, N2, N3, N4}, { N5, N6, N7, N8} the largest common subset is { N5, N6}, delete the copy of the block of data on the difference of { N4, N5, N6} and { N5, N6}, i.e., the copy of the block of data on N4, add a placement copy in the copy subset of R=4 that contains { N5, N6}, i.e., add a placement copy on N7, N8 on the copy subset { N5, N6, N7, N8}. )
D12. When R 'is greater than R, i.e., the priority of the data block becomes lower, it is checked whether all replication factors equal to R' are equal to the subset of the replicated subset in which the data block is located, if there is a direct deletion of the replicas on the bad node of the two sets. Two sets refer to two sets, one set is a copy subset (the copy subset is a set of nodes) used when the current data block places a copy, and the other set is to find out all copy subsets with a copy factor equal to R, so that the set has the most common part with the first set. It is then prioritized whether the two constitute a subset relationship, and the largest common subset of the two is considered. (e.g., a current data block falls from the priority queue of R=3 to the priority queue of R=2, requiring deletion of a copy.) the copy subset of the data block is { N1, N2, N3}, there is a copy subset { N1, N2} in the system, checking to find that { N1, N2} is a subset of { N1, N2, N3} and therefore, deleting only the data copy on N3.) if not, a copy subset of the data block and the largest common subset of all copy subsets with a copy factor equal to R are found, the data copy on the node of the difference between the copy subset of the data block and the largest common subset is deleted, and the placed copy is added on the non-largest common subset node in the copy subset containing the largest common subset with a copy factor equal to R. ( For example, there is a block of data that drops from a priority queue of r=5 to a priority queue of r=4, where the duplicate subset where the block was located is { N6, N7, N8, N9, N10}, where there is a duplicate subset of r=4 { N5, N6, N7, N8}, { N9, N10, N11, N12}, in the system. { N5, N6, N7, N8}, { N9, N10, N11, N12} is not a subset of { N6, N7, N8, N9, N10}, find out { N6, N7, N8, N9, N10} and { N5, N6, N7, N8}, { N9, N10, N11, N12} the largest common subset is { N6, N7, N8}, delete the copy of the block of data on the difference of { N6, N7, N8, N9, N10} from { N6, N7, N8}, i.e., the copy of the block of data on N9 and N10, add a placed copy to the replicated subset of R=4 that contains { N6, N7, N8}, i.e., add a placed copy to N5 on the replicated subset { N5, N6, N7, N8}. )
D2. And after the placement adjustment of the copy is completed, returning to C to count the access times of the data block in the next period.
According to the invention, the access times of the data blocks in the period are counted and sequenced, the data blocks are added into different priority queues, and different copy numbers are placed for the data in the different priority queues, so that the purpose of load balancing is achieved.
The following is a system example corresponding to the above method example, and this embodiment mode may be implemented in cooperation with the above embodiment mode. The related technical details mentioned in the above embodiments are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the related technical details mentioned in the present embodiment can also be applied to the above-described embodiments.
The invention also provides a cloud copy storage system based on the dynamic adjustment of the replication factors, which comprises:
the method comprises the steps that a module 1, according to a preset dispersion width and the number of nodes in a distributed storage system, an arrangement of a plurality of all nodes is generated, and a replication subset of replication factors is divided according to the arrangement;
the module 2 initializes the replication factor of the data blocks in the distributed storage system, adds all the data blocks into a priority queue corresponding to the replication factor, and places the copies of the data blocks into corresponding replication subsets according to the priority queue;
the module 3 periodically counts the access times of all the data blocks, increases the replication factor of the data block with the highest access times, adds the data block with the highest access times into a high-level priority queue, and reduces the replication factor of the data block with the lowest access times, so as to add the data block with the lowest access times into a low-level priority queue;
and a module 4, comparing the level of the priority queue of the last cycle of each data block with the level of the priority queue of the current cycle, and adjusting the copy subset of each data block according to the comparison result, and adding or deleting or maintaining the copy.
Module 5, looping the module 3 and the module 4 dynamically adjusts the replication factor of the data blocks in the distributed storage system to dynamically change the number of multiple copies of each data block.
The cloud copy storage system based on the dynamic adjustment replication factors comprises constraint conditions in the arrangement process of generating a plurality of all nodes in the module 1;
and judging whether the generated arrangement meets the constraint condition, if not, rearranging all nodes randomly to generate new arrangements until P arrangements meeting the constraint condition are generated.
The cloud copy storage system based on the dynamic adjustment replication factor, wherein the process of arranging the number P of the plurality of all nodes in the module 1 comprises the following steps:
wherein S is the dispersion width; r is the smallest replication factor in the dynamic adjustment range.
The cloud replica storage system based on dynamically adjusting replication factors, wherein the constraint includes restricting nodes in the same replication subset to appear on the same rack of the distributed storage system.
The cloud copy storage system based on the dynamic adjustment of the replication factors, wherein the module 4 adjusts the replication subset of each data block according to the comparison result, and adds, deletes or maintains the copies, specifically includes:
when the comparison result is that the level R' of the priority queue of the data block in the previous period is larger than the level R of the priority queue of the current period, checking whether the replication subset of the data block is a subset of replication subsets with replication factors equal to R, if so, adding and placing a copy in the replication subset with replication factors equal to R;
when R' is greater than R, checking whether all replication factors are equal to the replication subset of R, and if so, directly deleting the replicas on the nodes with the difference between the two sets.
Claims (2)
1. The cloud copy storage method based on the dynamic adjustment of the replication factors is characterized by comprising the following steps of:
step 1, generating a plurality of arrangements of all nodes according to a preset dispersion width and the number of nodes in a distributed storage system, and dividing a replication subset of replication factors according to the arrangements;
initializing replication factors of data blocks in the distributed storage system, adding all the data blocks into priority queues corresponding to the replication factors, and placing copies of the data blocks into corresponding replication subsets according to the priority queues;
step 3, periodically counting the access times of each data block, increasing the replication factor of the data block with the highest access times, adding the data block with the highest access times into a high-level priority queue, and reducing the replication factor of the data block with the lowest access times, so as to add the data block with the lowest access times into a low-level priority queue;
step 4, comparing the level of the priority queue of the previous cycle of each data block with the level of the priority queue of the current cycle, and adjusting the copy subset of each data block according to the comparison result, and adding or deleting or maintaining the copy;
step 5, the step 3 and the step 4 are circulated, and the replication factors of the data blocks in the distributed storage system are dynamically adjusted so as to dynamically change the number of the multiple copies of each data block;
wherein the arrangement process of generating a plurality of all nodes in the step 1 comprises constraint conditions; judging whether the generated arrangement meets the constraint condition, if not, rearranging all nodes randomly to generate new arrangements until P arrangements meeting the constraint condition are generated;
the process of arranging the number P of the plurality of all nodes in the step 1 comprises the following steps:
wherein S is a dispersion width, and the data of one data node can be uniformly dispersed on S other nodes; r is the smallest replication factor in the dynamic adjustment range;
in the step 4, according to the comparison result, the copy subset of each data block is adjusted, and the copies are added, deleted or kept, which specifically includes:
when the comparison result is that the level R' of the priority queue of the data block in the previous period is larger than the level R of the priority queue of the current period, checking whether the replication subset of the data block is a subset of replication subsets with replication factors equal to R, if so, adding and placing a copy in the replication subset with replication factors equal to R;
when R' is larger than R, checking whether all replication factors are equal to the replication subsets of R or not, if yes, directly deleting the replicas on the nodes with the difference between the two sets;
the constraint includes restricting nodes in the same replication subset to appear on the same chassis of the distributed storage system.
2. A cloud replica storage system based on dynamically adjusting replication factors, comprising:
the method comprises the steps that a module 1, according to a preset dispersion width and the number of nodes in a distributed storage system, an arrangement of a plurality of all nodes is generated, and a replication subset of replication factors is divided according to the arrangement;
the module 2 initializes the replication factor of the data blocks in the distributed storage system, adds all the data blocks into a priority queue corresponding to the replication factor, and places the copies of the data blocks into corresponding replication subsets according to the priority queue;
the module 3 periodically counts the access times of all the data blocks, increases the replication factor of the data block with the highest access times, adds the data block with the highest access times into a high-level priority queue, and reduces the replication factor of the data block with the lowest access times, so as to add the data block with the lowest access times into a low-level priority queue;
the module 4 compares the level of the priority queue of the last cycle of each data block with the level of the priority queue of the current cycle, adjusts the copy subset of each data block according to the comparison result, and adds or deletes or keeps copies;
module 5, cycling the module 3 and the module 4 to dynamically adjust the replication factors of the data blocks in the distributed storage system so as to dynamically change the number of the multiple copies of each data block;
the arrangement process of generating a plurality of all nodes in the module 1 comprises constraint conditions; judging whether the generated arrangement meets the constraint condition, if not, rearranging all nodes randomly to generate new arrangements until P arrangements meeting the constraint condition are generated;
the process of arranging the number P of the plurality of all nodes in the module 1 comprises the following steps:
wherein S is a dispersion width, and the data of one data node can be uniformly dispersed on S other nodes; r is the smallest replication factor in the dynamic adjustment range;
the module 4 adjusts the copy subset of each data block according to the comparison result, and adds, deletes or maintains the copy, which specifically includes:
when the comparison result is that the level R' of the priority queue of the data block in the previous period is larger than the level R of the priority queue of the current period, checking whether the replication subset of the data block is a subset of replication subsets with replication factors equal to R, if so, adding and placing a copy in the replication subset with replication factors equal to R;
when R' is larger than R, checking whether all replication factors are equal to the replication subsets of R or not, if yes, directly deleting the replicas on the nodes with the difference between the two sets;
the constraint includes restricting nodes in the same replication subset to appear on the same chassis of the distributed storage system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010063948.7A CN111290710B (en) | 2020-01-20 | 2020-01-20 | Cloud copy storage method and system based on dynamic adjustment of replication factors |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010063948.7A CN111290710B (en) | 2020-01-20 | 2020-01-20 | Cloud copy storage method and system based on dynamic adjustment of replication factors |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111290710A CN111290710A (en) | 2020-06-16 |
| CN111290710B true CN111290710B (en) | 2024-04-05 |
Family
ID=71023299
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010063948.7A Active CN111290710B (en) | 2020-01-20 | 2020-01-20 | Cloud copy storage method and system based on dynamic adjustment of replication factors |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111290710B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111860609B (en) * | 2020-06-29 | 2023-08-25 | 深圳大学 | Cross-data-center data analysis method, device, equipment and storage medium |
| CN111796973B (en) * | 2020-06-30 | 2024-08-06 | 南京航空航天大学 | Data recovery and optimization method for data center storage faults |
| CN112527751B (en) * | 2020-12-16 | 2023-10-31 | 中国联合网络通信集团有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN115277858B (en) * | 2022-09-23 | 2022-12-20 | 太极计算机股份有限公司 | Data processing method and system for big data |
| KR20240060293A (en) * | 2022-10-28 | 2024-05-08 | 삼성에스디에스 주식회사 | System, method and computer readable storage medium for partitioning keyspace |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105760391A (en) * | 2014-12-18 | 2016-07-13 | 华为技术有限公司 | Data dynamic redistribution method and system, data node and name node |
| CN107003933A (en) * | 2014-05-27 | 2017-08-01 | 北京大学深圳研究生院 | The method that construction method, device and its data of part replica code are repaired |
| CN107807793A (en) * | 2017-10-27 | 2018-03-16 | 清华大学 | The storage of data trnascription isomery and access method in distributed computer storage system |
| CN110457280A (en) * | 2019-08-02 | 2019-11-15 | 南京邮电大学 | A kind of HDFS copy replicates improvement strategy again |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10102086B2 (en) * | 2014-12-19 | 2018-10-16 | Futurewei Technologies, Inc. | Replicated database distribution for workload balancing after cluster reconfiguration |
-
2020
- 2020-01-20 CN CN202010063948.7A patent/CN111290710B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107003933A (en) * | 2014-05-27 | 2017-08-01 | 北京大学深圳研究生院 | The method that construction method, device and its data of part replica code are repaired |
| CN105760391A (en) * | 2014-12-18 | 2016-07-13 | 华为技术有限公司 | Data dynamic redistribution method and system, data node and name node |
| CN107807793A (en) * | 2017-10-27 | 2018-03-16 | 清华大学 | The storage of data trnascription isomery and access method in distributed computer storage system |
| CN110457280A (en) * | 2019-08-02 | 2019-11-15 | 南京邮电大学 | A kind of HDFS copy replicates improvement strategy again |
Non-Patent Citations (3)
| Title |
|---|
| 分布式存储系统中数据修复的自适应流控机制;陈志鹏;李小勇;;信息技术(第11期);全文 * |
| 基于可分组设计的部分重复码研究;朱兵;李挥;陈俊;侯韩旭;周泰;;通信学报(第02期);全文 * |
| 异构Hadoop集群中数据副本放置策略优化;刘艳;蔡燕冬;谢晓东;张庆磊;;华中科技大学学报(自然科学版)(第07期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111290710A (en) | 2020-06-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111290710B (en) | Cloud copy storage method and system based on dynamic adjustment of replication factors | |
| US8543782B2 (en) | Content-based, compression-enhancing routing in distributed, differential electronic-data storage systems | |
| CN102968503B (en) | The data processing method of Database Systems and Database Systems | |
| JP6553649B2 (en) | Clustering storage method and apparatus | |
| CN106990915B (en) | Storage resource management method based on storage medium type and weighted quota | |
| EP4340331A2 (en) | Replicated database distribution for workload balancing after cluster reconfiguration | |
| US10228857B2 (en) | Expanding variable sub-column widths as needed to store data in memory | |
| CN107729535B (en) | Method for configuring bloom filter in key value database | |
| CN103838860A (en) | File storing system based on dynamic transcript strategy and storage method of file storing system | |
| US8352470B2 (en) | Adaptive aggregation: improving the performance of grouping and duplicate elimination by avoiding unnecessary disk access | |
| CN105242983A (en) | Data storage method and data storage management server | |
| WO2016180049A1 (en) | Storage management method and distributed file system | |
| CN108073472B (en) | Memory erasure code distribution method based on heat perception | |
| KR20150035876A (en) | Method for de-duplicating data and apparatus therefor | |
| CN103473368A (en) | Virtual machine real-time migration method and system based on counting rank ordering | |
| CN112306692B (en) | Method and device for group-in-group balancing of distributed storage system | |
| CN112380004B (en) | Memory management method, memory management device, computer readable storage medium and electronic equipment | |
| CN108920282A (en) | A kind of copy of content generation, placement and the update method of holding load equilibrium | |
| Berkovich et al. | Reversing the error-correction scheme for a fault-tolerant indexing | |
| CN109992196B (en) | Index data storage method and device and storage system | |
| Mense et al. | Spread: An adaptive scheme for redundant and fair storage in dynamic heterogeneous storage systems | |
| CN107341113B (en) | Cache compression method and device | |
| US20130290378A1 (en) | Adaptive probabilistic indexing with skip lists | |
| CN107168649B (en) | method and device for data distribution in distributed storage system | |
| US11036678B2 (en) | Optimizing files stored in a distributed file system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |