CN114880297A - Distributed data deduplication method and system based on fingerprints - Google Patents
Distributed data deduplication method and system based on fingerprints Download PDFInfo
- Publication number
- CN114880297A CN114880297A CN202210364761.XA CN202210364761A CN114880297A CN 114880297 A CN114880297 A CN 114880297A CN 202210364761 A CN202210364761 A CN 202210364761A CN 114880297 A CN114880297 A CN 114880297A
- Authority
- CN
- China
- Prior art keywords
- file
- fingerprint
- data
- data block
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
- G06F16/1752—De-duplication implemented within the file system, e.g. based on file segments based on file chunks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/137—Hash-based
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a distributed data deduplication method and a distributed data deduplication system based on fingerprints, wherein the method comprises the following steps: initiating a file execution management request; slicing the received file into at least one data block; calculating the fingerprint of each data block and the fingerprint of the whole file by using a consistent Hash storage algorithm, and respectively routing each data block and the whole file to corresponding target storage servers; and constructing the description information of each data block and the whole file, and executing file management through the target storage server. According to the technical scheme of the embodiment of the invention, in the process of file management execution by an enterprise, efficient data deduplication can be realized, and meanwhile, data can be distributed and stored in different storage servers through the system, so that load balancing is realized.
Description
Technical Field
The invention relates to the technical field of information interaction input processing, in particular to a distributed data deduplication method and system based on fingerprints.
Background
At present, an enterprise data center manages massive data, and a cloud storage service provider provides storage service for a plurality of enterprise users. However, enterprise data storage and management costs are high, and expanding storage resources is extremely challenging, and researches show that almost 75% of data is redundant, and particularly, the redundancy of data in a backup and archiving storage system is higher than 90%, so that the memory requirement and the hard disk cost can be greatly reduced by the data deduplication technology in cloud storage.
For data already stored in the system, only links to the data need to be saved to reduce data redundancy by the data deduplication model at present. Although the data deduplication model can avoid redundant data in a single file and further identify redundant data between different files through data comparison, the traditional data deduplication process needs higher computational complexity and I/O throughput, and then server performance is affected.
Disclosure of Invention
The invention provides a distributed data deduplication method and a distributed data deduplication system based on fingerprints, which can solve the problems so as to improve data deduplication efficiency and reduce redundancy of data in a backup and archiving storage system.
Additional features and advantages of the invention will be set forth in the detailed description which follows, or may be learned by practice of the invention.
In a first aspect, the present invention provides a fingerprint-based distributed data deduplication method, including:
initiating a file execution management request;
slicing the received file into at least one data block;
calculating the fingerprint of each data block and the fingerprint of the whole file by using a consistent Hash storage algorithm, and respectively routing each data block and the whole file to corresponding target storage servers;
and constructing the description information of each data block and the whole file, and executing file management through the target storage server.
Preferably, the method for computing the fingerprint of each data block and the fingerprint of the whole file by using the consistent hash storage algorithm comprises the following steps:
respectively carrying out Hash operation on each data block, and calculating to obtain the fingerprint of each data block;
and calculating the fingerprint of the file according to the fingerprint of each data block.
Preferably, the method of routing the fingerprint of each data chunk and the fingerprint of the entire file to the corresponding target storage server comprises:
taking the fingerprint of each data block and the fingerprint of the whole file as target values key;
each storage server node maintains a consistent Hash routing table, and performs routing operation through the consistent Hash routing table maintained by the storage server node according to the determined target value key to find out the direct successor of the target value key and the target storage server information corresponding to the direct successor;
determining a target storage server according to the target storage server information;
wherein, the direct successor of the target value key is the storage server node which is closest to the target value key and is smaller than the target value key.
Preferably, the method for obtaining the consistent hash routing table includes:
respectively carrying out Hash operation on each storage server to obtain fingerprint information of each server;
taking the fingerprint information of each server as a storage value key;
and establishing and storing a corresponding relation among each storage server node, successors of the storage value keys and the information of the storage servers to obtain a consistent Hash routing table preset in the storage servers.
Preferably, the method of slicing the file into at least one data block comprises:
receiving the file transmitted in a data stream;
the data stream is partitioned into at least one data block of a fixed size according to its size.
Preferably, the method for performing file management by the target storage server includes:
constructing description information of the data block according to the fingerprint of the data block and target storage server information thereof, and constructing description information of the file according to the fingerprint of the file and the target storage server information thereof;
transmitting the description information of the data block and the file to a target storage server;
comparing the original data in the target storage server with the description information of the data blocks and the files, and executing file management on the data blocks and the files according to the comparison result.
In a second aspect, the present invention provides a fingerprint-based distributed data deduplication system, comprising:
the client initiates a file execution management request to the file;
the cloud data processing server is used for slicing the received file into at least one data block, performing a consistent Hash storage algorithm on each data block, calculating the fingerprint of each data block and the fingerprint of the whole file, and routing the fingerprints to corresponding target storage servers;
and the target storage server executes file management on each data block and the whole file according to the description information of each data block and the whole file constructed by the cloud data processing server.
Preferably, the cloud data processing server includes:
the cutting module is used for receiving and analyzing the data stream of the file and slicing the data stream into at least one data block according to the size of the data stream;
the fingerprint calculation module is used for calculating each data block according to a stored consistent Hash storage algorithm, calculating the fingerprint of each data block, and calculating the fingerprint of the file according to the fingerprint of each data block;
the routing module is used for carrying out routing operation through each consistent Hash routing table maintained by each storage server according to the fingerprint of each data block and the fingerprint of the file transmitted by the fingerprint calculation module, finding out the direct successor of the target value key and the target storage server information corresponding to the direct successor of the target value key, and determining the target storage server of each data block and the file according to the target storage server information;
and the information construction module is used for constructing the description information of each data block and the file according to the fingerprint of each data block and the information of the target storage server and sending the description information to the corresponding target storage server.
Preferably, the target storage server includes:
the local data storage module is used for storing an information file of original data, and establishing and storing a data block information description file and a file information description file corresponding to transmission according to the description information of each data block and file transmitted by the information construction module;
and the local data management module is used for comparing the data block information description file and the file information description file transmitted by the local data storage module with the information description file of the original data and executing file management on the data block and the file according to a comparison result.
In a third aspect, the present invention provides a computer readable medium having stored thereon a computer program which, when executed by a processor, implements a fingerprint based distributed data deduplication method as described in any of the first aspects above.
In a fourth aspect, the present invention provides an electronic device, comprising: a processor, a memory, and a bus; the memory is configured to store a computer program, the processor is connected to the memory through the bus, and when the electronic device is running, the processor executes the computer program of the memory, so as to enable the processor to execute the fingerprint-based distributed data deduplication method according to any one of the first aspect.
The technical scheme provided by the embodiment of the invention has the following beneficial effects:
1. the fingerprints of the data blocks obtained by calculating the file and the fingerprint of each data block obtained by slicing are the same, and the target storage server does not need to repeatedly store the data blocks and the files with the same fingerprints, so that the duplication removal of the data blocks and the files can be realized, the redundancy of data in a backup and archiving storage system in enterprise data storage is greatly reduced, and the management cost is reduced.
2. The calculated fingerprint of each data block and the fingerprint of the whole file are routed to the corresponding target storage server, and the data are respectively distributed and stored on different storage servers, so that load balance is realized.
Drawings
In order to more clearly illustrate the embodiments or the prior art solutions of the present invention, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments described in the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
FIG. 1 is a schematic block diagram of a distributed fingerprint-based data deduplication method according to an embodiment of the present invention;
FIG. 2 is a functional block diagram of an embodiment of the method of FIG. 1 for slicing the file into at least one data block;
FIG. 3 is a schematic block diagram of one embodiment of a method for obtaining a consistent hash routing table maintained by the storage server node of FIG. 1;
FIG. 4 is a schematic structural diagram illustrating an embodiment of a specific construction process of the consistent hash storage algorithm in FIG. 1;
FIG. 5 is a consistent hash routing table maintained as stored in node N1 of FIG. 4;
FIG. 6 is a block diagram illustrating a process for node N1 to find node N53 using the consistent hash routing table of FIG. 5;
FIG. 7 is a functional block diagram of one embodiment of a method of performing file management by a target storage server of FIG. 1;
FIG. 8 is a block diagram of a distributed fingerprint-based data deduplication system according to an embodiment of the present invention;
FIG. 9 is a block diagram of an embodiment of the cloud data processing server of FIG. 8;
FIG. 10 is a block diagram of one embodiment of the cloud data processing server of FIG. 8, preferably a proxy server, slicing a received file into data chunks and computing a fingerprint for each data chunk;
fig. 11 is a schematic diagram illustrating the structure and content of an embodiment of constructing a description information file of data blocks and files for abc _ file when the cloud data processing server in fig. 8 is preferably a proxy server;
fig. 12 is a schematic diagram illustrating the structure and content of an embodiment of the proxy server in fig. 11 constructing a description information file for the first data block "blocks _ 1".
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail and completely with reference to the following embodiments and accompanying drawings. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiments of the present invention are described in a progressive manner, and the same and similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
As shown in fig. 1, an embodiment of the present invention provides a fingerprint-based distributed data deduplication method, including:
s1, initiating a file execution management request;
s2, slicing the received file into at least one data block;
s3, calculating the fingerprint of each data block and the fingerprint of the whole file by using a consistent Hash storage algorithm, and respectively routing each data block and the whole file to the corresponding target storage server;
and S4, constructing the description information of each data block and the whole file, and performing file management through the target storage server.
The patent manages company data in parallel by constructing a data storage rule by utilizing rules on a plurality of data servers, and the mechanism mainly comprises three main stages:
in the initial stage, a data storage system is constructed, and the system is constructed through a consistent Hash storage algorithm model.
And in the second stage, dividing the data of the file into a plurality of data blocks, calculating 64 bits (including but not limited to 128 bits, 256 bits, 1024 bits and the like) of a hash value of each data block by using a consistent hash algorithm, wherein the hash value is a data block fingerprint, and as can be understood, routing the data block to a corresponding storage server by taking the data block fingerprint as a key.
At the final stage, the storage server performs a file management process. When saving data of a data block and a file, creating a file name with a fingerprint value as the data block, simultaneously creating a corresponding description file comprising the fingerprint value and target storage server information, saving the reference count of the data block in the description file, reducing the reference count of the data block when deleting the data, and deleting the data block and the description file if the reference count is 0.
Because the fingerprints of the same data are the same, the repeated data blocks and the repeated files are deleted according to whether the fingerprints of the data of the files and the data blocks needing to be executed and managed are consistent with the fingerprints originally stored on the target storage server or not, namely the same data does not need to be stored repeatedly, and therefore data deduplication can be achieved. Meanwhile, through the mechanism, data can be distributed and stored in different storage servers, so that load balancing is realized.
To achieve efficient data deduplication, the computation and I/O throughput required for data deduplication may be distributed to different nodes in the storage cluster to improve performance, thereby utilizing the computation and storage capabilities of many nodes in the cloud storage to address the limitation of data deduplication.
In view of this, as shown in fig. 2, in an embodiment of the present invention, the method for slicing the file into at least one data block in step S2 includes:
s21, receiving the file transmitted by data stream;
the data stream is divided into at least one data block of a fixed size according to its size S22.
In the specific embodiment provided by the present invention, the step S3 is a method for calculating the fingerprint of each data block and the fingerprint of the entire file by using a consistent hash storage algorithm, and the method includes:
s31, calculating the fingerprint of each data block by performing hash operation on each data block;
and S32, calculating the fingerprint of the file according to the fingerprint of each data block.
In this embodiment, the preferred hash value space is 2 160 Then the specific fingerprint information is binary value data with a length of 160 bits, for example, the following data is fingerprint information "80 b281b067b893854f3f279f901254750b 604576" of a certain block of data. For convenience of description, the former 20 bits "80 b 281" will be used in this document to describe the fingerprint information of this patent. When the fingerprint information length is long enough, the possibility of data repetition can be reduced, and the scheme considers that if the fingerprint information of two data blocks is the same, the two data blocks are the same. And SHA-1 is used as a hash function to calculate the fingerprint of each data block, and the fingerprint of the whole file is calculated according to the calculated fingerprint of each data block and the function, which can refer to the specific contents of hash calculation in the prior art and is not described herein again.
Referring again to fig. 2, in an embodiment of the present invention, the step S3 is a method for routing the fingerprint of each data chunk and the fingerprint of the entire file to the corresponding target storage server, including:
s33, determining a target value key according to the fingerprint of each data block and the fingerprint of the whole file;
s34, each storage server node maintains a consistent Hash routing table, and carries out routing algorithm according to the determined target value key and the consistent Hash routing table of the storage server to find out the direct successor of the target value key and the information of the target storage server corresponding to the direct successor;
s35, determining the target storage server according to the target storage server information;
wherein, the direct successor of the target value key is the storage server node which is closest to the target value key and is smaller than the target value key.
As shown in fig. 3, in an embodiment provided by the present invention, the method for obtaining the consistent hash routing table in step S34 includes:
s340a, respectively carrying out hash operation on each storage server to obtain the fingerprint information of each server;
s340b, taking the fingerprint information of each server as a storage value key;
s340c, establishing and storing the corresponding relation between each storage server node, successor of the storage value key and the information of the storage server;
and S340d, obtaining a consistent Hash routing table preset in the storage server.
The method comprises the steps of performing Hash operation on the distribution of each storage server of the cloud storage system to obtain a consistent Hash routing table which is preset in each server and maintained by using a server node, and using the consistent Hash routing table as a routing model in a consistent Hash storage algorithm.
Referring to fig. 4, a specific construction process of the consistent hash storage algorithm in the specific embodiment of the present invention includes the following steps:
setting the hash value space into a hash ring connected end to end;
and mapping the storage server to the same hash value space to which the data block and the file are mapped, and using the same hash algorithm to ensure consistent hash and non-repeatability of the hash.
Specifically, the consistent hash algorithm ensures consistent hash by mapping the network Node of the storage server and the value keys of the data blocks and files to the same space; the fingerprint information of the data block is that the repeatability of the data block and the file subjected to the Hash operation is related to the number of bits of the Hash value, and the greater the number of bits of the Hash value, the lower the possibility of the data block and the file subjected to the Hash operation is, and the Hash value space is 2 160 In this embodiment, since each hash entry is a large integer of 160 bits, the probability of data duplication is substantially 0, and therefore, in order to ensure accurate deduplication of data blocks and files, the hash value space is preferably 2 160 And using SHA-1 as hash function, specifically including the following:
the integers are arranged according to the size clockwise to form a head (0) and a tail (2) 160 -1) contiguous hash rings;
the storage server nodes and the value keys are all hashed to the hash ring, that is, assuming that the state of the whole storage space of all the storage servers is set as a virtual ring, and each network Node includes a hash routing table of the correspondence between the IP address and the port information of the machine of the storage server and the successor of the value Key, it can be understood that all the storage servers on the ring are respectively operated by hash routing tables maintained by different network nodes.
In the preferred embodiment, 2 160 Each Node on the hash ring of (1) is an identifier, if a certain Node is mapped to a certain identifier, the identifier is continuously called as a Node, the identifier is used as a reference, the Node in front of the Node is called as a previous succession, and the Node behind the Node is called as a subsequent succession.
Due to 2 160 The hash value space of (2) is too large, for easy understanding of the scheme 6 The hash value space of (2) further describes the process, as shown by the consistent hash path on node N1 in FIG. 5The hash space represented by a table, i.e. in this table, is 2 6 To further illustrate the principles of the consistent hash routing algorithm: as shown in fig. 4 and fig. 6, the square on the hash ring is denoted as Node, the dot is denoted as identifier (value Key), and the following nodes in the routing table are illustrated by taking Node N1 as an example again in conjunction with the table of fig. 5, so that consistent hash can be guaranteed by mapping both Node and Key to a value range.
It is clear that as shown in fig. 4, usually only one hash loop in the clockwise direction can be found with certainty, since N is the number of network nodes, for a hash network with millions of network nodes, and in which network nodes join and leave frequently, the time complexity o (n) is intolerable, therefore, the embodiment of the invention presets and maintains a consistent Hash routing table for the corresponding network node of each storage server, and takes the length m of the routing table as the bit number of the Hash space, determining a Hash target value key through the fingerprint of the file or each data block, and finding out target storage server information corresponding to each data block and target storage server information of the file through the target value key and routing tables maintained by different network nodes on different storage servers to realize that each data block and the file are distributed to different target storage servers; in this reference example, the following direct subsequent method of non-linearly finding the target value key is specifically adopted:
1. the network node of each storage server in fig. 4 maintains a consistent hash routing table shown in fig. 5, where the length of the table is m (m is 6), where m is the number of bits in the hash space (i.e., the number of storage servers), and the ith entry of the table stores the (n + 2) th entry of the network node n i-1 )mod 2 m Successor (1)<＝i<＝m)。
2. Each network node maintains a successor list and a successor list, which serve to quickly locate the successor and successor of the value key.
3. Successors stored in the table are increased progressively according to the multiple equal ratio of 2, and the next node of the largest node is defined as a first node;
4. along the hash ring, the first Node of hash (Node) > hash (Key) is the successor to this Key.
5. Given a target value Key, look for the direct successor of the Key.
If the search is performed on node n, further description includes the following:
a. finding out the successor of n which is closest to hash (Key) and is less than hash (Key) in the routing table A of the node n, taking the successor node of the found n as the predecessor which is closest to the target value Key in the routing table B corresponding to the successor node, generating a request for continuously searching the routing table B on the successor node of the n, and forwarding the searching request to the successor node of the n;
b. in the routing table B of the successor node to n, the above lookup process … … continues.
c. And (4) until whether the hash of the target value Key falls between the node n and the direct successor of the node n is checked, if the hash falls between the node n and the direct successor, the successor of the node n is found.
Referring again to FIG. 5, the length m of the consistent hash routing table is the number of bits 6 of the hash space, which is first (0) and last (2) 6 -1 ═ 63), and if N1 is set to 1, table 2 maintained by it on node N1 in conjunction with fig. 1 can be found: corresponding to N1+2 0 Successor of 2 is stored on node N18 and corresponds to the target storage server for IP and port information stored on node N18, corresponding to N1+2 1 Successors of 3 are stored on node N18 and correspond to the target storage server for IP and port information stored on node N18, corresponding to N1+2 2 The successor of 5, corresponding to N1+2, is stored on node N18 and corresponds to the target storage server for IP and port information stored on node N18 3 The successor of 9, corresponding to N1+2, is stored on node N18 and corresponds to the target storage server for IP and port information stored on node N18 4 The successor of 17, stored on node N18 and corresponding to the target storage server for IP and port information stored on node N18, corresponds to N1+2 5 The successor of 31 is stored on node N45 and corresponds to the target storage server for IP and port information stored on node N45, corresponding to N1+2 6 The successor, 65, is stored on node N1 and corresponds to the target storage server for the IP and port information stored on node N1.
For ease of understanding, FIG. 6, which refers to node N1 for the process of finding node N53, is further described in conjunction with FIG. 5:
first, it is known on the storage server of network node N1 from its locally stored consistent hash routing table (fig. 5): node N45 is smaller than node N53 and is closest to successor node N53, node N1 sends a search request to node N45, and node N45 continues searching according to the received search request sent by node N1 and a consistent hash routing table (not shown) stored at node N45, and finally finds the direct successor of node N53 and the corresponding target storage server information, so as to realize that each data block and the entire file are correspondingly stored on the target storage server.
The searching process is the searching of exponential convergence, particularly similar to the searching of dichotomy, and the convergence speed is high, even if 2 is used 160 The hash value space is searched, successors of n can be found by searching for 80 times, the searching speed is high, and the searching efficiency is high.
As shown in fig. 7, in an embodiment of the present invention, the step S4 is a method for performing file management by a target storage server, including:
s301: constructing description information of the data block according to the fingerprint of the data block and the target storage server information thereof, and constructing description information of the file according to the fingerprint of the file and the target storage server information thereof;
s302: transmitting the description information of the data blocks and the files to a target storage server;
s401: comparing the original data in the target storage server with the description information of the data blocks and the files, and performing file management on the data blocks and the files according to the comparison result.
In this embodiment, the target storage server receives and stores the data block and the entire file, establishes a description request form for the data block and the entire file, determines that yes or no exists in the target storage server, and performs file management according to whether yes or no exists in the target storage server, which specifically refers to fig. 7 again, where the process further includes:
s402: determines that yes or no exists in the target storage server,
s403: when the data is judged to be yes, the data block does not need to be stored any more, and a counter on the storage server is started to add 1 to the reference count of the data;
s404: and if the data is no in the storage server, that is, the data file does not exist, starting the storage of the data file.
The invention can realize non-repeated storage of data through the process, and avoid repeated storage of data for many times.
The embodiment of the invention also constructs the construction process of the description information file when slicing the file, calculating the fingerprint and calculating the route. Further, as the description information file in fig. 11 is expressed by JSON, it can be understood that the information file may also be expressed by XML or other formats, and the present solution will take abc _ file.manifest in fig. 11 as an example to illustrate the process of saving and reading the description information and constructing the original data file, and specifically take abc _ file.manifest in fig. 11 as an example to illustrate the process of saving the file abc _ file.manifest as follows:
1. create an abc _ file.
2. The storage server is found by using a fingerprint file finger value "745 ba" of abc _ file.
3. The proxy server sends abc _ file.manifest to the storage server, requesting the storage server to save the abc _ file.manifest file.
4. Manifest is received by the storage server and the description information file is saved with its fingerprint "745 ba" as the filename.
As shown in fig. 8, an embodiment of the present invention further provides a fingerprint-based distributed data deduplication system, which includes:
the client initiates a file execution management request to the file;
the cloud data processing server is used for slicing the received file into at least one data block, performing a consistent Hash storage algorithm on each data block, calculating the fingerprint of each data block and the fingerprint of the whole file, and routing the fingerprints to corresponding target storage servers;
and the target storage server executes file management on the data blocks and the whole file according to the description information of each data block and the whole file constructed by the proxy server.
In the specific embodiment, the distributed data deduplication system based on the fingerprint is specifically divided into a client, a proxy server and a target storage server according to the functions of the nodes:
(1) the client is preferably a node initiating a request to save a file, which may be a smart phone, a computer terminal or a server.
(2) The client sends the files to be stored to the proxy server, and the proxy server executes the data storage process of the files, including the steps of constructing the file description files to be stored, dividing the files, calculating fingerprints, and sending data blocks and the file description files to the target storage server.
(3) The target storage server saves the file as file description information, and each data block is saved as a data block of the corresponding file and description information thereof.
The client, the proxy server and the storage server may be the same node or different nodes. Because the same data block has the same fingerprint and does not need to be stored repeatedly, the invention can realize data deduplication. Meanwhile, through the mechanism, data can be distributed and stored in different storage servers, so that load balancing is realized.
In this embodiment, the reading and data file assembling process for the description information file abc _ file.
1. The client or the proxy server (hereinafter referred to as a configuration node) finds the storage server by using a fingerprint "745 ba" (refer to fig. 11) as a key through a consistent hash storage algorithm.
2. The configuration node downloads the "745 ba" file through the storage server.
3. The configuration node opens and reads the original data information of '745 ba'.
4. And the configuration node requests a server corresponding to each data block to download the data block according to the '745 ba' metadata information.
5. The configuration node assembles the original data file abc _ file.
6 data files and their fingerprint management.
As shown in fig. 9, in an embodiment of the present invention, the cloud data processing server further includes:
the cutting module is used for receiving and analyzing the data stream of the file and slicing the data stream into at least one data block according to the size of the data stream;
the fingerprint calculation module is used for calculating each data block according to a stored consistent Hash storage algorithm, calculating the fingerprint of each data block and calculating the fingerprint of the file according to the fingerprint of each data block;
the routing module is used for carrying out a routing algorithm according to the fingerprint of each data block and the fingerprint of the file transmitted by the fingerprint calculation module and each consistent Hash routing table stored by each storage server, finding out the direct successor of the target value key and the target storage server information corresponding to the direct successor of the target value key, and determining the target storage server of each data block and the file according to the target storage server information;
and the information construction module is used for constructing the description information of each data block and the file according to the fingerprint of each data block and the information of the target storage server and sending the description information to the corresponding target storage server.
In this embodiment, the cloud data processing server is preferably a proxy server, the client sends the file to the proxy server in a data stream form, the proxy server is configured to receive the file, slice the file into different data blocks, calculate a fingerprint of each data block, calculate a fingerprint of the entire file according to the fingerprint of each data block, send each data block and the entire file to corresponding different target storage servers according to a routing algorithm, and construct a data file description list.
Referring to fig. 10, in the process of slicing a file and calculating a fingerprint, in the embodiment of the present invention, after a storage server receives a data stream of a file to be stored, the file is divided into data blocks of a fixed size, preferably, the size of the data block may be set to 4MB, hash operation is performed on each data block to obtain a fingerprint corresponding to the data block, and referring to fig. 10, after data block i and data block i +1 … data block n are operated, fingerprint i and fingerprint i +1 … fingerprint n are obtained, respectively.
And determining a target value key according to the fingerprint i, finding a storage server of the data block (file) through a routing algorithm, and sending the data block (file) to the found target storage server.
In detail, referring to fig. 11, when the proxy server receives the data file abc _ file, the proxy server constructs an information description file abc _ file for the abc _ file, which has a structure and content shown in fig. 11 and includes the following contents:
1. file name information (name), the data file name.
2. File type information (data _ type), data file type. Preferably binary and text types and their subdivisions, such as MP4, XML, JSON, etc.
3. File length information (file _ size), which indicates the length of the file in bytes.
4. File fingerprint information (file _ finger), fingerprint information of the entire data file, which is a unique identifier of the file in the storage system, and is obtained by hashing the entire data file.
5. Number information (data _ block _ numbers) of file data blocks, and the number of data blocks into which the data file is divided.
The proxy server constructs corresponding information description file information for each data block (block _1, block _2.. block _100, etc. in fig. 11), including the following:
1. fingerprint information (finger): fingerprint information for each data block.
2. Stored server information (server): server information such as server name or IP is stored for each data block.
3. Size information (size) of data block: length information of each data block.
As can be seen from fig. 11: in the embodiment provided by the present invention, the fingerprint information is preferably represented by 160-bit binary, for convenience of description, fig. 11 uses the first 20 bits to represent the fingerprint information, and according to the description information shown in fig. 11, the file is called abc _ file binary file, and includes 100 data blocks, each data block has a length of 4MB, the fingerprint of the first data block is "84 f2 d", and the data block is stored in the storage node "server _ 11" (storage server node).
In another embodiment of the present invention, the target storage server further comprises:
the local data storage module is used for storing an information file of original data, and establishing and storing a data block information description file and a file information description file corresponding to transmission according to the description information of each data block and file transmitted by the information construction module;
and the local data management module compares the data block information description file and the file information description file transmitted by the local data storage module with the information description file of the original data, and executes file management on the data block and the file according to a comparison result.
In this embodiment, the target storage server receives and stores the data block and the entire file, and establishes a description request form for the data block, and determines whether the data exists in the target storage server, if it is determined that the data file exists in the storage server, the data block does not need to be stored, a counter on the storage server is started to add 1 to a reference count of the data, and if it is determined that the data file does not exist, the storage of the data file is started to implement non-repeated storage of the data, thereby avoiding repeated storage of the data.
The process of storing and reading the description information file and assembling the data file comprises the following steps:
referring to fig. 11 and 12, the target storage server creates a data block file and its description information file for each data block, for example, when the target storage server receives the first data block "blocks _ 1" of 00000 in fig. 11, the target storage server creates a "84 f2 d.data" data block file and a "84 f2 d.manifest" file in the local file system of the target storage server, where "84 f2 d" is the fingerprint of the data block, and the content of the 84f2d.manifest is as shown in fig. 12, if the local file system already exists 84f2d.manifest, it is not necessary to create two files, namely, the "84 f2 d.data" data block file and the "84 f2 d.manifest", and only needs to increase the reference count reference of the 84f2 d.manifest.
Further, in the embodiment of the present invention, the following references are made to the file (or data) deletion process:
1. the client initiates a delete file request to the proxy server.
2. The proxy server downloads a data description list of the file to be deleted, finds out all data blocks and files listed in the list and storage server information corresponding to the data blocks and files, and initiates a request for deleting the data blocks to the storage server.
3. After receiving a data block deletion request from the proxy server, the storage server opens the description list of the data block, subtracts 1 from the reference count of the description list, and deletes the data block and the description information if the reference count is already 0.
The present invention also provides a computer readable medium having stored thereon a computer program which, when executed by a processor, implements a fingerprint based distributed data deduplication method as described in any of the first aspects above.
The present invention also provides an electronic device, comprising: a processor, a memory, and a bus; the memory is configured to store a computer program, the processor is connected to the memory through the bus, and when the electronic device is running, the processor executes the computer program of the memory, so as to enable the processor to execute the fingerprint-based distributed data deduplication method according to any one of the first aspect.
The embodiment provided by the invention has the following beneficial effects:
1. the fingerprints of the files and each data block obtained by slicing are obtained through calculation, and because the fingerprints of the same data block and the same file are the same, the target storage server does not need to repeatedly store the data blocks and the files with the same fingerprints, duplicate removal of the data blocks and the files can be realized, the redundancy of data in a backup and filing storage system in enterprise data storage is greatly reduced, and the management cost is reduced.
2. The calculated fingerprint of each data block and the fingerprint of the whole file are routed to the corresponding target storage server, and the data are respectively distributed and stored on different storage servers, so that load balance is realized.
The above description is only an example of the present invention, and is not intended to limit the present invention. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.
Claims (10)
1. A distributed data deduplication method based on fingerprints is characterized by comprising the following steps:
initiating a file execution management request;
slicing the received file into at least one data block;
calculating the fingerprint of each data block and the fingerprint of the whole file by using a consistent Hash storage algorithm, and respectively routing each data block and the whole file to corresponding target storage servers;
and constructing the description information of each data block and the whole file, and executing file management through the target storage server.
2. The fingerprint-based distributed data deduplication method of claim 1, wherein the method of computing the fingerprint of each data chunk and the fingerprint of the entire file by using a consistent hash storage algorithm comprises:
performing hash operation on each data block respectively to obtain the fingerprint of each data block;
and calculating the fingerprint of the file according to the fingerprint of each data block.
3. The fingerprint-based distributed data deduplication method of claim 2, wherein the method of routing the fingerprint of each data chunk and the fingerprint of the entire file to the corresponding target storage server comprises:
taking the fingerprint of each data block and the fingerprint of the whole file as target values key;
each storage server node maintains a consistent Hash routing table, and performs routing operation through the consistent Hash routing table maintained by the storage server node according to the determined target value key to find out the direct successor of the target value key and the target storage server information corresponding to the direct successor;
determining a target storage server according to the target storage server information;
wherein, the direct successor of the target value key is the storage server node which is closest to the target value key and is smaller than the target value key.
4. The fingerprint-based distributed data deduplication method of claim 3, wherein the method of obtaining the consistent hash routing table comprises:
respectively carrying out Hash operation on each storage server to obtain fingerprint information of each server;
taking the fingerprint information of each server as a storage value key;
and establishing and storing a corresponding relation among each storage server node, successors of the storage value keys and the information of the storage servers to obtain a consistent Hash routing table preset in the storage servers.
5. The fingerprint-based distributed data deduplication method of claim 1, wherein the method of slicing the file into at least one data block comprises:
receiving the file transmitted in a data stream;
the data stream is partitioned into at least one data block of a fixed size according to its size.
6. The fingerprint-based distributed data deduplication method of claim 2, wherein the method of performing file management by the target storage server comprises:
constructing description information of the data block according to the fingerprint of the data block and target storage server information thereof, and constructing description information of the file according to the fingerprint of the file and the target storage server information thereof;
transmitting the description information of the data block and the file to a target storage server;
comparing the original data in the target storage server with the description information of the data blocks and the files, and executing file management on the data blocks and the files according to the comparison result.
7. A fingerprint-based distributed data deduplication system, the system comprising:
the client initiates a file execution management request to the file;
the cloud data processing server is used for slicing the received file into at least one data block, performing a consistent Hash storage algorithm on each data block, calculating the fingerprint of each data block and the fingerprint of the whole file, and routing the fingerprints to corresponding target storage servers;
and the target storage server executes file management on each data block and the whole file according to the description information of each data block and the whole file constructed by the cloud data processing server.
8. The fingerprint-based distributed data deduplication system of claim 7, wherein the cloud data processing server comprises:
the cutting module is used for receiving and analyzing the data stream of the file and slicing the data stream into at least one data block according to the size of the data stream;
the fingerprint calculation module is used for calculating each data block according to a stored consistent Hash storage algorithm, calculating the fingerprint of each data block, and calculating the fingerprint of the file according to the fingerprint of each data block;
the routing module is used for carrying out routing operation through a consistent Hash routing table maintained by each storage server according to the fingerprint of each data block and the fingerprint of the file transmitted by the fingerprint calculation module, finding out the direct successor of the target value key and the target storage server information corresponding to the target value key, and determining the target storage server of each data block and the file according to the target storage server information;
and the information construction module is used for constructing the description information of each data block and the file according to the fingerprint of each data block and the information of the target storage server and sending the description information to the corresponding target storage server.
9. The fingerprint-based distributed data deduplication system of claim 7 or 8, wherein the target storage server comprises;
the local data storage module is used for storing an information file of original data, and establishing and storing a data block information description file and a file information description file corresponding to transmission according to the description information of each data block and file transmitted by the information construction module;
and the local data management module is used for comparing the data block information description file and the file information description file transmitted by the local data storage module with the information description file of the original data and executing file management on the data block and the file according to a comparison result.
10. An electronic device, comprising: a processor, a memory, and a bus; the memory is used for storing a computer program, the processor is connected with the memory through the bus, and when the electronic device runs, the processor executes the computer program of the memory to enable the processor to execute the fingerprint-based distributed data deduplication method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210364761.XA CN114880297A (en) | 2022-04-07 | 2022-04-07 | Distributed data deduplication method and system based on fingerprints |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210364761.XA CN114880297A (en) | 2022-04-07 | 2022-04-07 | Distributed data deduplication method and system based on fingerprints |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114880297A true CN114880297A (en) | 2022-08-09 |
Family
ID=82669222
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210364761.XA Pending CN114880297A (en) | 2022-04-07 | 2022-04-07 | Distributed data deduplication method and system based on fingerprints |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114880297A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US12056093B2 (en) * | 2022-10-25 | 2024-08-06 | Dell Products L.P. | Deduplication for cloud storage operations |
-
2022
- 2022-04-07 CN CN202210364761.XA patent/CN114880297A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US12056093B2 (en) * | 2022-10-25 | 2024-08-06 | Dell Products L.P. | Deduplication for cloud storage operations |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6094487B2 (en) | Information system, management apparatus, data processing method, data structure, program, and recording medium | |
| US20050108368A1 (en) | Method and apparatus for representing data available in a peer-to-peer network using bloom-filters | |
| JP4263477B2 (en) | System for identifying common digital sequences | |
| JP4846156B2 (en) | Hash file system and method for use in a commonality factoring system | |
| US9935919B2 (en) | Directory partitioned system and method | |
| US20140244794A1 (en) | Information System, Method and Program for Managing the Same, Method and Program for Processing Data, and Data Structure | |
| US10339124B2 (en) | Data fingerprint strengthening | |
| JP2009295127A (en) | Access method, access device and distributed data management system | |
| EP4231167A1 (en) | Data storage method and apparatus based on blockchain network | |
| CN116578746A (en) | Object de-duplication method and device | |
| US20180107404A1 (en) | Garbage collection system and process | |
| CN117176796A (en) | Message pushing method, device, computer equipment and storage medium | |
| WO2020024446A1 (en) | Data storage method and apparatus, storage medium, and computer device | |
| CN114880297A (en) | Distributed data deduplication method and system based on fingerprints | |
| US9020977B1 (en) | Managing multiprotocol directories | |
| TWI420333B (en) | A distributed de-duplication system and the method therefore | |
| US20240015135A1 (en) | Domain management and synchronization system | |
| Guirat et al. | An efficient data replication approach for structured peer-to-peer systems | |
| EP4394619A1 (en) | Data processing method and apparatus based on blockchain, and device and readable storage medium | |
| Kniesburges et al. | Hashed Patricia Trie: Efficient longest prefix matching in peer-to-peer systems | |
| US9319245B2 (en) | Information processing device, recording medium storing information search program, and information search method | |
| CN115129779A (en) | Database synchronization method, device and readable medium | |
| JP2017123040A (en) | Server device, distribution file system, distribution file system control method, and program | |
| Xu et al. | Two-side data deduplication mechanism for non-center cloud storage systems | |
| CN116756177B (en) | Multi-table index maintenance method and system for mysql database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |