CN117539389A - Cloud edge end longitudinal fusion deduplication storage system, method, equipment and medium - Google Patents

Abstract

The application relates to a cloud edge end longitudinal fusion deduplication storage system, a cloud edge end longitudinal fusion deduplication storage method, equipment and a medium. The edge layer reduces bandwidth resource overhead of the backbone network by storing hotter data blocks. The uploaded metadata can also be pre-de-duplicated at the edge layer to further reduce the data transmission quantity; the cloud layer maintains a global fingerprint index table for global data deduplication and supports distributed parallel indexing across cloud storage servers by partitioning the global fingerprint index table and incoming metadata to different servers of the cloud data center, thereby globally improving data deduplication and storage performance. The end-side-cloud longitudinal fusion deduplication storage architecture remarkably reduces the bidirectional data transmission quantity between layers while ensuring the optimal data deduplication performance, and achieves the effect of greatly reducing the resource overhead of a backbone network of the cloud side end architecture.

Description

Deduplicated storage systems, methods, equipment and media for vertical integration of cloud, edge and end

Technical field

The invention belongs to the field of data processing technology and relates to a deduplication storage system, method, equipment and medium for vertical integration of cloud, edge and terminal.

Background technique

As the volume and value of digital information continues to grow, it has attracted widespread attention in the industry to data protection. Cloud backup services can provide a cost-effective, efficient, on-demand and always-available option for data protection by saving continuous backup files of customers' important data. According to existing reports, the number of organizations adopting cloud data protection is rising rapidly. However, transferring these original files from the endpoint to the remote cloud server will bring a large data transmission burden to the backbone network and also generate considerable costs. Transmission costs.

Under this trend, applying data deduplication technology in backup storage systems has become a new paradigm. Due to the internal derivation relationship between these consecutive backup files, there may be a large amount of non-negligible redundant data between the files. A large study reports that the data redundancy of file system contents on desktop Windows machines can be as high as 87% of its original storage space. A common practice in data deduplication technology is to split the backup file into data blocks and calculate a fingerprint for each data block. Two data blocks with the same fingerprint are considered duplicates without the need for byte-by-byte comparison of the data blocks. Only one copy of multiple duplicate data blocks is retained, thereby saving storage space in the storage system.

There are already some emerging data protection strategies that adopt data deduplication technology, such as performing data deduplication operations at cloud providers, trying to allocate similar files to the same storage server to improve the data deduplication rate, etc. However, the aforementioned traditional data deduplication technology still has the technical problem of high backbone network resource overhead in the deduplication and storage process of cloud-edge architecture.

Contents of the invention

In view of the problems existing in the above traditional methods, the present invention proposes a deduplication storage system with vertical integration of cloud and edge, a deduplication storage method with vertical integration of cloud and edge, a computer device and a computer-readable storage. media, which can significantly reduce the resource overhead of the backbone network in the cloud-edge architecture.

In order to achieve the above objects, the embodiments of the present invention adopt the following technical solutions:

On the one hand, a deduplication storage system with vertical integration of cloud, edge, and end is provided, including the terminal layer, edge layer, and cloud layer. After the terminal device in the terminal layer divides the original backup file into each data block, it generates unprocessed data corresponding to each data block. The processed files are sequenced and uploaded to the edge layer;

In the edge layer, the edge server hashes the fingerprint information contained in the unprocessed file sequence into a fixed-size compact sketch data structure to estimate the data block heat, allocates edge storage locations for the estimated new hot data blocks, and then stores the edge The storage address of the location is recorded in the unprocessed file sequence and an edge upload tag is attached;

The edge server in the edge layer matches the entries in the unprocessed file sequence with the entries in the set partial index table, and copies the storage locations corresponding to the data blocks with matching fingerprints from the set partial index table to the unprocessed files. After the spectrum sequence, upload the partial unprocessed file sequence corresponding to the data block without matching fingerprint to the cloud layer; set the partial index table to the partial index table that the user perceives and is adjacent to the version;

The cloud server in the cloud layer performs redundancy detection on the uploaded part of the unprocessed file sequence with the global fingerprint index table maintained in the cloud, and appends an entry in the unprocessed file sequence corresponding to the unrecognized unstored data block. After the cloud uploads the tag and allocates a new cloud storage location corresponding to the unstored data block, the processed file sequence is returned to the edge layer;

The edge server of the edge layer assembles the processed file sequence and the partially unprocessed file sequence corresponding to the data block with matching fingerprints into a complete processed file sequence, and returns the processed file sequence to the terminal layer;

The terminal device at the terminal layer uploads new hot data blocks to the edge layer for storage based on the edge storage location and edge upload tag in the processed file sequence, and based on the new cloud storage location and cloud upload in the processed file sequence The tag uploads unstored data blocks to the cloud for storage.

On the other hand, it also provides a cloud-edge-device vertical integration deduplication storage method, which includes the following steps:

Hash the fingerprint information contained in the unprocessed file sequence uploaded by the terminal layer into a fixed-size compact sketch data structure to estimate the data block heat, and allocate edge storage locations to the estimated new hot data blocks. The storage address is recorded in the unprocessed file sequence and an edge upload tag is attached;

Match the entries in the unprocessed file sequence with the entries in the set partial index table. After copying the storage location corresponding to the data block with matching fingerprint from the set partial index table to the unprocessed file sequence, Part of the unprocessed file sequence corresponding to the data block that does not match the fingerprint is uploaded to the cloud; the partial index table is set to the partial index table that the user perceives and is adjacent to the version;

Assemble the processed file sequence and the partially unprocessed file sequence corresponding to the data block with matching fingerprint into a complete processed file sequence, and return the processed file sequence to the terminal layer; where, the processed file sequence The file sequence uses the cloud server in the cloud layer to perform redundancy detection on the uploaded part of the unprocessed file sequence and the global fingerprint index table maintained in the cloud, and checks the unprocessed file sequence corresponding to the unidentified unstored data block. The entry is obtained and returned after appending a cloud upload tag and assigning a new cloud storage location corresponding to the unstored data block;

The terminal device at the receiving terminal layer uploads and stores new hot data blocks based on the edge storage location and edge upload tag in the processed file sequence; the new cloud storage location and cloud upload tag in the processed file sequence are also used Instruct the terminal device at the terminal layer to upload the unstored data blocks to the cloud layer for storage.

On another aspect, a computer device is also provided, including a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements the above steps of the cloud-edge-device vertical integration deduplication storage method.

On the other hand, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned cloud-edge-device vertical integration deduplication storage method are implemented.

One of the above technical solutions has the following advantages and beneficial effects:

The above-mentioned cloud-edge-end vertical integration deduplication storage system, method, device and media are responsible for dividing files into blocks through the terminal layer and uploading the generated metadata information to the edge layer and cloud layer for redundancy detection. The edge layer reduces the bandwidth resource overhead of the backbone network by storing hotter data blocks. In addition, the uploaded metadata can also be pre-deduplicated at the edge layer to further reduce the amount of data transmission; the cloud layer maintains a global fingerprint index table for global data deduplication and combines the global fingerprint index table with the incoming metadata. Divide it into different servers in the cloud data center to support distributed parallel indexing across cloud storage servers, thereby globally improving data deduplication and storage performance. Compared with traditional technology, this end-edge-cloud vertical integration deduplication storage architecture effectively integrates different levels of technology and storage resources in the process of file transmission, storage and retrieval, while ensuring the best data deduplication performance. At the same time, the amount of two-way data transmission between layers is significantly reduced, achieving the effect of significantly reducing the resource overhead of the backbone network of the cloud-edge architecture.

Description of drawings

In order to more clearly explain the technical solutions in the embodiments of the present application or the traditional technology, the drawings needed to be used in the description of the embodiments or the traditional technology will be briefly introduced below. Obviously, the drawings in the following description are only for the purpose of explaining the embodiments or the technical solutions of the traditional technology. For some embodiments of the application, those of ordinary skill in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of the CoopDedup architecture of a cloud-edge-end vertical integration deduplication storage system in one embodiment;

Figure 2 is a schematic diagram of data sharing dependencies between backup files in one embodiment;

Figure 3 is a schematic diagram of the data deduplication rate of backup files in one embodiment;

Figure 4 is a schematic diagram of the overall structure of a deduplication storage system for vertical integration of cloud, edge and end in one embodiment;

Figure 5 is a schematic diagram of the data flow of the CoopDedup architecture in one embodiment;

Figure 6 is a schematic diagram of evaluation performance during backup file uploading in one embodiment;

Figure 7 is a schematic diagram of the data block transmission volume of the backup file in one embodiment;

Figure 8 is a schematic diagram of data transmission performance during backup file retrieval in one embodiment;

Figure 9 is a schematic diagram of bandwidth saving rate performance in one embodiment;

Figure 10 is a schematic flowchart of a deduplication storage method for cloud-edge-end vertical integration in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used in the description of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

It should be noted that reference to "embodiments" herein means that specific features, structures or characteristics described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearances of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments.

Those skilled in the art will appreciate that the embodiments described herein may be combined with other embodiments. As used in this specification and the appended claims, the term "and/or" means and includes any and all possible combinations of one or more of the associated listed items.

Contiguous files in personal computing environments require cloud backup services for data protection. However, transmitting these original files from the terminal to the remote cloud server will bring a greater transmission burden to the backbone network. Even though the backup service can use data deduplication technology to delete redundant data blocks, the limited computing and memory resources of the terminal make it difficult to actually deploy and apply the deduplication technology on the terminal side.

Among the existing technologies in this field, there are also methods that perform data deduplication on the terminal side and only upload new data blocks (that have not been stored) to the cloud. The disadvantage is that the scarce resources of the terminal layer limit the application of data deduplication technology. . The current most advanced deduplication storage architecture uploads the metadata of data blocks (such as fingerprints) to the cloud server for redundancy detection and only uploads new data blocks. However, it still does not fully utilize edge-side storage resources, resulting in terminal Frequent data exchange with remote clouds.

To this end, this application proposes CoopDedup, a deduplication storage architecture with end-edge-cloud vertical integration. This innovative architecture enables the end layer, cloud layer, and intermediate edge layer to collaborate to complete data transmission, storage, and index access. An example of the CoopDedup architecture can be shown in Figure 1. The terminal layer divides the file into data blocks and calculates its metadata for uploading. The cloud layer maintains a global index table and stores all deduplicated data blocks. The innovation of this architecture is mainly focused on the edge layer between the end and the cloud. It pre-deduplicates uploaded metadata (UFR) and identifies the hottest data blocks and stores them at the edge. Based on this, when accessing backup files, these hotter data blocks can be obtained directly from nearby edge layers, thus optimizing the data retrieval process and reducing data block transmission distance. In addition, the edge layer pre-deduplicates the uploaded UFR and only transmits the processed partial metadata (P-UFR) to the cloud, which greatly reduces the amount of metadata transmission and thus saves the bandwidth resources of the backbone network. The CoopDedup architecture fully integrates the storage and computing resources of the three layers of end-edge-cloud, significantly reducing the two-way data exchange between layers, while still ensuring the best data deduplication performance.

The embodiments of the present invention will be described in detail below with reference to the accompanying drawings of the embodiments of the present invention.

First of all, it needs to be explained that data deduplication technology is a widely used data reduction technology, which can delete redundant data blocks and avoid repeated writing of the same data. This not only reduces the storage space occupied by the system, but also reduces the network bandwidth consumption during data transmission and retrieval. A common practice for data deduplication is to split a file into multiple fixed-size or variable-size data blocks. Each block is uniquely identified with a fingerprint, which is essentially a cryptographically secure hash signature, and two data blocks with the same fingerprint are considered duplicates without the need for a byte-by-byte comparison of the data blocks. The fingerprints of all stored data blocks will be recorded in the fingerprint index table. By looking up the fingerprint index table, duplicate data blocks will be directly found and deleted, and only data blocks with unique fingerprints will be stored in the storage system.

Although fingerprint comparison is an efficient redundancy detection method, it also requires a large amount of extra space to store the fingerprint index table. For example, when storing an 800TB file data set, assuming the average block size is 4KB, at least 4TB of fingerprints will be generated (using SHA-1 encoding, each fingerprint is 20B). The transmission and storage of these large-capacity fingerprints is a huge challenge for backup file systems. These impacts will be further aggravated when stored data is frequently retrieved, as these generated fingerprints need to be transmitted back and forth between the user and the cloud for duplicate data block comparisons, placing a huge communication burden on the backbone network. But compared with the direct transmission of data blocks, it is already a relatively better choice.

It should be noted that assuming that the cloud storage cluster consists of a large number of storage servers, the resources reserved for the cloud backup business are relatively sufficient. The cloud layer can therefore store a global fingerprint index table (containing the fingerprints of all data blocks stored in the cloud storage cluster) for thorough redundancy detection. These stored data blocks should also adopt some fault-tolerant mechanisms, such as replicas and erasure coding, to ensure the reliability of the data stored in the cloud.

Reducing storage space can significantly reduce data protection costs. Therefore, data deduplication technology is widely used in backup storage systems to reduce storage space usage. In addition to storage costs, bandwidth consumption from the end side of data generation to the cloud layer is also a concern. Because the network throughput between layers is limited, large amounts of data transmission will occupy a large amount of bandwidth resources and cause delays in other applications. Additionally, the cost of transferring data once may exceed the monthly cost of storing the data.

To this end, placing some data blocks at the edge of the network is an innovative and effective practical attempt. Because edge resources are mostly located close to users, when accessing backup files, relevant data blocks can be retrieved from the edge layer, while only data blocks that do not exist in the edge layer are downloaded from the remote cloud data center. In addition, this application can also use edge resources to pre-deduplicate the transmitted data, and only transmit the pre-deduplicated data to the cloud layer. In this way, these edge-assisted backup service models can effectively save valuable bandwidth resources on the backbone network.

However, the storage space and computing resources of edge clusters are limited, making it difficult to cope with the huge storage and computing demands brought about by the explosive growth of files. For example, in a deduplication data storage scenario, as files continue to arrive, the number of stored data blocks continues to grow, and the corresponding fingerprint index table will continue to grow in size. Therefore, how to effectively utilize precious resources on edge clusters to place partial data blocks and perform data pre-deduplication is an urgent technical issue that needs to be solved.

Observation on backup file characteristics: Important digital information generally has a series of backup versions. For example, users regularly take snapshots of their virtual machines, where each snapshot corresponds to a backup file. In these contiguous files, most of the data blocks remain unchanged. This article also provides some systematic observations based on backup files, which can help to efficiently utilize limited storage and computing resources for data deduplication operations.

Observation 1: Due to the diversity of data sources, the number of duplicate data blocks between backup files of different users is negligible.

In the era of big data, different users may back up files with different contents and formats. This paper observes the data sharing dependencies between these files. The test data set is a snapshot of college students' home directories commonly used in deduplication systems. In order to discover the amount of data redundancy within and between different users, this paper divides these data sets into variable-length data blocks with an average size of 4KB, and records the duplication of the respective data blocks.

This article records the intra-user and inter-user deduplication rates. The experimental results show that this test contains five consecutive backup files of four users. The intra-user deduplication rate has always remained above 41%, even for individual users. The intra-user deduplication rate is as high as 55.02%. This indicates that there are a large number of data blocks maintained in multiple consecutive backup files of a user. In comparison, inter-user deduplication rates are typically around 2%, which is insignificant compared to intra-user deduplication rates. This also confirms the observation of this paper that data redundancy among different users is negligible due to the diversity of data sources.

Empirical observations show that the data redundancy rate in one user's consecutive backup files is very high, while the amount of data block duplication between backup files from different users is relatively small or even negligible. This phenomenon prompted this paper to study the differences between different users. This article can divide the global fingerprint index table into several independent sub-index tables according to user sources, and index the backup files based on the specific sub-index tables associated with their corresponding users. The division of this index table facilitates individual index management for each user; in addition, this index table speeds up fingerprint indexing and avoids index search bottlenecks, while the parallelism of the index can improve index throughput. The most important point is that this user-aware index partitioning takes up very little additional index memory space while maintaining the deduplication effect, because the number of duplicate data blocks between different users' backup files is negligible.

Observation 2: Most of the duplicate blocks of a backup file come from its previous adjacent backup versions, while the two more distant backup versions contain only a small number of duplicate blocks.

In order to observe the internal data sharing dependencies between consecutive backup files and detect their data block composition, this paper performs data slicing on 30 consecutive backup versions of a user. These data blocks are identified as duplicates when their fingerprints are detected from previous versions of the backup file. The four types of blocks in the backup file B _j are expressed as follows: One is the internal repeated data block. The second is adjacent duplicate data blocks: the data block of B _j is also referenced by the adjacent version B _j-1 . The third is skipping duplicate data blocks: the data block of B _j is referenced by its backup version before B _j-1 , such as B _j-2 , but is not referenced by the adjacent B _j-1 . The fourth is the unique data block: a non-duplicate data block.

Figures 2 and 3 illustrate the data sharing dependencies between 30 consecutive backup files from one user. First, Figure 2 shows the data block distribution of these backup files. It can be observed that most of the duplicate data blocks of the backup file come from its previous version (i.e., adjacent duplicate data blocks), and they account for 55% of the entire backup file data volume. about. Skipping duplicate data blocks only contain a small portion, not even 0.3% of all data blocks. Figure 3 shows the data deduplication rate for the current 30th backup file based on the previous 29 versions of the fingerprint index table. Observations show that the data deduplication rate gradually increases from approximately 42% (based on the initial backup version 1) to over 65% (based on the adjacent version 29 of the current backup).

The results of these observations verify the data sharing dependencies and version derivation relationships between successive backup files of a user. If backup versions are closer, more duplicate data blocks can be detected from each other, whereas fingerprinting indexes with more distant versions will weaken the effectiveness of redundancy detection. This makes this article focus more on adjacent versions of backup files when the memory space is insufficient to save the fingerprints of all previous backup versions.

Referring to Figure 4, in one embodiment, a deduplication storage system with vertical integration of cloud, edge and end is provided, including a terminal layer 12, an edge layer 14 and a cloud layer 16. After dividing the original backup file into data blocks, the terminal device of the terminal layer 12 generates an unprocessed file sequence corresponding to each data block and uploads it to the edge layer 14 . In edge layer 14, the edge server hashes the fingerprint information contained in the unprocessed file sequence into a fixed-size compact sketch data structure to estimate the data block heat, allocates edge storage locations for the estimated new hot data blocks, and then The storage address of the storage location is recorded in the unprocessed file sequence and an edge upload tag is attached. The edge server in edge layer 14 matches the entries in the unprocessed file sequence with the entries in the set part index table, and copies the storage locations corresponding to the data blocks with matching fingerprints from the set part index table to the unprocessed After the file is sequenced, the partially unprocessed file sequence corresponding to the data block that does not match the fingerprint is uploaded to the cloud layer 16; the partial index table is set to a partial index table that is perceived by the user and adjacent to the version. The cloud server in cloud layer 16 performs redundancy detection on the uploaded part of the unprocessed file sequence and the global fingerprint index table maintained by the cloud, and appends entries in the unprocessed file sequence corresponding to the unrecognized unstored data blocks. After a cloud uploads the tag and allocates a new cloud storage location corresponding to the unstored data block, it returns the processed file sequence to the edge layer 14. The edge server of the edge layer 14 assembles the processed file spectrum sequence and the partially unprocessed file spectrum sequence corresponding to the data block with matching fingerprints into a complete processed file spectrum sequence, and returns the processed file spectrum sequence to the terminal layer 12 sequence. The terminal device of the terminal layer 12 uploads the new hot data block to the edge layer 14 for storage according to the edge storage location and edge upload tag in the processed file sequence, and based on the new cloud storage location and edge upload tag in the processed file sequence. The cloud upload tag uploads unstored data blocks to cloud layer 16 for storage.

The above-mentioned cloud-edge-end vertical integration deduplication storage system 100 is responsible for dividing files into blocks through the terminal layer 12 and uploading the generated metadata information to the edge layer 14 and the cloud layer 16 for redundancy detection. The edge layer 14 reduces the bandwidth resource overhead of the backbone network by storing hotter data blocks. In addition, the uploaded metadata can also be pre-deduplicated at the edge layer 14 to further reduce the amount of data transmission; the cloud layer 16 maintains a global fingerprint index table for global data deduplication and combines the global fingerprint index table with the incoming Metadata is divided into different servers in the cloud data center to support distributed parallel indexing across cloud storage servers, thereby globally improving data deduplication and storage performance. Compared with traditional technology, this end-edge-cloud vertical integration deduplication storage architecture effectively integrates different levels of technology and storage resources in the process of file transmission, storage and retrieval, while ensuring the best data deduplication performance. At the same time, the amount of two-way data transmission between layers is significantly reduced, achieving the effect of significantly reducing the resource overhead of the backbone network of the cloud-edge architecture.

It is understandable that the CoopDedup architecture has several problems that need to be solved in the deduplication storage service. The first key question is how to reasonably estimate the data block access frequency and store the most frequently accessed blocks (hot blocks) in the edge layer 14? Due to the limited storage and computing resources of the edge layer 14, it is unrealistic to record the access frequency of each data block. The second key question is how to effectively utilize the limited memory space at the edge to detect more redundant metadata? Maintaining a global fingerprint index table at the edge for redundancy detection poses a serious challenge to memory-scarce edge servers. Even if the edge server inserts large storage disks to assist in maintaining this large-capacity index table, this slow disk index will still become the main performance bottleneck of the deduplication system.

For the first problem, this article uses the space-friendly Count-Min Sketch (an algorithm that can be used for counting. When the data size is very large, the efficiency can be improved by sacrificing accuracy) to estimate the access heat of the data block. This size Fixed data structures identify hotter data blocks with higher accuracy. For the second problem, this paper uses the backup file derivation relationship to design an efficient lightweight index table to pre-deduplicate the uploaded metadata. Most of the metadata redundancy can be detected and deleted. These edge-assisted methods effectively save valuable bandwidth resources on the backbone network from the edge to the remote cloud.

Some concepts that need to be explained first: metadata structure, fingerprint is the unique identifier of the data block. By comparing fingerprints, it can be determined whether two data blocks are duplicates. During file storage or retrieval, these fingerprints are usually combined into a file sequence in sequence. File sequence is essentially a sequential list of metadata for each data block in the file, reflecting the order in which these data blocks appear in the file. Even if a data block appears multiple times in the file, its corresponding metadata will still be listed multiple times in the corresponding file sequence. There are two types of file sequences in the CoopDedup architecture of this article: Unprocessed File recipes (UFR) and Processed File recipes (PFR).

Among them, the unprocessed file sequence (UFR) is generated when the terminal layer 12 divides the backup file, and the metadata in it only contains the fingerprint of the data block. UFR records the data block composition of the backup file and serves as input to the subsequent data deduplication process. Once the storage location of any data block is determined at the edge layer 14 or cloud layer 16, the UFR is converted into a processed file sequence (PFR), where each entry further adds the storage address of the corresponding data block.

PFR has two roles in backup protection. The first role is that during the backup file upload process, data blocks can be uploaded according to the storage address recorded on the PFR. To avoid duplication of data blocks that have already been stored, each entry will have an additional Uploading tag attached to the unique block that needs to be uploaded. Only data blocks with marked entries will be uploaded, other data blocks are considered duplicate data blocks and do not need to be uploaded. During the backup file access process, the second role downloads the data blocks contained in the backup file from the edge layer 14 or the cloud layer 16 according to the address recorded in the PFR, and assembles the downloaded blocks into a complete file according to the sequence of the data blocks in the PFR. The backup file is returned to the client requesting access.

Another important metadata structure is the fingerprint index table (which can be referred to as FingIdx for short), which records the mapping from the fingerprint of the data block to the storage address of the data block. The capacity of FingIdx is generally smaller than that of PFR because it contains only one entry for any data block. A global fingerprint index table records the metadata information of all stored data blocks. During the data backup file upload process, when a data block has its fingerprint detected in FingIdx, the data block is identified as a duplicate block. In this case, their storage locations will be passed directly to the UFR without the need for reallocation of addresses.

Terminal layer 12 of CoopDedup architecture: Terminal layer 12 is the first layer of CoopDedup architecture. Different users' terminal devices will generate a large number of backup files. It is uneconomical to upload the original files of the terminal device to the remote cloud layer 16 for data deduplication, because this will cause a large amount of redundant data to be transmitted multiple times on the backbone network. Therefore, this article advances the data deduplication process to the terminal layer 12 and edge layer 14 of the front end. The main task of the terminal device is to divide the generated backup file into data blocks of different sizes and calculate fingerprints for these divided data blocks. The terminal layer 12 shown in Figure 5 is also called the terminal layer. Fingerprints calculated from a backup file will be assembled into UFR sequentially. This UFR will be uploaded to the subsequent edge layer 14 for further processing. Transmitting UFR instead of transmitting all segmented data blocks can effectively reduce the network transmission burden.

Therefore, the innovation on the terminal layer 12 is that instead of deduplicating data blocks on the terminal device, UFR is chosen to be uploaded for redundancy identification, because firstly due to the limited storage and computing resources on the terminal device, the storage It is impractical to use a global fingerprint index table to record the metadata information of all data blocks split by the terminal device. Scarce computing resources also hinder redundant data detection based on fingerprint information. Secondly, performing isolated data deduplication on each terminal device may ignore the data redundancy that may exist between multiple terminal devices, thereby affecting the effect of data deduplication.

Edge layer 14 of the CoopDedup architecture: An edge server provides services for multiple terminal devices in an area. All these edge servers constitute the edge layer 14 of the CoopDedup architecture. As an intermediate bridge between the terminal layer 12 and the cloud layer 16, this article innovatively proposes to use the edge layer 14 to undertake some of the tasks performed by the terminal layer 12 and the cloud layer 16 in the traditional deduplication backup method. Pushing the tasks of the terminal layer 12 to the edge layer 14 can release the computing and storage pressure of the terminal device. Bringing the cloud layer 16 tasks closer to the edge layer 14 can reduce frequent data transmission between the terminal and the remote cloud, thereby reducing network transmission overhead.

It should be noted that the storage space of edge layer 14 is incomparable with that of cloud data centers. Therefore, how to effectively utilize the limited resources of the edge layer 14, maximize the effect of data deduplication and reduce transmission costs is the focus of the edge layer 14. This article considers solving this problem from the following two aspects: The first aspect is to store some hot data blocks with high access frequency at the edge of the network. Under the same space resources, storing hot data blocks can serve more end-side data requests than storing cold data blocks, thereby minimizing the transmission cost during data retrieval. The second aspect is that a partial index table can be maintained at the edge layer 14 to pre-deduplicate the uploaded metadata information (UFR), and the detected redundant metadata information will not be further transmitted to the remote cloud layer 16 , to reduce the data transmission volume of the backbone network. These two methods use the storage and computing resources of the edge layer 14 to assist in the deduplication process of backup files, which can effectively save the bandwidth resources of the backbone network and alleviate possible network transmission congestion.

Count-based sketch-based block selection: It is uneconomical to record the access popularity of a data block by directly tracking the count of references, because each data block needs to record its fingerprint and a count of its reference frequency, which incurs non-negligible memory overhead. Therefore, this article chooses to use a fixed-size compact sketch data structure (i.e., Count-Min Sketch) to estimate the reference count (data block popularity) of the data block. Count-Min Sketch is a two-dimensional array, the width is recorded as r, and the depth is recorded as w (r and w are both configurable parameters). For each arriving data block, its fingerprint will be mapped by w independent hash functions to w counters for each r row, as shown in edge layer 14 of Figure 5. The counter in the respective hash location will then be incremented by 1. The reference count of a data block, that is, the popularity of the data block, is estimated by the minimum value Min of all counters that its fingerprint hashed to. The estimation error has been proven to be bounded within n*e/r with probability at least 1-1/w ^e , where n is the total number of data blocks and e is Euler's number.

This article can prove through a simple analysis that this sketch-based data block heat estimation can significantly save memory. For example, if you need to record the reference counts of all divided data blocks (n=2 ¹² ), then 2 ¹² * (20B + 4B) memory space is required. And if you use Count-Min Sketch and set its parameters to r=2 ² and w=2 ¹⁰ , there are 2 ⁽¹⁰⁺²⁾ counters in total, which only occupy 1/1 of the memory required by the direct tracking reference counter method. 6. The space-saving effect of this sketch structure increases exponentially as more data blocks are generated without additional space overhead.

User-perceived and version-adjacent partial fingerprint index: Pre-deduplication of metadata can reduce the transmission burden of the network. For example, when this article uses SHA-1 encoding to calculate block fingerprints and assumes the average block size is 4KB, storing a file of 800TB requires uploading at least 4TB of fingerprints. However, due to limited edge memory space, it is impractical to save a global fingerprint index table (containing fingerprints of all data blocks of processed files). Therefore, the key consideration is how to select a subset of the global fingerprint index table for redundancy detection to maximize the redundancy detection rate while reducing the size of the index table. According to system observations, it is found that the content of backup files from different users is quite different, but a large number of data blocks are shared between adjacent backup files of the same user. Based on this, this paper innovatively proposes the User-aware and Version-adjacent Partial Index Table (UVPIdx)}. The UVPIdx index table divides the fingerprint index table into several independent sub-index tables based on the affiliated user information. Each sub-index table records the metadata information of an adjacent backup version of the user.

The user-perceived fingerprint index speeds up the fingerprint comparison process and facilitates the update operation of the sub-index table corresponding to each user. It should be noted that each UVPIdx index table should always be updated to the latest version of the user's backup file to avoid affecting the duplicate data detection effect due to too large a difference in backup versions. The fingerprint index table with adjacent versions can detect most duplicates while ensuring that the table size is small, and the redundancy detection accuracy is high. As shown in 3, based on an adjacent version of the index table, approximately 66.538%/66.568% = 99.95% duplicates can be detected. When the UFR reaches edge layer 14, the matching entries are directly added to the storage address of the data block through the fingerprint index based on UVPIdx. Only the unmatched portion of UFR will be further uploaded to the remote cloud, which can effectively save bandwidth resources between the edge layer 14 and the cloud layer 16 .

Cloud layer 16 of CoopDedup architecture: Cloud layer 16 is actually a large storage cluster, including a large number of homogeneous storage servers. Sufficient storage and computing resources support cloud layer 16 to have its own data deduplication structure. Cloud layer 16 will maintain a global fingerprint index table that records the fingerprints of all data blocks split by all processed backup files. By comparing the uploaded UFR with the global fingerprint index table, the data blocks corresponding to the matching entries are deemed to have been stored in the cloud, while the unmatched data blocks are considered to have not been stored. Such data blocks should be subsequently retrieved from the terminal. Upload to the cloud for storage. Cloud 16's global fingerprint index table compares data from all terminals to detect all duplicate data blocks and achieve complete redundancy deletion.

In one embodiment, the edge server of the edge layer 14 is also used to upload copies corresponding to hot data blocks stored at the edge to the cloud layer 16 for storage.

It can be understood that further, all data blocks, including hot data blocks stored at the edge, need to retain at least one copy in the cloud. This can still ensure the availability of data blocks when edge resources are unreliable. In addition, cloud storage can also adopt fault-tolerance mechanisms, such as implementing replicas or erasure codes, to further ensure the reliability of data storage.

In one embodiment, each cloud server in cloud layer 16 performs redundancy detection on the uploaded partial unprocessed file spectrum sequence and the global fingerprint index table maintained on the cloud in a distributed index manner.

It can be understood that, further, fingerprint indexing on a storage server is time-consuming and resource-intensive. In order to support distributed fingerprint indexing across cloud storage servers, this embodiment can choose to map the global fingerprint index table and the incoming UFR to different buckets based on data block fingerprints, as shown in the distributed index in Figure 5. Entries mapped to the same bucket will be assigned to a cloud server. Since fingerprints are unique identifiers of data blocks, fingerprint-based bucket mapping can ensure that all matching entries (UFR and global fingerprint index table) will be assigned to the same bucket, accelerating fingerprint indexing without affecting the data deduplication effect. process. After distributed indexing, new unstored data blocks are identified and assigned new cloud storage locations.

In general, through the cooperation of the three layers of end-edge-cloud, the CoopDedup architecture reduces two-way data exchange, saves bandwidth resources, and reduces transmission costs. Two types of data are transmitted between the three layers of end-edge-cloud. The first is metadata for data deduplication detection, namely the UFR and PFR introduced earlier. The second is new unstored data blocks that need to be uploaded. When the file sequence is processed back to the terminal, these new data blocks will be uploaded from the terminal layer 12 to the edge layer 14 or cloud layer 16 for data storage according to the address information in the PFR. Therefore, the data communication model across three layers in the CoopDedup architecture It can be summarized as shown below.

At the terminal layer 12, the original backup file is first divided into data blocks, generating an unprocessed file sequence (UFR), that is, a fingerprint list of all data blocks in the file. This UFR is uploaded and used as input to edge layer 14 for further processing.

At edge layer 14, the fingerprint information contained in the UFR will be hashed into the Count-Min Sketch. Once the estimated data block hotness just exceeds the set threshold, the corresponding data block will be identified as a new hot data block. In this embodiment, it is considered that these hot data blocks should store a copy in the edge layer 14 and assign an edge storage location to them. This storage address will be recorded in the corresponding UFR entry and an edge upload tag will be appended. Additionally, entries in UFR are compared with entries in UAPIdx. Data blocks with matching fingerprints are considered duplicates, i.e. already stored in cloud layer 16, in which case their locations will be copied directly from UAPIdx to UFR. These edge-processed partial file sequences (P-PFRs) will wait at the edge for further assembly. The unmatched portion of UFR will be further uploaded to the resource-rich cloud layer 16.

In cloud layer 16, some of the uploaded UFR and global index tables are redundantly detected in a distributed index manner. The UFR entry corresponding to the identified unstored data block will be appended with a cloud upload tag and a new cloud storage location will be assigned to the data block. When the locations (storage containers and offsets) of all unstored data blocks are determined, the cloud-processed file sequence will be returned to the edge layer 14 and assembled with the file sequence processed by the edge layer 14 into a complete PFR. It should be noted that the entries in the complete PFR will update the UVPIdx in edge layer 14. This is because the UVPIdx in edge layer 14 should always record the metadata of the latest backup version of each user to avoid degradation of the data deduplication effect.

The complete PFR is finally sent back to the terminal layer 12 with the location information of the data block and the upload tag. The terminal is responsible for uploading the corresponding data block according to the storage location and upload tag of the PFR. New data chunks are uploaded to cloud layer 16 designated locations based on cloud upload tags, while hot data chunks are uploaded to nearby edge storage locations designated by edge layer 14 based on edge upload tags. Retrieval of the backup file is marked as ready only after all data blocks have been uploaded successfully.

In one embodiment, further, during the file retrieval process, the terminal device of the terminal layer 12 sends a block request to the edge server and/or cloud server based on the location information of the data blocks in the processed file sequence, and retrieves The obtained data blocks are assembled sequentially into the original backup file.

It can be understood that during the file retrieval process, the terminal sends a block request to the edge server or cloud server based on the location information of the data blocks in the PFR, and assembles the retrieved data blocks into the original backup file in order. Frequently accessed hot data blocks can be downloaded directly from nearby edge servers, reducing the bandwidth usage of the backbone network. Even if the edge server is damaged or other unreliable storage problems occur, the data block request can still find the copy of the data block in the cloud based on the cloud's global fingerprint index table, achieving data block availability and data storage reliability.

In summary, the collaboration between the three vertical interconnection layers of end-edge-cloud significantly reduces the two-way data exchange between layers while ensuring optimal data deduplication performance.

In some embodiments, in order to more clearly and intuitively demonstrate the effect of the above-mentioned system of the present application, some experimental examples are also provided here, which are only for auxiliary explanation and are not the sole limitation of the above-mentioned technical solutions of the present application. This example uses a real-world dataset to empirically evaluate the performance of our CoopDedup architecture. Experimental setup: A desktop computer was used. Dataset: This example evaluates the generalizability of our CoopDedup architecture using a real-world FSL dataset, which contains consecutive snapshots of the file home directories of 13 students. Each snapshot corresponds to a backup file. These snapshots come with a wide variety of typical workloads, such as file system snapshots and virtual machine images. The total dataset size is 67.0GB, and the average snapshot file size is approximately 154.2MB. The global deduplication rate is about 48.62%, that is, about half of the data blocks are duplicates.

In order to more comprehensively illustrate the data deduplication performance, four comparison methods were considered: CoopDedup, the end-edge-cloud vertical integration deduplication storage architecture proposed above. InftyDedup is the latest deduplication storage architecture. It sends metadata information across cloud layers and terminals for redundancy detection, but it ignores the efficient use of edge layer resources. PostDedup is a widely adopted post-processing data deduplication strategy. It transfers backup files directly to the remote cloud for storage and data deduplication. CoopDedup_FIFO is a variant of the CoopDedup architecture. Some index tables and stored data blocks in the edge layer are maintained on a first-in, first-out principle. For the size of the index table and storage data blocks, please refer to the CoopDdup architecture. When space overflows, the data in CoopDedup_FIFO is updated on a first-in, first-out basis.

Comparison indicators: During the backup file upload process, the main comparison indicator of the experiment is the data transmission volume of the backbone network, which is composed of metadata transmission volume and data block transmission volume. The smaller the value of the uploaded data amount, it means that a lot of redundancy has been eliminated before the data is transmitted to the remote cloud. This saves valuable network resources on the backbone network and is crucial to improving network performance.

During the backup file retrieval process, focus on the Bandwidth Saving Ratio (BSR), which represents the percentage reduction in data transmission volume on the backbone network caused by obtaining data blocks from nearby edges. This example also records data downloads from the remote cloud. Of course, the auxiliary advantage of the edge layer in data deduplication comes at the cost of occupying additional storage space at the edge. Therefore, this example also records the extra space occupancy (ESO) of the edge layer, which contains edge layer storage data blocks. The volume and the size of CM-Sketch used.

Parameter settings: This example first decompresses the files in the dataset and divides them into data blocks. The fingerprint of each data block is represented using SHA-1 encoding. For the access popularity of backup files, this example uses the widely used Zipf distribution to generate the access popularity of backup files, in which the concentration of data access is set to 1. The default settings of CM-Sketch parameters used in this example are r=1000000 and w=10. In the CoopDedup architecture, the volume of data blocks stored in the edge layer is set to 30% of the volume of all deduplicated data blocks by default. The amount of edge storage for the CoopDedup_FIFO comparison method is always consistent with the CoopDedup architecture.

Numerical results: This example conducted a large-scale experiment to test the performance of the CoopDedup architecture and its comparison methods in the backup file upload and retrieval processes. The evaluation performance during the backup file upload process is shown in Figure 6. Among them, in Figure 6, InftyDedup has the largest metadata transfer volume: the metadata transfer volume for uploading 450 files is as high as 371.27MB. The metadata transmission volume of the CoopDedup architecture is only about half that of InftyDedup. The reason is that InftyDedup performs redundancy detection by transmitting the entire metadata information between the terminal and the remote cloud, while the edge pre-deduplication function in the CoopDedup architecture can effectively reduce metadata transmission. Data transmission volume from edge to cloud. Although CoopDedup_FIFO also maintains an index table at the edge for metadata pre-deduplication, because it ignores the key role of the backup file derivation relationship, the metadata transmission volume is still very large, that is, approximately 297.27MB is required to upload 450 files. The amount of metadata transferred.

It is important to note that the metadata transfer amount of the traditional PostDedup method remains zero. This is because it transfers raw files directly to the cloud without the need to work out metadata information for redundancy detection. However, as shown in Figure 7, the data block transmission volume of PostDedup is the highest among all comparison methods. This is because other methods only need to upload deduplicated data blocks after redundancy detection of metadata information, and the data block transmission volume is only about half of PostDedup.

The data transfer performance during backup file retrieval is shown in Figure 8. This example randomly generates 1,000 file requests based on the popularity of the file. In Figure 8, the CoopDedup architecture has an absolute advantage in data download volume. Specifically, when 60% of the data blocks can be stored in the edge layer, the data download volume of the CoopDedup architecture is only about 31.79GB. On the contrary, since PostDedup and InftyDedup do not fully utilize the edge layer resources, all involved data blocks are always downloaded from the remote cloud layer, resulting in high data download volume (the data download volume for 1000 file requests is about 150GB).

The bandwidth saving rate performance is shown in Figure 9. When only 10% of data blocks can be stored in the edge layer, the CoopDedup architecture can save 32.77% of bandwidth resources. This is due to CM-Sketch's precise selection of hot data blocks. This allows more data blocks to be fetched at the edge during file retrieval without accessing the remote cloud layer. This trend of increasing the bandwidth saving rate as the amount of stored data increases gradually becomes slow, and the bandwidth saving rate corresponding to the last 10% of data blocks (90%-100%) is only 4.33%. This is because the last selected data blocks are usually cooler data blocks, which may only serve a small number of file access requests. In addition, the CoopDeup_FIFO method can only provide linear growth in bandwidth savings due to the use of first-in, first-out rules to maintain stored data blocks.

Finally, in the performance results of the CoopDedup architecture with different sketch (CM-Sketch) widths, the proportion of data blocks stored in the edge layer is fixed at 30%. As the sketch width increases from 100 to 10 ⁸ , the additional space occupied by the edge layer gradually increases because the CM-Sketch used requires more space to store the hash counters in the sketch. However, this increasing trend in additional space usage is not significant because most of the additional space usage consists of data blocks stored at the edge. As the width of the CM-Sketch sketch increases, the bandwidth saving rate also increases. This demonstrates that large sketch structures can reduce bias in estimates of data block access frequency. When the sketch width is set to greater than 10 ⁷ , the bandwidth saving rate remains unchanged. This shows that the sketch structure under this width setting can accurately estimate the heat of the data block.

To sum up, the CoopDedup architecture makes full use of the three-layer resources of end-edge-cloud and reduces the amount of uploaded metadata by half compared with the current state-of-the-art InftyDedup. Furthermore, when only 10% of data blocks are stored at the edge, bandwidth savings can reach around 33%.

In one embodiment, as shown in Figure 10, a cloud-edge-device vertical integration deduplication storage method 100 is provided, which may include the following processing steps S12 to S18:

S12, hash the fingerprint information contained in the unprocessed file sequence uploaded by the terminal layer into a fixed-size compact sketch data structure to estimate the data block heat, allocate edge storage locations for the estimated new hot data blocks, and then add the edge The storage address of the storage location is recorded in the unprocessed file sequence and an edge upload tag is attached.

S14: Match the entries in the unprocessed file sequence with the entries in the set partial index table, and copy the storage locations corresponding to the data blocks with matching fingerprints from the set partial index table to the unprocessed file sequence. , upload the partial unprocessed file sequence corresponding to the data block that does not match the fingerprint to the cloud layer; set the partial index table to the partial index table that the user perceives and is adjacent to the version.

S16: Assemble the processed file sequence and the partial unprocessed file sequence corresponding to the data block with matching fingerprints into a complete processed file sequence, and return the processed file sequence to the terminal layer. Among them, the processed file sequence is used through the cloud server in the cloud layer to perform redundancy detection on the uploaded part of the unprocessed file sequence and the global fingerprint index table maintained in the cloud, and the unprocessed data corresponding to the unidentified unstored data blocks is checked. The entry in the file sequence is obtained and returned after appending a cloud upload tag and assigning a new cloud storage location corresponding to the unstored data block.

S18, the terminal device at the receiving terminal layer uploads and stores the new hot data block according to the edge storage location and edge upload tag in the processed file sequence; the new cloud storage location and cloud upload tag in the processed file sequence It is also used to instruct the terminal device at the terminal layer to upload unstored data blocks to the cloud layer for storage.

It can be understood that for specific limitations on the deduplication storage method for cloud-edge-end vertical integration, please refer to the corresponding limitations of the deduplication storage system 100 for cloud-edge-end vertical integration mentioned above, which will not be described again here. The above deduplication storage method for vertical integration of cloud and edge devices is described from the perspective of the edge layer, so as to facilitate a more intuitive understanding of the technical solution of this application.

The above-mentioned deduplication storage method of vertical integration of cloud, edge and end is responsible for dividing files into blocks through the terminal layer, and uploading the generated metadata information to the edge layer and cloud layer for redundancy detection. The edge layer reduces the bandwidth resource overhead of the backbone network by storing hotter data blocks. In addition, the uploaded metadata can also be pre-deduplicated at the edge layer to further reduce the amount of data transmission; the cloud layer maintains a global fingerprint index table for global data deduplication and combines the global fingerprint index table with the incoming metadata. Divide it into different servers in the cloud data center to support distributed parallel indexing across cloud storage servers, thereby globally improving data deduplication and storage performance. Compared with traditional technology, this end-edge-cloud vertical integration deduplication storage architecture effectively integrates different levels of technology and storage resources in the process of file transmission, storage and retrieval, while ensuring the best data deduplication performance. At the same time, the amount of two-way data transmission between layers is significantly reduced, achieving the effect of significantly reducing the resource overhead of the backbone network of the cloud-edge architecture.

In one embodiment, each cloud server in the cloud layer performs redundancy detection in a distributed index manner between the uploaded partial unprocessed file spectrum sequence and the global fingerprint index table maintained in the cloud.

In one embodiment, the above-mentioned deduplication storage method for vertical integration of cloud and edge devices may also include the following processing steps:

The edge server uploads a copy of the hot data block stored at the edge to the cloud for storage.

In one embodiment, during the file retrieval process, the terminal device at the terminal layer sends a block request to the edge server and/or the cloud server based on the location information of the data blocks in the processed file sequence, and sends the retrieved data blocks Assemble the original backup file in sequence.

It should be understood that although each step in the above flowchart 10 is shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps of the above-mentioned flow chart 10 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution of these sub-steps or stages The sequence is not necessarily sequential, but may be performed in turn or alternately with other steps or sub-steps of other steps or at least part of the stages.

In one embodiment, a computer device is also provided, including a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements the following processing steps: convert the unprocessed file sequence uploaded by the terminal layer to The fingerprint information is hashed into a fixed-size compact sketch data structure to estimate the data block heat. After allocating an edge storage location for the estimated new hot data block, the storage address of the edge storage location is recorded into the unprocessed file sequence and attached. An edge upload tag; matches entries in the unprocessed file's spectrum sequence with entries in the set part index table, and copies the storage locations corresponding to data blocks with matching fingerprints from the set part index table to the unprocessed file After the spectrum is sequenced, upload the partially unprocessed file spectrum sequence corresponding to the data block that does not match the fingerprint to the cloud; set the partial index table to the partial index table that the user perceives and is adjacent to the version; compare the processed file spectrum sequence with Part of the unprocessed file sequence corresponding to the data block matching the fingerprint is assembled into a complete processed file sequence, and the processed file sequence is returned to the terminal layer; among which, the processed file sequence is passed through the cloud server in the cloud layer The partially unprocessed file sequence uploaded is checked for redundancy with the global fingerprint index table maintained in the cloud, and a cloud upload tag is attached to the entry in the unprocessed file sequence corresponding to the unidentified unstored data block and assigned The new cloud storage location corresponding to the unstored data block is obtained and returned; the terminal device receiving the terminal layer uploads the new hot data block and stores it according to the edge storage location and edge upload tag in the processed file sequence; the processed The new cloud storage location and cloud upload tag in the file sequence are also used to instruct the terminal device at the terminal layer to upload unstored data blocks to the cloud layer for storage.

It can be understood that, in addition to the above-mentioned memory and processor, the above-mentioned computer equipment also includes other software and hardware components not listed in this manual. The details can be determined according to the model of the specific edge server device in different application scenarios. This manual does not List the details one by one.

In one embodiment, when the processor executes the computer program, it can also implement the steps or sub-steps added in the above embodiments of the cloud-edge-device vertical integration deduplication storage method.

In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by the processor, the following processing steps are implemented: fingerprints contained in the unprocessed file sequence uploaded by the terminal layer Hash the information into a fixed-size compact sketch data structure to estimate the data block heat. After allocating edge storage locations for the estimated new hot data blocks, record the storage addresses of the edge storage locations into the unprocessed file sequence and attach a Edge upload tag; match the entries in the unprocessed file spectrum sequence with the entries in the set part index table, and copy the storage locations corresponding to the data blocks with matching fingerprints from the set part index table to the unprocessed file spectrum. After the sequence, upload the partial unprocessed file sequence corresponding to the data block without matching fingerprint to the cloud layer; set the partial index table to the partial index table that the user perceives and is adjacent to the version; compare the processed file sequence with the matching file sequence Part of the unprocessed file sequence corresponding to the fingerprint data block is assembled into a complete processed file sequence, and the processed file sequence is returned to the terminal layer; among which, the processed file sequence is sent to the terminal layer through the cloud server in the cloud layer. The uploaded part of the unprocessed file sequence is checked for redundancy with the global fingerprint index table maintained in the cloud, and a cloud upload tag is attached to the entry in the unprocessed file sequence corresponding to the unidentified unstored data block and assigned a corresponding The new cloud storage location of the unstored data block is obtained and returned; the terminal device receiving the terminal layer uploads the new hot data block and stores it according to the edge storage location and edge upload tag in the processed file sequence; the processed file The new cloud storage location and cloud upload tag in the spectrum sequence are also used to instruct the terminal device at the terminal layer to upload unstored data blocks to the cloud layer for storage.

In one embodiment, when the computer program is executed by the processor, the added steps or sub-steps in the above embodiments of the cloud-edge-device vertical integration deduplication storage method can also be implemented.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program may include the processes of the above method embodiments. Any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Synchlink DRAM (SLDRAM), memory bus dynamic random access memory (RambusDRAM, RDRAM for short) and interface dynamic random access memory (DRDRAM), etc.

The technical features of the above embodiments can be combined in any way. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, all possible combinations should be used. It is considered to be within the scope of this manual.

The above embodiments only express several implementation modes of the present application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, which all fall within the protection scope of the present application. Therefore, the scope of protection of this patent application shall be determined by the appended claims.

Claims (10)

1. A deduplication storage system with vertical integration of cloud, edge and end, characterized in that it includes a terminal layer, an edge layer and a cloud layer. After the terminal device of the terminal layer divides the original backup file into each data block, it generates each data block. The unprocessed file sequence corresponding to the data block is sequenced and uploaded to the edge layer; The edge server in the edge layer hashes the fingerprint information contained in the unprocessed file sequence into a fixed-size compact sketch data structure to estimate the data block heat, and allocates edge storage locations for the estimated new hot data blocks. Then record the storage address of the edge storage location into the unprocessed file sequence and attach an edge upload tag; The edge server in the edge layer matches the entries in the unprocessed file sequence with the entries in the set partial index table, and removes the storage locations corresponding to the data blocks with matching fingerprints from the set partial index table. After copying to the unprocessed file sequence, upload the part of the unprocessed file sequence corresponding to the data block that does not match the fingerprint to the cloud layer; the set partial index table is user-perceived and version-adjacent. Partial index table; The cloud server in the cloud layer performs redundancy detection on the uploaded part of the unprocessed file spectrum sequence and the global fingerprint index table maintained by the cloud, and checks the unprocessed file spectrum sequence corresponding to the unidentified unstored data block. After attaching a cloud upload tag to the entry in the sequence and assigning a new cloud storage location corresponding to the unstored data block, return the processed file sequence to the edge layer; The edge server of the edge layer assembles the processed file sequence and the unprocessed file sequence corresponding to the data block with matching fingerprint into a complete processed file sequence, and sends it to the terminal layer Return the processed file sequence; The terminal device of the terminal layer uploads a new hot data block to the edge layer for storage according to the edge storage location and edge upload tag in the processed file sequence, and uploads the new hot data block to the edge layer according to the processed file sequence. The new cloud storage location and cloud upload tag upload the unstored data chunks to the cloud layer for storage. 2. The cloud-edge-end vertical integration deduplication storage system according to claim 1, characterized in that each cloud server in the cloud layer combines the uploaded part of the unprocessed file sequence with the global fingerprint maintained by the cloud. The index table performs redundancy detection in a distributed index manner. 3. The cloud-edge-end vertical integration deduplication storage system according to claim 2, characterized in that the edge server of the edge layer is also used to upload corresponding copies of hot data blocks stored at the edge to the cloud layer. stored in. 4. The cloud-edge-end vertical integration deduplication storage system according to any one of claims 1 to 3, characterized in that, during the file retrieval process, the terminal device at the terminal layer performs the deduplication according to the processed file spectrum. The block request is sent to the edge server and/or the cloud server based on the location information of the data blocks in the sequence, and the retrieved data blocks are assembled into the original backup file in order. 5. A deduplication storage method for vertical integration of cloud, edge and end, which is characterized by including the steps: Hash the fingerprint information contained in the unprocessed file sequence uploaded by the terminal layer into a fixed-size compact sketch data structure to estimate the data block heat, and allocate edge storage locations to the estimated new hot data blocks. The storage address is recorded in the unprocessed file sequence and an edge upload tag is attached; Match the entries in the unprocessed file sequence with the entries in the setting part index table, and copy the storage locations corresponding to the data blocks with matching fingerprints from the setting part index table to the unprocessed After the file sequence, the unprocessed file sequence corresponding to the data block without matching fingerprint is uploaded to the cloud layer; the set partial index table is the partial index table that the user perceives and is adjacent to the version; Assemble the processed file sequence and the unprocessed file sequence corresponding to the data block with matching fingerprint into a complete processed file sequence, and return the processed file sequence to the terminal layer ; Wherein, the processed file sequence is used by the cloud server in the cloud layer to perform redundancy detection on the uploaded part of the unprocessed file sequence and the global fingerprint index table maintained by the cloud, and the unidentified and unidentified file sequences are detected for redundancy. The entry in the unprocessed file sequence corresponding to the stored data block is obtained and returned after appending a cloud upload tag and assigning a new cloud storage location corresponding to the unstored data block; The terminal device that receives the terminal layer uploads and stores new hot data blocks according to the edge storage location and edge upload tag in the processed file sequence; the new cloud storage location in the processed file sequence and the cloud upload tag are also used to instruct the terminal device of the terminal layer to upload unstored data blocks to the cloud layer for storage. 6. The cloud-edge-end vertical integration deduplication storage method according to claim 5, characterized in that each cloud server in the cloud layer combines the uploaded part of the unprocessed file sequence with the global fingerprint maintained by the cloud. The index table performs redundancy detection in a distributed index manner. 7. The cloud-edge-end vertical integration deduplication storage method according to claim 6, characterized in that it further includes the steps: Upload the corresponding copy of the hot data block stored at the edge to the cloud layer for storage. 8. The cloud-edge-end vertical integration deduplication storage method according to any one of claims 5 to 7, characterized in that, during the file retrieval process, the terminal device at the terminal layer performs the deduplication and storage according to the processed file spectrum. The block request is sent to the edge server and/or the cloud server based on the location information of the data blocks in the sequence, and the retrieved data blocks are assembled into the original backup file in order. 9. A computer device, including a memory and a processor. The memory stores a computer program. It is characterized in that when the processor executes the computer program, the cloud-edge-end vertical integration described in claim 5 or 7 is realized. Steps of deduplication storage method. 10. A computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the deduplication storage method for cloud-edge-end vertical integration described in claim 5 or 7 is implemented. step.