CN111984196B - File migration method, device, equipment and readable storage medium - Google Patents

Abstract

The invention discloses a file migration method, device, equipment and readable storage medium. The method includes: receiving a file migration task; judging whether the file migration task is an unfinished task; Invalid files are determined from the aggregated large files in the pool and each of the small files in the source storage pool; the aggregated large files have migrated small files corresponding to each small file after aggregation; the invalid files are garbage collected. The method can recycle the invalid data generated during the aggregation migration process, and can improve the disk utilization rate.

Description

A file migration method, apparatus, device and readable storage medium

technical field

The present invention relates to the field of storage technologies, and in particular, to a file migration method, apparatus, device and readable storage medium.

Background technique

Because the speed of reading and writing large files from disk is often significantly higher than that of small files. In order to take advantage of this feature, in the application scenario of massive small files, when writing small files, it is not directly placed on the disk, but aggregated into large files and then placed on the disk, thereby effectively reducing the number of times small files are written to disk and reducing the pressure of writing data, and at the same time Also improves read hit rates and shortens read I/O paths.

However, if the file grading migration client hangs up during the file migration and aggregation process, the entire migration and aggregation process cannot be completed, resulting in invalid garbage data. Invalid junk data will occupy storage space, resulting in low storage utilization.

To sum up, how to effectively solve the problems such as disk cleanup in the process of small file migration is a technical problem that those skilled in the art need to solve urgently.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a file migration method, device, device and readable storage medium, which can effectively recover garbage data generated due to aggregate migration failures.

In order to solve the above-mentioned technical problems, the present invention provides the following technical solutions:

A file migration method including:

Receive file migration tasks;

Determine whether the file migration task is an unfinished task;

If yes, determine an invalid file from the aggregated large file located in the destination storage pool corresponding to the file migration task and each small file located in the source storage pool; The migration small files corresponding to each other after aggregation;

The invalid files are garbage collected.

Preferably, the invalid files are determined from the aggregated large files located in the destination storage pool and each of the small files located in the source storage pool corresponding to the file migration task, including:

Obtain the aggregated attributes of each of the small files;

Using the aggregated attribute, the invalid file is determined from the aggregated large file and each of the small files.

Preferably, the invalid file is determined from the aggregated large file and each of the small files by using the aggregated attribute, including:

Determine whether the aggregation attribute changes;

If yes, then determine that the corresponding small file is the invalid file;

If not, it is determined that the corresponding migrated small file stored in the aggregated large file is an invalid file.

Preferably, garbage collection is performed on the invalid files, including:

If the invalid file is the small file, delete the small file in the source storage pool;

If the invalid file is the migrated small file, perform fragment recovery processing on the aggregated large file.

Preferably, performing fragment recovery processing on the aggregated large file, including:

obtain the file header information of the aggregated large file;

Using the file header information, determine the proportion of invalid data in the aggregated large file;

In the case that the invalid data ratio is greater than the threshold, the valid data in the aggregated large file is migrated to the target aggregated large file and the aggregated large file is deleted.

Preferably, performing fragment recovery processing on the aggregated large file, including:

obtaining the aggregated attributes corresponding to each file in the aggregated large file;

Determining the files whose aggregated attributes have not changed as invalid data, and count the proportion of invalid data;

In the case that the invalid data ratio is greater than the threshold, the valid data in the aggregated large file is migrated to the target aggregated large file and the aggregated large file is deleted.

Preferably, the receiving file migration task includes:

receiving the file migration task sent by the metadata server;

Correspondingly, when the garbage collection of the invalid file is completed, the method further includes:

Feedback cleanup response data to the metadata server.

A file migration device, comprising:

The task receiving module is used to receive file migration tasks;

a judging module for judging whether the file migration task is an unfinished task;

an invalid file determination module, configured to determine invalid files from the aggregated large files located in the destination storage pool and each of the small files located in the source storage pool corresponding to the file migration task if the judgment result is yes; the aggregated The large file has migration small files corresponding to each of the small files by aggregation;

A garbage collection processing module, configured to perform garbage collection on the invalid files.

A file migration device including:

memory for storing computer programs;

The processor is configured to implement the steps of the above file migration method when executing the computer program.

A readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the above-mentioned file migration method.

Applying the method provided by the embodiment of the present invention, receiving a file migration task; judging whether the file migration task is an unfinished task; if so, corresponding to the aggregated large file located in the destination storage pool and each located in the source storage pool from the file migration task In the small files, invalid files are determined; the aggregated large files have migration small files corresponding to each small file after aggregation; garbage collection is performed on the invalid files.

It is understandable that if the file grading migration client fails, the corresponding migration task cannot be completed. At this time, if the small file has been written to the aggregated large file, the same small file may appear in the source storage pool. , and the aggregated large files of the destination storage pool. Corresponding to the aggregation migration process, the same file exists in both the source storage pool and the target storage pool. Obviously, there are invalid files. Based on this, in this method, after receiving the file migration task, it is first determined whether the file migration task is an unfinished task, and if it is an unfinished task, there will be garbage data corresponding to the file migration task. At this time, invalid files can be determined from the aggregated large files in the target storage pool corresponding to the file migration task and each of the small files in the source storage pool; wherein, the aggregated large files have the ability to aggregate each small file to separate The corresponding migration small file. Then, recycle the invalid files. In this way, the invalid data generated during the aggregation migration process can be recycled, which can improve the disk utilization rate.

Correspondingly, the embodiments of the present invention also provide a file migration apparatus, a device, and a readable storage medium corresponding to the above-mentioned file migration method, which have the above-mentioned technical effects, and are not repeated here.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

FIG. 1 is an implementation flowchart of a file migration method in an embodiment of the present invention;

2 is a schematic structural diagram of a file migration device according to an embodiment of the present invention;

3 is a schematic structural diagram of a file migration device in an embodiment of the present invention;

FIG. 4 is a schematic diagram of a specific structure of a file migration device according to an embodiment of the present invention.

Detailed ways

In order to make those skilled in the art better understand the solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

For ease of understanding, the technical terms involved in the embodiments of the present invention are explained below:

A small file refers to a file with a small size. For example, a file smaller than 1M can be defined as a small file, and a file smaller than 512K can also be defined as a small file.

Aggregate large files, that is, large files obtained by aggregating batches of small files. For example, the size of the aggregated large file may be a large file with a maximum size of 512M.

Backend: The file grading migration client is responsible for receiving the migration request sent by the MDS, executing data migration from the source storage pool to the destination storage pool, and executing the relevant migration aggregation process according to the aggregation conditions;

MDS: Metadata server, which maintains the metadata information of all files in the file system, and is also responsible for organizing the small files to be migrated and aggregated and sent to the Backend;

Distributed file system: a cluster composed of multiple file storage node servers, files are stored in blocks, with an object as the basic unit, supporting a piece of data to be stored on multiple nodes, and each node can obtain complete data through inter-node communication. Data, when the node is down, the complete data can be recovered according to the configured strategy, which has the characteristics of high availability, high performance, and high scalability. Each node provides metadata services, namely MDS, for metadata A variety of access operations to balance business pressure;

File classification: File classification is the file migration function. By configuring the corresponding policy, files can be transferred between different pools. By specifying the relevant aggregation policy, the aggregation can be performed first when writing to a new pool during migration, and then the new pool can be written. pond;

Small file aggregation: Aggregate batches of small files into a large file, because the speed of reading and writing large files from disk is often significantly higher than that of small files. In order to take advantage of this feature, in the application scenario of massive small files, when writing small files, it is not directly placed on the disk, but merged into large files and then placed on the disk, thereby effectively reducing the number of small files written to disk and reducing the pressure of writing data, and at the same time Also improves read hit rates and shortens read I/O paths.

Please refer to FIG. 1. FIG. 1 is a flowchart of a file migration method according to an embodiment of the present invention. The method includes the following steps:

S101. Receive a file migration task.

Specifically, after the file grading migration client recovers from a fault, the metadata server may send a file migration task to the file grading migration client. That is, a file migration task can be a new task or a task that has not been completed due to a failure.

In this embodiment of the present invention, the file level migration client may receive the file migration task sent by the metadata server. Specifically, the metadata server may send a file migration task to the file classification migration client when it is detected that the file classification migration client after the failure recovery has an unfinished migration task.

That is to say, in this embodiment of the present invention, after the MDS sends the migration aggregation task, the MDS considers the migration aggregation to be completed only if the entire file migration aggregation process is completed. , the unfinished task will be restarted first, that is, the MDS will send a file migration task to Backend.

S102. Determine whether the file migration task is an unfinished task.

In the file migration task, the unfinished file migration task identifier and the migrated small file identifier can be carried, so that the file grading migration client can determine the invalid file corresponding to the unfinished file migration task based on the identifier information.

Whether the file migration task is incomplete can be determined based on the presence or absence of the flag of the incomplete file migration task. If yes, execute the operation of step S103; if not, process the file migration task normally, such as execute the operation of S105.

Of course, to determine whether the file migration task is an unfinished task, it can also be determined through the recorded file migration task record table whether it is an unfinished task.

S103: Determine invalid files from the file migration task corresponding to the aggregated large file located in the destination storage pool and each of the small files located in the source storage pool.

Wherein, the aggregated large file includes migration small files corresponding to each small file after aggregation.

File aggregation migration, that is, aggregating batches of small files into large files and then performing file migration.

In order to facilitate the understanding of the technical solutions provided by the embodiments of the present invention, an example is given below for the small file aggregation process. The small file aggregation migration process includes the following steps:

Step 1. The MDS (metadata server) sends the migration aggregation task, and the Backend (file grading migration client) receives the file migration task;

Step 2. Backend will read the data of the small file from the source storage pool;

Step 3. Backend writes the read data into the aggregate cache;

Step 4. The Backend aggregate cache reaches the flushing condition, that is, when multiple small files are written into the aggregate cache to reach a certain size, they are written to the disk in the destination storage pool;

Step 5. After backend writing is completed, set the aggregation properties of the file, that is, aggregate large files. The offset in the aggregated large file, the data pool where it is located, and sent to MDS;

Step 6. After the MDS setting is completed, reply to Backend. After Backend receives metadata such as the successfully updated aggregation attribute, it deletes the data of small files in the source storage pool;

Step 7. Respond to MDS after completion, indicating that the migration and aggregation process of the file is completed;

During the migration and aggregation process of small files, if the Backend hangs up, the entire migration and aggregation process cannot be completed, resulting in invalid garbage data. Since the migration and aggregation process is divided into several steps, problems may occur at any step, mainly resulting in garbage data. The situations include but are not limited to the following two situations:

Scenario 1: The small file has been written into the aggregated large file, but the aggregation attribute of the small file has not been modified. At this time, the valid data is the small file data in the source storage pool, and the small file data in the aggregated large file is garbage data. Clear it, otherwise it will continue to occupy space and waste system storage resources;

Case 2: The small file has been written into the aggregated large file, but the aggregation attribute of the small file has been modified, the valid data is the data in the aggregated large file, and the ordinary small file data in the source storage pool is garbage data, which needs to be cleared, otherwise Continue to occupy space and waste system storage resources.

Based on this, in the embodiment of the present invention, an invalid file may be determined from the aggregated large file located in the destination storage pool corresponding to the file migration task and each small file located in the source storage pool. Invalid files correspond to invalid data. According to the analysis of the small file aggregation migration process, the invalid files can be small files in the source storage pool or aggregated files in the target storage pool.

Specifically, the process of determining invalid files includes:

Step 1: Obtain the aggregated attributes of each small file.

Step 2: Use the aggregation attribute to determine invalid files from the aggregated large files and individual small files.

Specifically, the aggregated attributes of the small files can be obtained from the metadata server. That is, the aggregated attribute can be stored as metadata. Of course, the aggregated attribute can also be used as the tag information of the small file, and the aggregated attribute of the small file can also be obtained by reading the corresponding tag information.

The aggregated attribute of the small file may specifically include the offset of the small file in the aggregated large file and the data pool where the small file is located.

The second step may specifically include:

Step 2.1. Determine whether the aggregation attribute changes;

Step 2.2. If yes, then determine that the corresponding small file is an invalid file;

Step 2.3: If no, determine that the corresponding migrated small file stored in the aggregated large file is an invalid file.

After the file grading migration client obtains the aggregated attributes of the small files, it can determine whether each small file stored in different locations is an invalid file. Specifically, during the aggregation migration process, if the small file is written into the large aggregated file, the aggregation attribute will be changed, so whether the small file is valid can be determined according to whether the aggregation attribute changes.

In the embodiment of the present invention, in order to distinguish the small files before and after the migration, the small files located in the source storage pool before the migration are still called small files; the small files located in the aggregated large files in the destination storage pool after the migration are called as small files For migrating small files. A small file to migrate in the destination storage pool corresponds to a small file in the source storage pool.

Specifically, when the aggregate attribute of the small file changes, it is determined that the small file in the corresponding source storage pool is invalid, and the small file is an invalid small file. When the aggregate attribute of the small file does not change, it may be that the process of writing the small file to the aggregated large file has not been completed, that is, the corresponding migrated small file in the aggregated large file may have missing data content, so it can be determined that the migration Small files are invalid.

It should be noted that, in this embodiment of the present invention, the determination of whether a file is valid is performed with respect to different locations of the same small file. For example, for the small file a, it exists in the source storage pool (call it a1), and also exists in the migrated small file (call it a2) in the aggregated large file of the destination storage pool, at this time To judge whether the file is valid, it is to judge which one of a1 and a2 is valid. If a1 is valid, the small file is valid, but a2 is invalid; if a2 is valid, that is, the migration small file is valid, then a is invalid. That is to say, if there are corresponding a1 and a2, one of the files must be invalid.

S104. Perform garbage collection on invalid files.

In the embodiment of the present invention, garbage collection is performed on invalid files, that is, the occupation of the disk by the invalid files is released.

Specifically, for the same small file, if the small file exists in both the source storage pool and the destination storage pool (the small file existing in the destination storage pool is called a migrated small file), then there must be a storage pool. Small files in are invalid. That is to say, the invalid file can be a small file in the source storage pool, or it can be a migrated small file stored in the aggregated large file for the purpose.

Preferably, for invalid files in different storage locations, different processing methods can be used to perform garbage collection. Specific circumstances include:

Case 1: If the invalid file is a small file, delete the small file in the source storage pool;

Case 2: If the invalid file is a small migration file, the aggregated large file will be fragmented and recycled.

That is to say, if the small files in the source storage pool are invalid, you can directly delete the small files in the original storage pool. If the invalid file is the migrated small file in the aggregated large file in the destination storage pool, in order to avoid a large number of disk fragments caused by deleting the invalid migrated small file, you can use the fragment recovery method to remove the migrated small file in the aggregated large file.

It should be noted that the methods for performing fragment recovery processing on aggregated large files include but are not limited to the following two specific implementation methods:

Mode 1: Perform fragmentation cleaning based on file header information. The specific implementation steps include:

Step 2.1.1. Obtain the file header information of the aggregated large file;

Step 2.1.2. Use the file header information to determine the proportion of invalid data in the aggregated large file;

Step 2.1.3. When the proportion of invalid data is greater than the threshold, migrate the valid data in the aggregated large file to the target aggregated large file and delete the aggregated large file.

Generally speaking, a file aggregation migration task may be interrupted by a fault at any time during the migration. Therefore, the valid data corresponding to the corresponding small files to be migrated may be partly in the destination storage pool and partly in the source storage pool. Correspondingly, some of the migrated small files in the aggregated large files are valid data and some are invalid data. In order to reduce the number of repeated migrations, in this embodiment of the present invention, after the invalid part in the aggregated large file reaches a certain threshold, fragmentation may be cleaned up uniformly.

Specifically, the file header information can be used to determine the proportion of invalid data in the aggregated large file, and then when the proportion of invalid data is greater than the threshold, the valid data in the aggregated large file is migrated to the target aggregated large file and deleted. Aggregate large files. Among them, the threshold value can be set and adjusted according to the requirements of the actual application, for example, it can be set to specific values such as 80% and 75%. Specifically, the higher the threshold, the better the garbage collection effect, but it may bring more data migration times; the lower the threshold, the more garbage fragments in the aggregated large files, but it will reduce the amount of garbage collection caused by garbage collection. data migration.

Method 2: Perform fragmentation cleaning based on the aggregated attributes of small files. The specific implementation steps include:

Step 2.2.1. Obtain the aggregation attributes corresponding to each file in the aggregation large file;

Step 2.2.2. Determine the files whose aggregate attributes have not changed as invalid data, and count the proportion of invalid data;

Step 2.2.2. When the proportion of invalid data is greater than the threshold, migrate the valid data in the aggregated large file to the target aggregated large file and delete the aggregated large file.

As can be seen from the above, the aggregation attribute can determine whether the same small file is valid in the source storage pool or in the destination storage pool. Therefore, the wireless data in the aggregated large file can be determined based on the aggregation attribute, and the statistics can be obtained. Invalid data ratio. After the invalid data ratio is obtained, the subsequent processing method may refer to the above method 1 for details.

In particular, in the case where the proportion of invalid data is less than or equal to the threshold, it may not be necessary to perform fragmentation processing on the aggregated large files.

Preferably, after the data cleaning is completed, the normal file migration process can be resumed. Specifically, in the case of completing the garbage collection for invalid files, feedback cleanup response data to the metadata server. In this way, the metadata server can confirm that the data cleaning has been completed, and can issue a new file aggregation migration task.

S105. Execute a file migration task.

That is, the corresponding small files are read from the source storage pool, multiple small files are aggregated, large aggregated files are obtained, and the aggregated large files are migrated to the destination storage pool.

Applying the method provided by the embodiment of the present invention, receiving a file migration task; judging whether the file migration task is an unfinished task; if so, corresponding to the aggregated large file located in the destination storage pool and each located in the source storage pool from the file migration task In the small files, invalid files are determined; the aggregated large files have migration small files corresponding to each small file after aggregation; garbage collection is performed on the invalid files.

It is understandable that if the file grading migration client fails, the corresponding migration task cannot be completed. At this time, if the small file has been written to the aggregated large file, the same small file may appear in the source storage pool. , and the aggregated large files of the destination storage pool. Corresponding to the aggregation migration process, the same file exists in both the source storage pool and the target storage pool. Obviously, there are invalid files. Based on this, in this method, after receiving the file migration task, it is first determined whether the file migration task is an unfinished task, and if it is an unfinished task, there will be garbage data corresponding to the file migration task. At this time, invalid files can be determined from the aggregated large files in the target storage pool corresponding to the file migration task and each of the small files in the source storage pool; wherein, the aggregated large files have the ability to aggregate each small file to separate The corresponding migration small file. Then, recycle the invalid files. In this way, invalid data generated during the aggregation migration process can be recycled, which can improve disk utilization.

Corresponding to the above method embodiments, the embodiments of the present invention further provide a file migration apparatus, and the file migration apparatus described below and the file migration method described above may refer to each other correspondingly.

Referring to Figure 2, the device includes the following modules:

a task receiving module 101, configured to receive a file migration task;

Judging module 102, for judging whether the file migration task is an unfinished task;

The invalid file determination module 103 is configured to determine invalid files from the aggregated large files located in the destination storage pool and each of the small files located in the source storage pool corresponding to the file migration task if the judgment result is yes; the aggregated large files have Aggregate each small file to migrate the corresponding small files respectively;

The garbage collection processing module 104 is configured to perform garbage collection on invalid files.

Apply the device provided by the embodiment of the present invention to receive a file migration task; determine whether the file migration task is an unfinished task; if so, the slave file migration task corresponds to the aggregated large files located in the destination storage pool and each of them located in the source storage pool. In the small files, invalid files are determined; the aggregated large files have migration small files corresponding to each small file after aggregation; garbage collection is performed on the invalid files.

It is understandable that if the file grading migration client fails, the corresponding migration task cannot be completed. At this time, if the small file has been written to the aggregated large file, the same small file may appear in the source storage pool. , and the aggregated large files of the destination storage pool. Corresponding to the aggregation migration process, the same file exists in both the source storage pool and the target storage pool. Obviously, there are invalid files. Based on this, in this method, after receiving the file migration task, it is first determined whether the file migration task is an unfinished task, and if it is an unfinished task, there will be garbage data corresponding to the file migration task. At this time, invalid files can be determined from the aggregated large files in the target storage pool corresponding to the file migration task and each of the small files in the source storage pool; wherein, the aggregated large files have the ability to aggregate each small file to separate The corresponding migration small file. Then, recycle the invalid files. In this way, the invalid data generated during the aggregation migration process can be recycled, which can improve the disk utilization rate.

In a specific embodiment of the present invention, the invalid file determination module 103 is specifically configured to obtain the aggregated attributes of each small file; and by using the aggregated attributes, the invalid files are determined from the aggregated large files and each of the small files.

In a specific embodiment of the present invention, the invalid file determination module 103 is specifically configured to determine whether the aggregated attribute changes; if so, determine that the corresponding small file is an invalid file; if not, determine that it is stored in the aggregated large file The corresponding migration small file is invalid.

In a specific embodiment of the present invention, the garbage collection processing module 104 is specifically configured to delete the small files in the source storage pool if the invalid files are small files; Perform scrap recycling.

In a specific embodiment of the present invention, the garbage collection processing module 104 is specifically configured to obtain the file header information of the aggregated large file; use the file header information to determine the proportion of invalid data in the aggregated large file; when the proportion of invalid data is greater than If the threshold is reached, migrate the valid data in the aggregated large file to the target aggregated large file and delete the aggregated large file.

In a specific embodiment of the present invention, the garbage collection processing module 104 is specifically configured to obtain the aggregated attributes corresponding to each file in the aggregated large file; determine the files whose aggregated attributes have not changed as invalid data, and count the proportion of invalid data ; When the proportion of invalid data is greater than the threshold, migrate the valid data in the aggregated large file to the target aggregated large file and delete the aggregated large file.

In a specific embodiment of the present invention, the task receiving module 101 is specifically configured to receive a file migration task sent by a metadata server;

Correspondingly, it also includes: a cleanup feedback module, configured to feed back cleanup response data to the metadata server when garbage collection of invalid files is completed.

Corresponding to the above method embodiments, the embodiments of the present invention further provide a file migration device, and a file migration device described below and a file migration method described above may refer to each other correspondingly.

Referring to Figure 3, the file migration device includes:

memory 332 for storing computer programs;

The processor 322 is configured to implement the steps of the file migration method of the above method embodiments when executing the computer program.

Specifically, please refer to FIG. 4. FIG. 4 is a schematic diagram of a specific structure of a file migration device provided in this embodiment. The file migration device may have a relatively large difference due to different configurations or performances, and may include one or more processes. Central processing units (CPUs) 322 (eg, one or more processors) and memory 332 storing one or more computer applications 342 or data 344 . Among them, the memory 332 may be short-lived storage or persistent storage. The programs stored in memory 332 may include one or more modules (not shown), each of which may include a series of instructions to operate on a data processing device. Furthermore, the central processing unit 322 may be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the file migration device 301 .

File migration device 301 may also include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input output interfaces 358 , and/or, one or more operating systems 341 .

The steps in the file migration method described above can be implemented by the structure of the file migration device.

Corresponding to the above method embodiments, the embodiments of the present invention further provide a readable storage medium, and a readable storage medium described below and a file migration method described above may refer to each other correspondingly.

A readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the file migration method in the above method embodiment.

The readable storage medium may specifically be a USB flash drive, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, etc. that can store program codes. Readable storage medium.

Those skilled in the art may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, in order to clearly illustrate the hardware and software In the above description, the components and steps of each example have been generally described according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods of implementing the described functionality for each particular application, but such implementations should not be considered beyond the scope of the present invention.

Claims (8)

1. a file migration method, is characterized in that, comprises: Receive file migration tasks; Determine whether the file migration task is an unfinished task; If so, determine an invalid file from the aggregated large file in the destination storage pool corresponding to the migration task and each of the small files in the source storage pool; Aggregate and migrate the corresponding small files respectively; performing garbage collection on the invalid file; Invalid files are determined from the aggregated large files located in the destination storage pool and each of the small files located in the source storage pool corresponding to the file migration task, including: Obtain the aggregated attributes of each of the small files; Determine the invalid file from the aggregated large file and each of the small files by using the aggregated attribute; Wherein, using the aggregated attribute to determine the invalid file from the aggregated large file and each of the small files, including: Determine whether the aggregation attribute changes; If yes, then determine that the corresponding small file is the invalid file; If not, it is determined that the corresponding migrated small file stored in the aggregated large file is an invalid file. 2. The file migration method according to claim 1, characterized in that, performing garbage collection on the invalid files, comprising: If the invalid file is the small file, delete the small file in the source storage pool; If the invalid file is the migrated small file, perform fragment recovery processing on the aggregated large file. 3. The file migration method according to claim 2, characterized in that, performing fragment recovery processing on the aggregated large file, comprising: obtain the file header information of the aggregated large file; Using the file header information, determine the proportion of invalid data in the aggregated large file; In the case that the invalid data ratio is greater than the threshold, the valid data in the aggregated large file is migrated to the target aggregated large file and the aggregated large file is deleted. 4. The file migration method according to claim 2, characterized in that, performing fragment recovery processing on the aggregated large file, comprising: obtaining the aggregated attributes corresponding to each file in the aggregated large file; Determining the files whose aggregated attributes have not changed as invalid data, and count the proportion of invalid data; In the case that the invalid data ratio is greater than the threshold, the valid data in the aggregated large file is migrated to the target aggregated large file and the aggregated large file is deleted. 5. The file migration method according to any one of claims 1 to 4, wherein the receiving a file migration task comprises: receiving the file migration task sent by the metadata server; Correspondingly, when the garbage collection of the invalid file is completed, the method further includes: Feedback cleanup response data to the metadata server. 6. A file migration device, characterized in that, comprising: The task receiving module is used to receive file migration tasks; a judging module for judging whether the file migration task is an unfinished task; an invalid file determination module, configured to determine invalid files from the aggregated large files located in the destination storage pool and each of the small files located in the source storage pool corresponding to the file migration task if the judgment result is yes; the aggregated The large file has migration small files corresponding to each of the small files by aggregation; a garbage collection processing module for performing garbage collection on the invalid files; an invalid file determination module, specifically configured to obtain the aggregated attributes of each of the small files; using the aggregated attributes to determine the invalid file from the aggregated large file and each of the small files; wherein, using the Aggregate attributes, from the aggregated large file and each of the small files, determine the invalid file, including: Determine whether the aggregation attribute changes; If yes, then determine that the corresponding small file is the invalid file; If not, it is determined that the corresponding migrated small file stored in the aggregated large file is an invalid file. 7. A file migration device, comprising: memory for storing computer programs; The processor is configured to implement the steps of the file migration method according to any one of claims 1 to 5 when executing the computer program. 8. A readable storage medium, characterized in that, a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the file migration method according to any one of claims 1 to 5 is implemented. step.

Images