CN117806528A

CN117806528A - Data storage method and device

Info

Publication number: CN117806528A
Application number: CN202211178062.2A
Authority: CN
Inventors: 肖庚; 李林波
Original assignee: Huawei Cloud Computing Technologies Co Ltd
Current assignee: Huawei Cloud Computing Technologies Co Ltd
Priority date: 2022-09-23
Filing date: 2022-09-23
Publication date: 2024-04-02
Also published as: WO2024061212A1

Abstract

The application discloses a data storage method and device, which are used for saving resources of a storage system and ensuring the performance of the storage system. The method may be applied to a storage system that includes a plurality of magnetic tapes. The method comprises the following steps: and writing the n fragments obtained based on the verification method into n magnetic tapes of a target partition in a plurality of partitions, wherein each partition comprises n magnetic tapes in the plurality of magnetic tapes, and each magnetic tape in the plurality of magnetic tapes is divided into one partition at most. When determining to delete the target fragments and recovering the target tape where the target fragments are located, even if other fragments are migrated and marked for deletion, the tape where the other fragments are located can be limited to one partition (namely the target partition), so that the data migration operation of the storage system is reduced.

Description

Data storage method and device

Technical Field

The present disclosure relates to the field of storage technologies, and in particular, to a data storage method and apparatus.

Background

Magnetic tape has advantages such as low cost and long storage time, and is generally used as a storage medium for deep filing. In order to improve data reliability, a storage system generally writes data to be stored into data fragments in a data block, generates check fragments in the data block based on a check technique, and then writes the data fragments and the check fragments in the data block to a plurality of magnetic tapes, respectively. When a slice is lost, the target slice may be reconstructed from other slices in the data block (referred to as associated slices).

Since the tape does not support partial space reclamation, after a user deletes a fragment written to the tape, only that fragment in the tape can be marked as deleted (or invalidated), reducing the utilization of the tape. When the utilization rate of the magnetic tape is low, the storage system moves the effective data in the magnetic tape and recovers the whole storage space of the magnetic tape, and at the moment, the marked deleted fragments can be truly deleted.

When the storage system moves and deletes the associated fragments in other magnetic tapes in order to ensure the reliability of the associated fragments, the utilization rate of the magnetic tape (called the associated magnetic tape) where the associated fragments are located is reduced, so that a large amount of magnetic tapes are recovered, the storage system needs to execute a large amount of data moving operation, the resources of the storage system are wasted, and the performance of the storage system is reduced.

Disclosure of Invention

The application provides a data storage method and device, even if a storage system carries out relocation and mark deletion on an associated fragment deleted from the fragment, the data relocation operation of the storage system is reduced, the resources of the storage system are saved, and the stability of the performance of the storage system is ensured.

In a first aspect, a data storage method is provided, applicable to a storage system, the method comprising: and obtaining a target data block, wherein the target data block comprises n fragments, the n fragments comprise data fragments and check fragments, and n is a positive integer. And writing the n pieces of tapes into n magnetic tapes of a target partition, wherein the target partition is one partition of a plurality of partitions, each partition comprises n magnetic tapes of the plurality of magnetic tapes, and each magnetic tape of the plurality of magnetic tapes is divided into at most one partition. Optionally, the minimum unit of reclaiming storage space in the plurality of magnetic tapes is at least one magnetic tape of the plurality of magnetic tapes.

Assuming that a target tile of the n tiles is written to a target tape of the target partition, when it is determined to delete the target tile, the target tile may be marked as deleted or invalid, and the utilization of the target tape is reduced. When a target tape is recovered, if the storage system moves and marks and deletes the associated fragment of the target fragment in order to ensure the reliability of other fragments (referred to as the associated fragment of the target fragment) in the target data block, the utilization rate of the tape where the associated fragment is located (referred to as the associated tape of the target fragment) is reduced.

The target tape typically includes a plurality of target fragments that are marked for deletion, and reclaiming the target tape reduces the utilization of the associated tape of each target fragment. Since all the fragments in the same data block are stored in the same partition, even if the storage system moves and deletes the associated fragments of the target fragment, the tape affected by reclaiming the target tape will be limited to the partition where the target tape is located. Because the target tape is divided into one partition (i.e. the target partition) at most, the tape influenced by the recovery target tape is limited in a single partition (i.e. the target partition), so that the number of the tapes influenced by the recovery target tape is reduced, the data relocation operation of the storage system is reduced, and the stability of the performance of the storage system is ensured.

Optionally, the method further comprises: when determining to delete the target fragments in the n fragments, marking the target fragments stored in the target partition as invalid data, and updating the utilization rate of the target partition, wherein the utilization rate of the target partition is related to the total amount of invalid data in the target partition. And recycling the target partition according to the utilization rate of the target partition. And the method and the device are used for recycling all the magnetic tapes in the target partition according to the utilization rate of the target partition, which is equivalent to lifting the minimum unit of recycling storage space in the plurality of magnetic tapes to one partition, and recycling corresponding magnetic tapes does not need to be respectively instructed according to the utilization rate of each magnetic tape, so that recycling efficiency is improved.

Optionally, the target partition is a partition selected for the target data block from a plurality of partitions.

The storage system may include a storage server and n storage devices, with different tapes for each partition deployed in different storage devices. Alternatively, the storage server may select a target partition for a target data block from a plurality of partitions, and then send a storage instruction to the n storage devices, the storage instruction carrying one of the fragments of the target data block, and optionally, the storage instruction further indicating that the fragment is stored in a tape of the target partition. Optionally, the target partition is a partition randomly selected for the target data block from among the plurality of partitions.

Optionally, the target partition is determined from the plurality of partitions according to information of the target data block.

Optionally, the information of the target data block includes a lifecycle (e.g., a lifetime) of the target data block. In this way, the data blocks with the same or close life cycle can be written in the partition, and when the partition is reclaimed, the data quantity of the data which is not marked as invalid data is smaller, so that the overhead of moving (or backing up) the valid data can be reduced.

Optionally, the information of the target data block includes an identification of the target data block. After each storage device receives the storage instruction of the storage server, the fragments can be written into the magnetic tape of the target partition according to the identification of the data block to which the fragments belong, the storage server does not need to carry the identification of the target partition in the storage instruction, and the dependence of the storage device on the storage server is reduced, so that the load of the storage server is reduced.

Optionally, the information of the target data block includes an identification of the target data block and/or a life cycle of the target data block.

Optionally, the data slices in the n slices are a part of data selected from all data written according to the life cycle of the data. The deleted objects are limited to fewer partitions, and the overhead of relocation operation in the recycling process is reduced.

Optionally, the plurality of magnetic tapes are magnetic tapes selected from a storage cluster according to operation state information, wherein the operation state information indicates an operation state of each magnetic tape in the storage cluster, and the operation state indicates an available state or an unavailable state. In this way, it is advantageous to avoid storing fragments in a target data block to an unavailable tape, resulting in write failure or reduced write efficiency

In a second aspect, there is provided a data storage device comprising: the acquisition module is used for acquiring a target data block, wherein the target data block comprises n fragments, the n fragments comprise data fragments and check fragments, and n is a positive integer. And the storage module is used for writing the n fragments into n magnetic tapes of a target partition, wherein the target partition is one partition of a plurality of partitions, each partition comprises n magnetic tapes of the plurality of magnetic tapes, and each magnetic tape of the plurality of magnetic tapes is divided into one partition at most. Optionally, the minimum unit of reclaiming storage space in the plurality of magnetic tapes is at least one magnetic tape of the plurality of magnetic tapes.

Optionally, the apparatus further comprises a recycling module, the recycling module is used for: when determining to delete the target fragments in the n fragments, marking the target fragments stored in the target partition as invalid data, and updating the utilization rate of the target partition, wherein the utilization rate of the target partition is related to the total amount of the invalid data in the target partition; and recycling the target partition according to the utilization rate of the target partition.

Optionally, the data slices in the n slices are a part of data selected from all data written according to the life cycle of the data.

Optionally, the plurality of magnetic tapes are magnetic tapes selected from a storage cluster according to operation state information, wherein the operation state information indicates an operation state of each magnetic tape in the storage cluster, and the operation state indicates an available state or an unavailable state.

In a third aspect, a cluster of computing devices is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform a method as described in the first aspect or any one of the possible implementations of the first aspect.

In a fourth aspect, there is provided a computer program product comprising instructions which, when executed by a cluster of computing devices, cause the cluster of computing devices to perform a method as described in the first aspect or any one of the possible implementations of the first aspect.

In a fifth aspect, there is provided a computer readable storage medium comprising computer program instructions which, when executed by a cluster of computing devices, perform a method as described in the first aspect or any one of the possible implementations of the first aspect.

In a sixth aspect, there is provided a chip system comprising a processor and interface circuitry, the processor being coupled with a memory through the interface circuitry, the processor being for executing program code in the memory to perform the method described in the first aspect or any one of the possible implementations of the first aspect. The chip system may be formed of a chip or may include a chip and other discrete devices.

In a seventh aspect, a storage system is provided that includes a plurality of tapes or storage clusters that include at least a plurality of tapes. The storage system further comprises a data storage arrangement as provided in the second aspect or a cluster of computing devices as provided in the third aspect or a chip system as provided in the sixth aspect.

Because each device provided in the present application may be used to execute the corresponding method, the technical effects obtained by each device in the present application may refer to the technical effects obtained by the corresponding method, and will not be described herein.

Drawings

FIG. 1 schematically illustrates one possible configuration of a storage system;

FIG. 2 schematically illustrates a process by which a storage server distributes write requests to write to a tape pool for a plurality of objects;

3-1 through 3-2 schematically illustrate a process in which a storage server deletes at least one object written according to a delete request;

FIGS. 4-1 and 4-2 schematically illustrate a flow of a data storage method;

FIG. 5-1 schematically illustrates the result of storing data blocks 1 through 3 according to the method illustrated in FIG. 4-1;

FIG. 5-2 schematically illustrates the reclamation of the effect of the tape 1 shown in FIG. 5-1 on other tapes;

FIG. 6 schematically illustrates one possible configuration of a data storage device;

FIG. 7 illustrates one possible architecture of a computing device;

fig. 8 illustrates one possible architecture of a cluster of computing devices.

Detailed Description

Fig. 1 schematically shows one possible storage system to which the present application is applicable. The storage system includes a storage server and a storage cluster. Without limiting the number and types of storage devices in a storage cluster, FIG. 1 illustrates the storage cluster as a tape pool (pool), by way of example and not limitation. The tape pool includes a plurality of racks (racks), each rack including a server (server) and tape libraries (tape library), each tape library including a plurality of tapes (tape). Fig. 1 illustrates a tape pool comprising 6 racks, and the tape pool may comprise more or fewer racks.

The storage server may retrieve the write request data. The write request data is not limited to the data indicated to be written by one write request, and optionally, the write request data includes the data indicated to be written by each of the one or more write requests. The type of write request data is not limited. Alternatively, the write request data may include one or more objects, or, alternatively, one or more files. The description will be given hereinafter taking an example in which the write request data includes a plurality of objects.

The storage server may also write the write request data to the tape pool. Alternatively, the storage server may write the write request data into one or more data blocks by adding a write mode to the data blocks (chunk), and then write the written data blocks into the tape pool. The size of the write request data is not limited, and the write request data can be written with 3 data blocks.

In order to improve the reliability of the write request data, for each data block, the storage server may be configured to slice the data in the data block according to a certain verification technique, obtain k data slices, generate m verification slices for the k data slices, and then write the k data slices and the m verification slices into the tape pool in a distributed manner. Wherein k and m are positive integers. The type of verification technique is not limited and may alternatively be an erasure code technique or a duplicate technique. The erasure coding technique with the matching ratio of 4+2 is taken as an example.

Taking k and m as 4 and 2, respectively, for each data block, the storage server may write 4 data slices and 2 parity slices into 6 racks, respectively. After the server of each rack receives the data fragments or check fragments sent by the storage server, the data fragments or check fragments can be brushed into the tape library.

FIG. 2 schematically illustrates a process by which a storage server distributes write requests to write to a tape pool for a plurality of objects.

Referring to fig. 2, after receiving a write request, the storage server writes a plurality of objects written with write request instructions into data blocks 1 to 3 in a manner of data block additional writing, each data block is represented by a rectangular area including 6 small rectangles in fig. 2, and the 6 small rectangles in each rectangular area represent 4 data fragments and 2 check fragments in sequence from left to right. For convenience of description, 6 blocks in the data block i are respectively referred to as a slice i_1, a slice i_2, a slice … …, and a slice i_6, wherein i is 1 or 2 or 3, the slices i_1 to i_4 are data slices, and the slices i_5 and i_6 are check slices.

Referring to fig. 2, the storage server transmits data slice 1_1, data slice 2_1, and data slice 3_1 to a server (denoted as server 1) of the rack 1, respectively. The server 1 temporarily stores the received data fragments 1_1, 2_1 and 3_1 in a buffer (not shown in fig. 2), and then brushes the data fragments 1_1, 2_1 and 3_1 into the magnetic tape 1 of the magnetic tape library 1, the magnetic tape 1 of the magnetic tape library 1 and the magnetic tape 2 of the magnetic tape library 1 according to the running state of each magnetic tape in the magnetic tape library (denoted as the magnetic tape library 1) of the rack 1. The operational status of the tape may indicate that the tape is in an available state or an unavailable state. The server 1 may write the fragments to the tape with the operational state being available.

Alternatively, the usable state refers to a writable state, and when the writable storage space in the magnetic tape is large or the magnetic tape is operating normally, the running state of the magnetic tape may be the usable state. Alternatively, the unavailable state refers to an unavailable state, and the operating state of the magnetic tape may be an unavailable state when the writable storage space in the magnetic tape is too small or the magnetic tape fails.

With continued reference to fig. 2, the storage server sends data shard 1_2, data shard 2_2, and data shard 3_2, respectively, to the servers of rack 2 (denoted as server 2). The server 2 temporarily stores the received data fragments 1_2, 2_2, and 3_2 in a buffer (not shown in fig. 2), and then brushes the data fragments 1_2, 2_2, and 3_2 into the magnetic tape 2 of the magnetic tape library 2, the magnetic tape 3 of the magnetic tape library 2, and the magnetic tape 3 of the magnetic tape library 2, respectively, according to the operation states of the respective magnetic tapes in the magnetic tape library (denoted as the magnetic tape library 2) of the rack 2.

Similarly, the server of rack 3 (denoted as server 3) flushes data shard 1_3, data shard 2_3, and data shard 3_3 into the tape library of rack 3 (denoted as tape library 3), … …, and the server of rack 5 (denoted as server 5) flushes data shard 1_5, data shard 2_5, and data shard 3_5 into the tape library of rack 5 (denoted as tape library 5).

With continued reference to fig. 2, the storage server sends data shard 1_6, data shard 2_6, and data shard 3_6, respectively, to the servers of rack 6 (denoted as server 6). The server 6 temporarily stores the received data fragments 1_6, 2_6 and 3_6 in a buffer (not shown in fig. 2), and then brushes the data fragments 1_6, 2_6 and 3_6 into the magnetic tape 1 of the magnetic tape library 6, the magnetic tape 2 of the magnetic tape library 6 and the magnetic tape 3 of the magnetic tape library 6 according to the running state of each magnetic tape in the magnetic tape library (denoted as the magnetic tape library 6) of the rack 6.

The storage server may support not only writing one or more objects to the tape pool, but may also support deleting some or all of the written objects. Fig. 3-1 to 3-2 schematically illustrate a process in which a storage server deletes at least one object written according to a deletion request.

Referring to fig. 3-1, assume that the storage server receives a delete request indicating deletion of objects 1 to 3. The storage server determines that object 1 includes slice 1_1, object 2 includes slice 2_1, and object 3 includes slice 3_2, respectively, based on the metadata.

Because the tape does not support partial space reclamation, the storage system cannot actually delete the fragments 1_1, 2_1 and 3_2 in the tape, but marks to delete the fragments 1_1, 2_1 and 3_2, which reduces the utilization of the tape where the fragments are located. The fragments of fig. 3-1 in the dark background represent fragments that are marked for deletion (or marked as invalid data), and referring to fig. 3-1, the utilization of the magnetic tape 1 of the magnetic tape library 1 and the magnetic tape 1 of the magnetic tape library 2 is reduced.

When the tape is wholly recovered, the fragments marked for deletion in the tape can be truly deleted, and the storage space occupied by the fragments can be in a usable state again. Referring to fig. 3-2, assuming that the tape 1 of the tape library 1 is reclaimed, the marked deleted fragments (e.g., fragments 1_1 and fragments 2_1) in the tape 1 of the tape library 1 will be truly deleted, potentially affecting the reconstruction of other fragments in the data block in which they are located.

For ease of description, the reclaimed tape is referred to as a target tape, each shard marked for deletion in the target tape is referred to as a target shard, other shards in the same data block as the target shard are referred to as associated shards of the target shard, and the tape storing the associated shards of the target shard is referred to as associated tape of the target shard. In order to ensure the reliability of data, the storage system needs to move the fragments which are not marked for deletion in the target tape, and move and mark for deleting the associated fragments of the target fragments, so that the utilization rate of the tape where the associated fragments are located is reduced. Alternatively, the storage system may move the data fragments without moving the check fragments. The storage system may write the migrated data fragment as the data to be written to the tape pool again in the storage manner described above.

Referring to fig. 3-2, the storage system may also include other storage devices in addition to tape pools to store the migrated associated shards. Fig. 3-2 exemplifies other storage devices as a cluster of hard disks, which may include one or more hard disks. Referring to fig. 3-2, when the tape 1 of the tape library 1 is recovered, the storage system needs to migrate and mark delete the fragments other than the fragment 1_1 in the data block 1, migrate and mark delete the fragments other than the fragment 2_1 in the data block 2, and the tape utilization of the fragments marked for deletion is reduced.

Referring to fig. 3-1 and 3-2, the target shards marked for deletion (e.g., shards 1_1 and 2_1) may not only decrease the utilization of the target tape (e.g., tape 1 of tape library 1) where they are located, but may also decrease the utilization of the associated tape when the target tape is retrieved. For convenience of description, other magnetic tapes whose utilization rate decreases due to the reclamation of the target magnetic tape are referred to as magnetic tapes affected by the reclamation of the target magnetic tape. Assuming that the fragments in data block 1 and data block 2 are stored on different tapes except for fragments 1_1 and fragments 2_1, then the tape affected by tape 1 of reclamation tape library 1 comprises at least 10 tapes.

Further, referring to fig. 3-2, since the magnetic tape 3 of the magnetic tape library 2 further includes the marked-deleted fragments 3_2, when the magnetic tape 3 of the magnetic tape library 2 is recovered, in order to ensure the reliability of the data block 3, the storage system needs to move and mark-delete other fragments except for the fragments 3_2 in the data block 3, and the utilization rate of the magnetic tape where the marked-deleted fragments are located is reduced. Similarly, reclaiming the tape affected by the target tape may spread across all of the tapes in the entire tape pool, requiring a large number of shards to be migrated to the hard disk cluster and rewritten to the tape pool, wasting storage and computing resources of the storage system.

In order to reduce the impact of reclaiming a target tape, the present application provides a data storage method. The method may be applied to the memory system shown in fig. 4-1.

Referring to fig. 4-1, the storage system includes a storage server and a storage cluster including a plurality of magnetic tapes. The type of each tape is not limited as long as the minimum unit of storage space recovered among the plurality of tapes is at least one of the plurality of tapes. Alternatively, the storage system may be the storage system shown in FIG. 1, the storage cluster being the tape pool shown in FIG. 1, and the single tape being a single or multiple tapes in the tape pool. Optionally, the storage system shown in fig. 4-1 further comprises other means, for example, the storage cluster further comprises one or more servers, each server being, for example, a server shown in fig. 1, for example, the storage system further comprises other storage devices (e.g., a hard disk cluster as shown in fig. 3-2) outside the storage cluster.

Referring to fig. 4-1, the data storage method includes steps S401 and S402.

S401, acquiring a target data block;

the storage system may obtain a target data block that includes n slices that include a data slice and a parity slice (or redundancy slice), where n is a positive integer greater than 1. Optionally, the check fragment is obtained by encoding the data fragment by using a check technology (or redundancy coding technology), for example, an erasure code technology or a copy technology. Taking the erasure coding technique with the matching of k+m as an example, where k and m are positive integers, n=k+m, and n slices include k data slices and m check slices.

In some examples, other devices outside of the storage system generate and send target data blocks to the storage server, e.g., a client or application server generates and sends target data blocks to the storage server.

Alternatively, in some examples, the storage system generates the target data block from the write request. Alternatively, referring to fig. 4-1, step S401 includes S4011 and S4012.

S4011, receiving a write request;

the storage system may receive a write request carrying data indicative of writing (referred to as write request data). Optionally, the write request may include one or more write requests, the write request data including data that each of the one or more write requests indicates to write.

The type of write request data is not limited. Alternatively, the write request data may include one or more objects, or include one or more files, etc. The description will be given hereinafter taking an example in which the write request data includes a plurality of objects.

Without limiting the source of the write request data, a client or application server may send a write request to a storage server to indicate that multiple objects are stored in a storage cluster, for example, or the write request is generated by the storage server itself, for example.

S4012, generating a target data block according to the write request;

after the storage system receives the write request, a target data block may be generated from the write request. Optionally, the target data block includes some or all of the write request data. In some examples, the storage system writes a portion of the write request data to the target data block and writes other data to other data blocks other than the target data block.

The method of generating the target data block by the storage system according to the write request is not limited, alternatively, as in the storage process shown in fig. 2, the storage system may write a part or all of the data in the write request data into the data slice in the target data block by adopting the data block additional write method, and then generate the check slice of the data slice based on the check technology. The target data block may be, for example, data block 1 or data block 2 or data block 3 shown in fig. 2, or the like.

S402, respectively writing n fragments into n magnetic tapes of a target partition;

the plurality of tapes in the storage cluster may be divided into a plurality of partitions, each partition comprising n of the plurality of tapes, and each tape being divided into at most one partition. Taking n as 6 as an example, fig. 4-1 schematically illustrates a plurality of partitions.

After the storage system acquires the target data block, n fragments in the target data block can be written into n magnetic tapes of the target partition respectively, so that the reliability of the target data block is improved. For example, when a shard in a certain tape is lost, the storage system may read other shards of the target data block from other tapes in the target partition, reconstruct the lost shard based on a parity technique.

When the storage system acquires a plurality of data blocks, the storage system may execute steps S401 and S402 with each data block as a target data block. It should be noted that, the target partition is not limited to one fixed partition among the plurality of partitions, for example, partition 1 is taken as an example of the target partition in fig. 4-1, and the storage system may select one partition from the plurality of partitions as the target partition for each data block. Thus, different blocks of data may be written to the same partition or to different partitions of the plurality of partitions. The manner in which the storage system selects a target partition for a target data block will be described later, and will not be described herein.

The plurality of magnetic tapes are not limited to all magnetic tapes in the storage cluster, and optionally, the plurality of magnetic tapes are determined from the storage cluster according to the running state information, so that the situation that the fragments in the target data block are stored to the unavailable magnetic tapes, so that writing failure is caused or writing efficiency is reduced is avoided. Wherein the operational status information indicates an operational status of each tape in the storage cluster, the operational status indicating an available status or an unavailable status. The meaning of the operation state may refer to the relevant content of the foregoing, and will not be described herein. Alternatively, the plurality of tapes is all or a portion of the tapes in the storage cluster that are available for operation.

The division manner of the plurality of partitions is not limited. Alternatively, to increase the reliability of the target data block, different tapes in each partition are provided in different storage devices. As shown in FIG. 1 in storage clusters, the different tapes in each partition are disposed in different tape libraries, e.g., each partition includes tapes as shown in Table 1.

TABLE 1

In some examples, the storage system stores partition information indicating a tape included in each of the plurality of partitions, and the user or the storage system sets the number of partitions and the tape included in each partition by configuring the partition information. Alternatively, the user or the storage system may configure partition information according to the running state information, which is beneficial to avoiding that the fragments in the target data block are stored to an unavailable tape, resulting in writing failure or reducing writing efficiency. The data form of the partition information is not limited, and for example, the partition information is shown in table 1.

In some examples, the partition information also indicates an operational status of each of the plurality of magnetic tapes, as shown in table 2. Alternatively, the storage system may update the operational status of each tape in the partition information based on the operational status information. In table 2, the usable state is represented by "OK", and the unusable state is represented by "UNOK". Referring to table 2, since the operation state of the magnetic tape 4 in the magnetic tape library 1 is the unavailable state, when the storage system selects a target partition for a target data block from a plurality of partitions, the plurality of partitions does not include the partition 4.

TABLE 2

Since a single tape is the smallest unit of reclamation of storage space in a storage cluster, when a first partition of a first tape in a target partition is deleted, it can only be marked that the first partition in that tape has been deleted or invalidated. Optionally, referring to fig. 4-2, the data storage method further includes steps S403 and S404.

S403, determining to delete the target fragment in the n fragments;

after the storage system writes the n fragments to the n magnetic tapes of the target partition, it may be determined to delete the target fragment of the n fragments.

Alternatively, the storage system may obtain an instruction indicating the deletion of the target tile to determine the deletion of the target tile. The instructions may be generated internally to the storage system or from other devices outside the storage system, such as a client or application server.

Optionally, step S403 includes S4031 and S4032.

S4031, receiving a deletion request;

the storage system may receive a delete request indicating that some or all of the write request data is deleted.

The source of the delete request is not limited, and alternatively, the application server or client sends the delete request to the storage system, or the delete request is generated internally in the storage system.

S4032, determining a target fragment in the target data block according to the deletion request;

the storage system may determine the target shard based on the delete request. For example, the delete request indicates that a part or all of the data deleted is written to the target tile in the target data block in step S401. Optionally, the storage system may also determine other shards based on the delete request.

Optionally, the storage system may determine the fragment corresponding to the deletion request according to metadata information, where the metadata information indicates a correspondence between the write request data and the fragment. Alternatively, the storage system generates data blocks 1 to 3 shown in fig. 2 from the write request data and updates the metadata information in step S401. The data structure of the metadata information is not limited, and for example, the metadata information may be as shown in table 3.

TABLE 3 Table 3

Object identification	Layout information of objects
		Object 1	Data block 1, offset 1, length 1
Object 2	Data block 2, offset 2, length 2
		Object 3	Data block 3, offset 3, length 3
……	……

S404, marking the target fragments in the target tape as invalid data;

assume in step S402 that the storage system writes a target shard to a target tape in a target partition. After the storage system determines to delete a target tile of the n tiles, the target tile in the target tape may be marked as invalid data (or marked as deleted).

Alternatively, the storage system may maintain deletion information including one or more deletion records, each deletion record indicating a deleted shard and a storage location of the shard. The storage system may mark the target shard as invalid data by adding a delete record corresponding to the target shard to the delete information.

When the whole storage space of the target tape is recovered, the target fragments in the target tape can be truly deleted. When the target tape is reclaimed, the reliability of other tiles in the target data block (i.e., the associated tiles of the target tile) is reduced. Therefore, when the target tape is recovered, the associated fragment of the target fragment needs to be migrated and marked for deletion, so that the utilization rate of the tape where the associated fragment is located (i.e., the associated tape of the target fragment) is reduced.

The target tape to be reclaimed typically includes a plurality of target segments that are marked for deletion, and reclaiming the target tape reduces the utilization of the associated tape of each target segment. Since all the fragments in the same data block are stored in the same partition, the tape affected by reclaiming the target tape is limited to the partition in which the target tape is located. Since the target tape is divided into at most one partition (i.e., target partition), it is advantageous to limit the tape affected by reclaiming the target tape to a single partition (i.e., target partition), thereby reducing the number of tapes affected by reclaiming the target tape.

Referring to fig. 5-1, it is assumed that the storage system writes data block 1 and data block 2 shown in fig. 2 to partition 1 shown in table 1 (i.e., tape 1 of tape library 1 through tape 1 of tape library 6), writes data block 3 to partition 2 shown in table 1 (i.e., tape 2 of tape library 1 through tape 2 of tape library 6), and marks partition 1_1, partition 2_1, and partition 3_2 as invalid data, respectively. Referring to fig. 5-2, when the magnetic tape 1 of the magnetic tape library 1 is recovered, the storage system carries out relocation and mark deletion of the fragments 1_2 to 1_6 and fragments 2_2 to 2_6, respectively. As shown in fig. 5-2, the tape 1 of the reclaimed tape library 1 is limited to the partition 1, and the number of affected tapes is 5, which is far less than 10 shown in fig. 3-2.

After the storage system marks the target shards stored in the target partition as invalid data, the utilization of the target tape may be updated, and when the utilization is low (e.g., below a threshold), the storage system may reclaim the target tape. The utilization of the tape is related to the total amount of data in the tape that is marked as invalid data. The specific formula of the utilization ratio of the magnetic tape is not limited, for example, the utilization ratio of the magnetic tape=effective data amount/total data amount, the effective data amount refers to the total size of other data than the invalid data in the magnetic tape, and the total data amount refers to the total size of all data stored in the magnetic tape.

Optionally, referring to fig. 4-2, the data storage method further includes step S405.

S405, recycling the target partition according to the utilization rate of the target partition.

After the storage system marks the target partition stored in the target partition as invalid data, the utilization of the target partition may be updated, where the utilization of the target partition is related to the total amount of data in the target partition marked as deleted. The specific determination method of the utilization rate of the partition is not limited, for example, the utilization rate of the target partition is the utilization rate of the tape with the lowest utilization rate in the target partition, or the utilization rate of the target partition=valid data amount/total data amount, the valid data amount refers to the total size of other data except for invalid data stored in all tapes in the target partition, and the total data amount refers to the total size of all data stored in all tapes in the target partition.

The specific method for determining the utilization rate of the target partition by the storage system is not limited, and optionally, after the storage system determines to delete the target partition, the partition (i.e., the target partition) where the target partition is located can be determined, so that the utilization rate of the target partition is updated. Optionally, the storage system maintains partition usage information, e.g., as shown in table 4, where the partition usage information indicates a partition utilization, and after the storage system determines a target partition in which the target partition is located, the partition usage information may be updated to update the partition utilization. The manner in which the partition in which the target slice (or the target data block in the target slice) is determined is described below is not described herein.

TABLE 4 Table 4

Partition identification	Utilization rate
		1	30％
2	50％
		3	80％
……	……

The storage system may reclaim the target partition based on the utilization of the target partition to reclaim storage space of all tapes in the target partition. With continued reference to fig. 5-2, after the tape 1 of the tape library 1 is recovered, the utilization of other tapes (i.e., the tapes 1 of the tape library 2 to the tape 1 of the tape library 6) in the same partition as the tape 1 of the tape library 1 is reduced, the storage system needs to update the utilization of each of the other tapes respectively, and instruct to recover the other tapes respectively, which results in low recovery efficiency.

In step S405, all the tapes in the target partition can be collectively recovered according to the utilization rate of the target partition, which is equivalent to lifting the minimum unit of the storage space recovered in the plurality of tapes to one partition, without respectively recovering the corresponding tapes according to the commands under the utilization rate of each tape, thereby improving the recovery efficiency.

The foregoing describes that when the storage system acquires a plurality of data blocks, the storage system may execute steps S401 and S402 with each data block as a target data block. Without limiting the target partition to be a fixed one of the plurality of partitions, the storage system may select one of the plurality of partitions as the target partition for each data block, and thus, different data blocks may be written to the same one of the plurality of partitions or to different ones of the plurality of partitions. The manner in which the storage system selects a target partition for a target data block is described below by way of example.

Optionally, the target partition is determined from a plurality of partitions based on information of the data block. After the storage system acquires the target data block, the target partition can be determined from the multiple partitions according to the information of the target data block. Optionally, the information of the target data block includes an identification of the target data block and/or a life cycle of the target data block, etc.

The storage system can determine the corresponding relation between the target data block and the target partition through a preset algorithm. Optionally, the identification PT of the target partition=the identification% pt_max of the target data block, wherein pt_max is a fixed integer value. For example, the plurality of partitions are identified as an arithmetic progression (partition identification shown in table 1) with a tolerance of 1, and pt_max is the partition identification having the largest partition identification among the plurality of partitions. Wherein, "%" may represent a remainder operation.

Alternatively, the lifecycle of the target data block may be determined from the lifecycle of the data fragments in the target data block. For example, the lifecycle of the target data block is the average result of the lifecycle of all the data fragments in the target data block, which may be an average or median, etc. In this way, the data blocks with the same or close life cycle can be written in the partition, and when the partition is reclaimed, the data quantity of the data which is not marked as invalid data is smaller, so that the overhead of moving (or backing up) the valid data can be reduced.

Optionally, the data blocks in the n slices are a part of data selected from all data written according to the lifecycle of the data. For example, in step S4012, the storage system may cache write request data (e.g., a plurality of objects) and update lifecycle information indicating a lifecycle of each object. The data format of the life cycle information is not limited, and the life cycle information is shown in table 5, for example.

TABLE 5

Object identification	Survival of objects
		Object 1	For 30 days
Object 2	For 30 days
		Object 3	For 30 days
Object 4	90 days
		……	……

Optionally, the storage system writes objects with the same or similar lifecycle into the same data block according to the lifecycle information, for example, writes object 1, object 2, and object 3 into the same data block (e.g., data block 1), which is beneficial to limit deleted objects to fewer partitions. Alternatively, the size of the data block may be the size of a storage space of a partition, for example, k+m is 4+2, each tape capacity is 12TB, the size of the data block is 12tb×6, and the write request data of 12tb×6 needs to be aggregated and written to the tape in a partition. Because the objects have the same or similar life cycle, the objects can be released simultaneously when the space is released, and the moving overhead is reduced.

The application also provides a data storage device, as shown in fig. 6, which comprises an acquisition module and a storage module. The obtaining module is configured to implement step S401, and/or implement steps S4011 and S4012, and/or implement step S403, and/or implement steps S4031 and S4032. The storage module is used for implementing step S402.

Optionally, the data storage device further includes a recycling module, where the recycling module is configured to implement step S404 and/or implement step S405

The acquisition module, the storage module and the recovery module can be realized by software or hardware. Illustratively, the implementation of the acquisition module is described next as an example of the acquisition module. Similarly, the implementation of the storage module and the reclamation module may refer to the implementation of the acquisition module.

Module as an example of a software functional unit, the acquisition module may comprise code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container, among others. Further, the above-described computing examples may be one or more. For example, the acquisition module may include code running on multiple hosts/virtual machines/containers. It should be noted that, multiple hosts/virtual machines/containers for running the code may be distributed in the same region (region), or may be distributed in different regions. Further, multiple hosts/virtual machines/containers for running the code may be distributed in the same availability zone (availability zone, AZ) or may be distributed in different AZs, each AZ comprising a data center or multiple geographically close data centers. Wherein typically a region may comprise a plurality of AZs.

Also, multiple hosts/virtual machines/containers for running the code may be distributed in the same virtual private cloud (virtual private cloud, VPC) or in multiple VPCs. In general, one VPC is disposed in one region, and a communication gateway is disposed in each VPC for implementing inter-connection between VPCs in the same region and between VPCs in different regions.

Module as an example of a hardware functional unit, the acquisition module may comprise at least one computing device, such as a server or the like. Alternatively, the acquisition module may be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (programmable logic device, PLD), etc. The PLD may be implemented as a complex program logic device (complex programmable logical device, CPLD), a field-programmable gate array (FPGA), a general-purpose array logic (generic array logic, GAL), or any combination thereof.

Multiple computing devices included in the acquisition module may be distributed in the same region or may be distributed in different regions. The plurality of computing devices included in the acquisition module may be distributed in the same AZ or may be distributed in different AZ. Likewise, multiple computing devices included in the acquisition module may be distributed in the same VPC, or may be distributed among multiple VPCs. Wherein the plurality of computing devices may be any combination of computing devices such as servers, ASIC, PLD, CPLD, FPGA, and GAL.

It should be noted that, in other embodiments, the obtaining module may be configured to perform any step in the method corresponding to fig. 4-1, or may be configured to perform any step in the method corresponding to fig. 4-2. The memory module may be used to perform any of the steps of the method corresponding to fig. 4-1 or may be used to perform any of the steps of the method corresponding to fig. 4-2. The steps that the acquisition module and the storage module are responsible for implementing may be specified as needed, and all functions of the data storage device are implemented by implementing at least one module of the acquisition module, the storage module, and the recovery module to implement different steps in the method corresponding to fig. 4-1 or fig. 4-2, respectively.

The present application also provides a computing device 700. As shown in fig. 7, the computing device 700 includes: bus 702, processor 704, memory 706, and communication interface 708. Communication between processor 704, memory 706, and communication interface 708 is via bus 702. Computing device 700 may be a server or a terminal device. It should be understood that the present application is not limited to the number of processors, memories in computing device 700.

Bus 702 may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, only one line is shown in fig. 7, but not only one bus or one type of bus. Bus 704 may include a path to transfer information between various components of computing device 700 (e.g., memory 706, processor 704, communication interface 708).

The processor 704 may include any one or more of a central processing unit (central processing unit, CPU), a graphics processor (graphics processing unit, GPU), a Microprocessor (MP), or a digital signal processor (digital signal processor, DSP).

The memory 706 may include volatile memory (RAM), such as random access memory (random access memory). The processor 704 may also include non-volatile memory (ROM), such as read-only memory (ROM), flash memory, a mechanical hard disk (HDD), or a solid state disk (solid state drive, SSD).

The memory 706 has stored therein executable program code that is executed by the processor 704 to implement the functions of at least one of the aforementioned acquisition module, storage module, and reclamation module, respectively, to implement the method corresponding to fig. 4-1 or the method corresponding to fig. 4-2, i.e., the memory 706 has stored thereon instructions for executing the method corresponding to fig. 4-1 or the method corresponding to fig. 4-2.

Communication interface 703 enables communication between computing device 700 and other devices or communication networks using a transceiver module such as, but not limited to, a network interface card, transceiver, or the like.

In some examples, computing device 700 may be, for example, a storage system or storage server as shown in fig. 1, or any one or more of servers 1-6.

The embodiment of the application also provides a computing device cluster. The cluster of computing devices includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop, notebook, or smart phone.

As shown in fig. 8, a cluster of computing devices includes at least one computing device 700. The same instructions for performing the method corresponding to fig. 4-1 or the method corresponding to fig. 4-2 may be stored in the memory 706 in one or more computing devices 700 in the computing device cluster.

In some possible implementations, portions of instructions for performing the method corresponding to fig. 4-1 or the method corresponding to fig. 4-2, respectively, may also be stored in the memory 706 of one or more computing devices 700 in the computing device cluster. In other words, a combination of one or more computing devices 700 may collectively execute instructions for performing the method corresponding to FIG. 4-1 or the method corresponding to FIG. 4-2.

It should be noted that the memory 706 in different computing devices 700 in the computing device cluster may store different instructions for performing part of the functions of the data storage device. That is, the instructions stored by the memory 706 in the different computing devices 700 may implement the functionality of at least one of the acquisition module, the storage module, and the reclamation module.

In some possible implementations, one or more computing devices in a cluster of computing devices may be connected through a network. Wherein the network may be a wide area network or a local area network, etc. Specifically, the connection to the network is made through a communication interface in each computing device.

Embodiments of the present application also provide a computer program product comprising instructions. The computer program product may be software or a program product containing instructions capable of running on a computing device or stored in any useful medium. The computer program product, when run on at least one computing device, causes the at least one computing device to perform the method corresponding to fig. 4-1 or the method corresponding to fig. 4-2.

Embodiments of the present application also provide a computer-readable storage medium. The computer readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk), etc. The computer-readable storage medium includes instructions that instruct a computing device to perform a method corresponding to fig. 4-1 or a method corresponding to fig. 4-2.

The terms "first," "second," "third," "fourth" and the like in the description and in the claims of this application and in the above-described figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and are merely illustrative of the manner in which the embodiments of the application described herein have been described for objects of the same nature. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. The term "plurality" as used in the embodiments of the present application refers to two or more. It should be understood that the term "and/or" is merely an association relationship describing the associated object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; these modifications or substitutions do not depart from the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of data storage, comprising:

obtaining a target data block, wherein the target data block comprises n fragments, the n fragments comprise data fragments and check fragments, and n is a positive integer;

and writing the n pieces of tapes into n magnetic tapes of a target partition, wherein the target partition is one partition of a plurality of partitions, each partition comprises n magnetic tapes of the plurality of magnetic tapes, and each magnetic tape of the plurality of magnetic tapes is divided into at most one partition.

2. The method according to claim 1, wherein the method further comprises:

when determining to delete the target fragments in the n fragments, marking the target fragments stored in the target partition as invalid data, and updating the utilization rate of the target partition, wherein the utilization rate of the target partition is related to the total amount of the invalid data in the target partition;

And recycling the target partition according to the utilization rate of the target partition.

3. The method of claim 1 or 2, wherein the target partition is determined from the plurality of partitions based on information of the target data block.

4. A method according to claim 3, characterized in that the information of the target data block comprises an identification of the target data block and/or a lifecycle of the target data block.

5. The method of any of claims 1 to 4, wherein a data slice of the n slices is a portion of data selected from all data written according to a data lifecycle indication.

6. The method of any of claims 1-5, wherein the plurality of magnetic tapes are magnetic tapes selected from a storage cluster according to operational status information, the operational status information indicating an operational status of each magnetic tape in the storage cluster, the operational status indicating an available status or an unavailable status.

7. A data storage device, comprising:

the acquisition module is used for acquiring a target data block, wherein the target data block comprises n fragments, the n fragments comprise data fragments and check fragments, and n is a positive integer;

And the storage module is used for writing the n fragments into n magnetic tapes of a target partition, wherein the target partition is one partition of a plurality of partitions, each partition comprises n magnetic tapes of the plurality of magnetic tapes, and each magnetic tape of the plurality of magnetic tapes is divided into one partition at most.

8. The apparatus of claim 7, further comprising a recovery module for:

9. The apparatus of claim 7 or 8, wherein the target partition is determined from the plurality of partitions based on information of the target data block.

10. The apparatus of claim 9, wherein the information of the target data block comprises an identification of the target data block and/or a lifecycle of the target data block.

11. The apparatus according to any one of claims 7 to 10, wherein a data slice of the n slices is a portion of data selected from all data written according to a lifecycle of the data indicated by the write request.

12. The apparatus of any of claims 7 to 11, wherein the plurality of magnetic tapes are magnetic tapes selected from a storage cluster according to operational status information indicating an operational status of each magnetic tape in the storage cluster, the operational status indicating an available status or an unavailable status.

13. A cluster of computing devices, comprising at least one computing device, each computing device comprising a processor and a memory;

the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of claims 1 to 6.

14. A computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any of claims 1 to 6.

15. A computer readable storage medium comprising computer program instructions which, when executed by a cluster of computing devices, perform the method of any of claims 1 to 6.

16. A storage system comprising a plurality of tapes and the computing device cluster of claim 13.