CN111124304B - Data migration method and device, electronic equipment and storage medium - Google Patents
Data migration method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111124304B CN111124304B CN201911318556.4A CN201911318556A CN111124304B CN 111124304 B CN111124304 B CN 111124304B CN 201911318556 A CN201911318556 A CN 201911318556A CN 111124304 B CN111124304 B CN 111124304B
- Authority
- CN
- China
- Prior art keywords
- data
- determining
- hard disk
- disk drive
- storage medium
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The application discloses a data migration method, a data migration device, an electronic device and a computer readable storage medium, wherein the method comprises the following steps: determining a target bucket needing data migration based on the access heat of all buckets; wherein the target bucket comprises a nonvolatile multi-level cache medium and a hard disk drive; determining data to be migrated based on the read-write speed of the hard disk drive and the data heat of the data in the hard disk drive; and when the system time enters the protection period time period of the target storage bucket, migrating the data to be migrated to the multi-level cache medium. Therefore, the data migration method provided by the application can give full play to the performance of various storage media in the multi-plane storage structure, more reasonably distribute data on different media in the protection period, and improve the utilization rate of various storage media in the multi-plane storage structure.
Description
Technical Field
The present application relates to the field of storage technologies, and in particular, to a data migration method and apparatus, an electronic device, and a computer-readable storage medium.
Background
Due to the explosive increase of data, the demand on the capacity of the memory is more and more large, the benefits brought by increasing the capacity of the cache in the traditional method are reduced, and the frequent data exchange between the internal memory and the external memory generates larger expenses. A new type of Non-Volatile Memory (NVM) is gradually appearing in the field of view, and its characteristics of byte addressing, access speed close to that of Memory, and Non-volatility play a great role in improving the access efficiency of a file system. Based on this, a multi-plane storage structure is proposed in the related art, which can integrate NVM, SSD (Solid State Drive, hereinafter referred to as "Solid State Disk"), DRAM (Dynamic Random Access Memory "), HDD (Hard Disk Drive, hereinafter referred to as" Hard Disk Drive "), etc.
Therefore, how to improve the utilization rate of each storage medium in the multi-plane storage structure is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
The application aims to provide a data migration method, a data migration device, an electronic device and a computer readable storage medium, and the utilization rate of each storage medium in a multi-plane storage structure is improved.
In order to achieve the above object, the present application provides a data migration method, including:
determining a target bucket needing data migration based on the access heat of all buckets; wherein the target bucket comprises a nonvolatile multi-level cache medium and a hard disk drive;
determining data to be migrated based on the read-write speed of the hard disk drive and the data heat of the data in the hard disk drive;
and when the system time enters the protection period time period of the target storage bucket, migrating the data to be migrated to the multi-level cache medium.
Wherein, the determining a target bucket needing to be migrated based on the access heat of all buckets includes:
and calculating the access heat of each bucket, and determining the bucket with the access heat larger than a heat threshold value as a target bucket.
Wherein the determining data to be migrated based on the read-write speed of the hard disk drive and the data heat of the data in the hard disk drive comprises:
determining an access amount threshold corresponding to the hard disk drive according to the read-write speed of the hard disk drive, and determining the data heat of data in the hard disk drive;
judging whether first target data with the data heat degree larger than an access amount threshold corresponding to the hard disk drive exists in the hard disk drive;
if yes, determining the data to be migrated as the first target data;
and if not, re-entering the step of determining the access amount threshold corresponding to the hard disk drive according to the read-write speed of the hard disk drive.
Wherein the determining the data heat of the data in the hard disk drive comprises:
and determining the total download amount of the data in the hard disk drive in the first time window as the data heat of the data in the hard disk drive.
Wherein the determining the data heat of the data in the hard disk drive comprises:
determining the total download amount of data in the hard disk drive in a plurality of time windows, and determining a time attenuation coefficient corresponding to each time window;
and calculating the data heat of the data in the hard disk drive based on the total download amount in each time window and the time attenuation coefficient.
Before determining the data to be migrated based on the read-write speed of the hard disk drive and the access information of the data in the hard disk drive, the method further comprises the following steps:
determining whether the hard disk drive requires data migration based on a current occupied capacity of the hard disk drive;
if yes, executing the step of determining the data to be migrated based on the read-write speed of the hard disk drive and the access information of the data in the hard disk drive;
and if not, re-entering the step of determining the target bucket needing data migration based on the access heat of all the buckets.
Wherein the determining whether the hard disk drive requires data migration based on the current occupied capacity of the hard disk drive comprises:
determining a first capacity threshold based on a total capacity of the hard disk drive, determining whether the currently occupied capacity is greater than the first capacity threshold;
if yes, determining that the hard disk drive needs data migration.
Wherein migrating the data to be migrated to the multi-level cache medium includes:
determining a highest-level storage medium in the multi-level storage media as a target storage medium; wherein, in the multi-stage storage medium, the read-write speed of the storage medium is positively correlated with the level;
determining a target capacity threshold corresponding to the target storage medium based on the total capacity of the target storage medium;
judging whether the current occupied capacity of the target storage medium is larger than the target capacity threshold value;
if yes, re-determining a next-stage storage medium of the target storage medium as the target storage medium, and re-entering the step of determining a target capacity threshold corresponding to the target storage medium based on the total capacity of the target storage medium;
and if not, migrating the data to be migrated to the target storage medium.
Wherein, still include:
acquiring the anti-tampering attributes of all the buckets; wherein the tamper-resistant attribute comprises at least a guard period time period for the bucket;
and after the anti-tampering attribute takes effect, detecting whether the system time enters the protection period time period.
Wherein, still include:
determining the return data based on the read-write speed and the data heat of each storage medium in the multi-stage storage media;
and migrating the backlogged data to the hard disk drive when the system time enters the protection period of the target bucket.
Wherein, the determining the migration data based on the read-write speed and the data heat of each storage medium in the multi-level storage media comprises:
determining an access amount threshold corresponding to each storage medium according to the read-write speed of each storage medium in the multi-level storage media, and determining the data heat of data in the multi-level storage media;
judging whether second target data with the data heat degree smaller than the access amount threshold corresponding to the storage medium exists in each storage medium or not;
if yes, determining the backlogged data as the second target data;
and if not, re-entering the step of determining the access amount threshold corresponding to each storage medium according to the read-write speed of each storage medium in the multi-stage storage media.
Wherein the determining the data heat of the data in the multi-level storage medium comprises:
and determining the total download amount of the data in the multi-level storage medium in a second time window as the data heat of the data in the multi-level storage medium.
Wherein the determining the data heat of the data in the multi-level storage medium comprises:
determining the total download amount of data in the multi-level storage medium in a plurality of time windows, and determining a time attenuation coefficient corresponding to each time window;
and calculating the data heat of the data in the multi-stage storage medium based on the total download amount and the time attenuation coefficient in each time window.
To achieve the above object, the present application provides a data migration apparatus, including:
the first determining module is used for determining a target bucket needing data migration based on the access heat of all buckets; wherein the target bucket comprises a nonvolatile multi-level cache medium and a hard disk drive;
the second determining module is used for determining data to be migrated based on the read-write speed of the hard disk drive and the data heat of the data in the hard disk drive;
and the migration module is used for migrating the data to be migrated to the multi-level cache medium when the system time enters the protection period time period of the target storage bucket.
To achieve the above object, the present application provides an electronic device including:
a memory for storing a computer program;
a processor for implementing the steps of the data migration method when executing the computer program.
To achieve the above object, the present application provides a computer-readable storage medium having stored thereon a computer program, which when executed by a processor, implements the steps of the data migration method as described above.
According to the above scheme, the data migration method provided by the application comprises the following steps: determining a target bucket needing data migration based on the access heat of all buckets; wherein the target bucket comprises a nonvolatile multi-level cache medium and a hard disk drive; determining data to be migrated based on the read-write speed of the hard disk drive and the data heat of the data in the hard disk drive; and when the system time enters the protection period time period of the target storage bucket, migrating the data to be migrated to the multi-level cache medium.
The data migration method provided by the application is a data migration method in a target storage bucket, and the target storage bucket is of a multi-plane storage structure, namely comprises a nonvolatile multi-level cache medium and a hard disk drive. Because the capacity of the hard disk drive is large but the read-write speed is slow, in the data migration method provided by the application, the data to be migrated is determined based on the read-write speed of the hard disk drive and the data heat of the data, and is migrated to the multi-level cache medium within the protection period time period, i.e. the data with higher data heat is migrated to the multi-level cache medium with higher read-write speed, so as to improve the read-write efficiency of the data. Therefore, the data migration method provided by the application can give full play to the performance of various storage media in the multi-plane storage structure, more reasonably distribute data on different media in a protection period, improve the utilization rate of various storage media in the multi-plane storage structure, realize all heterogeneous hybrid storage pooling technologies of cluster nodes, uniformly manage and distribute data, reduce the access delay of the whole system, and achieve optimal performance. The application also discloses a data migration device, an electronic device and a computer readable storage medium, which can also achieve the technical effects.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure without limiting the disclosure. In the drawings:
FIG. 1 is a flow diagram illustrating a method of data migration in accordance with an exemplary embodiment;
FIG. 2 is a schematic cycle diagram of the tamper-resistant function;
FIG. 3 is a detailed flowchart of step S103 in FIG. 1;
FIG. 4 is a flow diagram illustrating another method of data migration in accordance with an illustrative embodiment;
FIG. 5 is a flow chart illustrating yet another method of data migration in accordance with an exemplary embodiment;
FIG. 6 is a block diagram illustrating a data migration apparatus in accordance with an exemplary embodiment;
FIG. 7 is a block diagram illustrating an electronic device in accordance with an exemplary embodiment.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The embodiment of the application discloses a data migration method, which improves the utilization rate of each storage medium in a multi-plane storage structure.
Referring to fig. 1, a flow chart of a data migration method is shown according to an exemplary embodiment, as shown in fig. 1, including:
s101: determining a target bucket needing data migration based on the access heat of all buckets; wherein the target bucket comprises a nonvolatile multi-level cache medium and a hard disk drive;
in this step, a target bucket that needs to perform data migration may be determined based on access heat of all buckets, where the access heat may be understood as an access amount of data in a bucket within a certain time period, and may also be understood as a weighted access amount of data within a plurality of time periods, that is, a corresponding weight is allocated to each time period, and a weighted average of the access amounts of the plurality of time periods is calculated. In a specific implementation, a bucket whose access heat is greater than the heat threshold may be determined as a target bucket, that is, this step includes: and calculating the access heat of each bucket, and determining the bucket with the access heat larger than a heat threshold value as a target bucket. It is understood that the heat threshold herein may be set according to actual situations, and is not particularly limited herein.
The target storage bucket in this embodiment is a multi-plane storage structure, that is, the target storage bucket includes a nonvolatile multi-level cache medium and a hard disk drive at a rear end, capacity of the multi-level cache medium increases sequentially from the front end to the rear end, and a read-write speed decreases sequentially. In the specific implementation, the single-node local DRAM is used as a primary storage medium, the NVM is used as a secondary storage medium, the NVM is used as a write cache of the DRAM, and the write operation is directly written into the DRAM as much as possible, so that the influence on the service life of the NVM is reduced under the condition of ensuring the high-speed read-write performance. Taking the SSD as a three-level storage medium and a high-speed storage pool for persistent storage, and taking the HDD as a four-level storage medium and a low-speed storage pool for persistent storage.
S102: determining data to be migrated based on the read-write speed of the hard disk drive and the data heat of the data in the hard disk drive;
the purpose of this step is to determine the data to be migrated in the hard disk drive for subsequent steps to migrate into the multi-level storage medium. Specifically, different hard disk drives have different read/write speeds, which results in different corresponding access amount thresholds, and the larger the read/write speed, the larger the corresponding access amount threshold corresponding to the hard disk drive is, the corresponding relationship between the read/write speed and the access amount threshold can be preset. And when data with the data heat degree larger than the access quantity threshold corresponding to the hard disk drive exists in the hard disk drive, determining the data as the data to be migrated. Namely, the method comprises the following steps: determining an access amount threshold corresponding to the hard disk drive according to the read-write speed of the hard disk drive, and determining the data heat of data in the hard disk drive; judging whether first target data with the data heat degree larger than an access amount threshold corresponding to the hard disk drive exists in the hard disk drive; if yes, determining the data to be migrated as the first target data; and if not, re-entering the step of determining the access amount threshold corresponding to the hard disk drive according to the read-write speed of the hard disk drive.
As a possible implementation, the determining the data heat of the data in the hard disk drive includes: and determining the total download amount of the data in the hard disk drive in the first time window as the data heat of the data in the hard disk drive. In specific implementation, the total download amount of all data in the hard disk drive in a first time window is collected as the data heat, and the first time window here can be flexibly set according to actual conditions, and is not specifically limited here.
As another possible implementation, the determining the data heat of the data in the hard disk drive includes: determining the total download amount of data in the hard disk drive in a plurality of time windows, and determining a time attenuation coefficient corresponding to each time window; and calculating the data heat of the data in the hard disk drive based on the total download amount in each time window and the time attenuation coefficient. In the specific implementation, the total download amount of all data in the hard disk drive in a plurality of time windows is collected, a time attenuation coefficient is set for each time window, which can be understood as a weight corresponding to the time window, and a weighted average of the download amount is calculated and is used as the data heat. For example, a weighted average of the download amount of a plurality of time windows, i.e. data heat:
wherein A isi＝nie-at+b，niIs the total download in the ith time window, j is the total number of time windows, e-at+bTime decay system for the time windowThe numbers, a and b, may be dynamically set based on experience or system access magnitude access latency.
S103: and when the system time enters the protection period time period of the target storage bucket, migrating the data to be migrated to the multi-level cache medium.
In a specific implementation, the operational authority of the target bucket is shown in fig. 2, and can be divided into three stages: the system comprises a delay protection period, a protection period and a non-protection period, wherein in the delay protection period, deletion, modification, check and download operations can be carried out. Only view and download operations are supported during the protection period and no delete or modify operations will be performed on the bucket or object. And the operations of deletion, modification, viewing and downloading can be carried out in the non-protection period. Therefore, the user only has the permission to delete and modify in the delayed protection period and the non-protection period of the target bucket. And after the protection period of the target storage bucket is ended, namely when the non-protection period is entered, closing the anti-tampering function. The protection period of the entire target bucket is the same, and when the protection period is over, the function can be reset and also can be closed. And when the system time enters the protection period time period of the target storage bucket, migrating the data to be migrated determined in the previous step to a multi-level cache medium.
As a preferred embodiment, this embodiment further includes: acquiring the anti-tampering attributes of all the buckets; wherein the tamper-resistant attribute comprises at least a guard period time period for the bucket; and after the anti-tampering attribute takes effect, detecting whether the system time enters the protection period time period. In a specific implementation, a user may set a tamper-resistant property of a bucket through a client, and when the system time enters the protection period of the target bucket, step S103 is performed.
The data migration method provided by the embodiment of the application is a data migration method in a target storage bucket, and the target storage bucket is of a multi-plane storage structure, namely comprises a nonvolatile multi-level cache medium and a hard disk drive. Because the capacity of the hard disk drive is large but the read-write speed is slow, in the data migration method provided in the embodiment of the present application, the data to be migrated is determined based on the read-write speed of the hard disk drive and the data heat of the data, and is migrated to the multi-level cache medium within the protection period time period, that is, the data with higher data heat is migrated to the multi-level cache medium with higher read-write speed, so as to improve the read-write efficiency of the data. Therefore, the data migration method provided by the embodiment of the application gives full play to the performance of various storage media in the multi-plane storage structure, more reasonably distributes data on different media in the protection period, improves the utilization rate of various storage media in the multi-plane storage structure, realizes all heterogeneous hybrid storage pooling technologies of cluster nodes, performs unified management and data distribution, reduces the access delay of the whole system, and achieves optimal performance.
In this embodiment, a specific data migration method will be described, that is, as shown in fig. 3, step S103 in the previous embodiment may include:
s31: determining a highest-level storage medium in the multi-level storage media as a target storage medium; wherein, in the multi-stage storage medium, the read-write speed of the storage medium is positively correlated with the level;
since the multi-level storage medium includes a plurality of nonvolatile storage media, the read/write speed of each storage medium is different. Therefore, when data migration is performed, it is necessary to determine to which storage medium the data to be migrated is to be migrated. In this embodiment, the data to be migrated is preferentially migrated to the storage medium with the fastest read/write speed.
In a specific implementation, each storage medium is classified based on the read-write speed of each storage medium, and the higher the read-write speed is, the higher the level is. In this step, the highest-level storage medium is first determined as the target storage medium, and the subsequent step determines whether the target storage medium satisfies the migration condition.
S32: determining a target capacity threshold corresponding to the target storage medium based on the total capacity of the target storage medium;
s33: judging whether the current occupied capacity of the target storage medium is larger than the target capacity threshold value; if yes, go to S34; if not, go to S35;
in this step, it is determined whether the target storage medium satisfies the migration condition based on its current occupied capacity. Different storage media correspond to different capacity thresholds and may be determined based on the total capacity of the storage media, for example, setting 80% of the total capacity as the capacity threshold of the storage media. When the current occupied capacity of the target storage medium is larger than the corresponding target capacity threshold value, the target storage medium does not meet the migration condition, the step S34 is entered, the next-level storage medium of the target storage medium is determined as the target storage medium again, when the current occupied capacity of the target storage medium is smaller than or equal to the corresponding target capacity threshold value, the target storage medium meets the migration condition, the step S35 is entered, and the data to be migrated is migrated to the target storage medium.
S34: re-determining the storage medium next to the target storage medium as the target storage medium, and re-entering step S32;
s35: and migrating the data to be migrated to the target storage medium.
Therefore, in the embodiment, the data to be migrated is migrated to the storage medium which meets the migration condition and has the highest read-write speed, and the data to be migrated is migrated to the storage medium which has the higher read-write speed due to the higher data heat of the data to be migrated, so that the read-write hit rate is improved and the read-write speed is improved.
The embodiment of the application discloses a data migration method, and compared with the first embodiment, the embodiment further explains and optimizes the technical scheme. Specifically, the method comprises the following steps:
referring to fig. 4, a flowchart of another data migration method according to an exemplary embodiment is shown, as shown in fig. 4, including:
s201: determining a target bucket needing data migration based on the access heat of all buckets; wherein the target bucket comprises a nonvolatile multi-level cache medium and a hard disk drive;
s202: determining whether the hard disk drive requires data migration based on a current occupied capacity of the hard disk drive; if yes, the process proceeds to S203: if not, the step S201 is re-entered;
in the present embodiment, whether it satisfies the migration condition is determined based on the current occupied capacity of the hard disk drive. Different hard disk drives correspond to different capacity thresholds, which may be determined based on the total capacity of the hard disk drive, for example, setting 80% of the total capacity as the capacity threshold of the hard disk drive. When the current occupied capacity of the hard disk drive is less than or equal to the corresponding first capacity threshold, it indicates that the hard disk drive meets the migration condition, and the process proceeds to S203. Namely, the method comprises the following steps: determining a first capacity threshold based on a total capacity of the hard disk drive, determining whether the currently occupied capacity is greater than the first capacity threshold; if yes, determining that the hard disk drive needs data migration.
S203: determining data to be migrated based on the read-write speed of the hard disk drive and access information of the data in the hard disk drive;
s204: and when the system time enters the protection period time period of the target storage bucket, migrating the data to be migrated to the multi-level cache medium.
Therefore, in the embodiment, the data to be migrated is determined based on the dual conditions of the capacity of the hard disk drive and the data heat of the data, and when the occupancy rate of the hard disk drive is too large, the data with the larger data heat can be migrated to the multilevel storage medium, so that the space of the hard disk drive is saved, and the read hit rate is improved.
The following describes a data migration method, specifically:
referring to FIG. 5, a flowchart of yet another data migration method is shown in accordance with an exemplary embodiment, as shown in FIG. 5, including:
s301: determining the return data based on the read-write speed and the data heat of each storage medium in the multi-stage storage media;
in this embodiment, specifically, different storage media have different read/write speeds, which results in different corresponding access amount thresholds, and the larger the access amount threshold corresponding to the hard disk drive with the higher read/write speed is, the corresponding relationship between the read/write speed and the access amount threshold may be preset. When data with the data heat degree larger than the access quantity threshold value corresponding to a certain storage medium exists in the certain storage medium, the certain storage medium is determined as the back migration data. Namely, the method comprises the following steps: determining an access amount threshold corresponding to each storage medium according to the read-write speed of each storage medium in the multi-level storage media, and determining the data heat of data in the multi-level storage media; judging whether second target data with the data heat degree smaller than the access amount threshold corresponding to the storage medium exists in each storage medium or not; if yes, determining the backlogged data as the second target data; and if not, re-entering the step of determining the access amount threshold corresponding to each storage medium according to the read-write speed of each storage medium in the multi-stage storage media.
As a possible implementation, the determining the data heat of the data in the multi-level storage medium includes: and determining the total download amount of the data in the multi-level storage medium in a second time window as the data heat of the data in the multi-level storage medium. In specific implementation, the total download amount of all data in the multi-level storage medium in the second time window is collected as the data heat, and the second time window can be flexibly set according to actual conditions, which is not specifically limited herein.
As another possible implementation, the determining the data heat of the data in the multi-level storage medium includes: determining the total download amount of data in the multi-level storage medium in a plurality of time windows, and determining a time attenuation coefficient corresponding to each time window; and calculating the data heat of the data in the multi-stage storage medium based on the total download amount and the time attenuation coefficient in each time window. In a specific implementation, the total download amount of all data in the multi-level storage medium in a plurality of time windows is collected, a time attenuation coefficient is set for each time window, which can be understood as a weight corresponding to the time window, and a weighted average of the total download amounts is calculated and used as the data heat. For example, a weighted average of the download amount of a plurality of time windows, i.e. data heat:
wherein A isi＝nie-at+b，niIs the total download in the ith time window, j is the total number of time windows, e-ct+dThe time decay coefficients for this time window, c and d, may be set dynamically based on experience or system access magnitude access delays.
S302: and migrating the backlogged data to the hard disk drive when the system time enters the protection period of the target bucket.
In this step, when the system time enters the protection period time period of the target storage bucket, the migration data determined in the previous step is directly migrated to the hard disk drive, so that the gradual migration of the migration data is avoided, and the migration efficiency is improved.
An application embodiment of the present application is introduced below, specifically, a single-node local DRAM is used as a primary storage medium, an NVM is used as a secondary storage medium, the NVM is used as a write cache of the DRAM, and write operations are directly written into the DRAM as much as possible, so that the influence on the service life of the NVM is reduced while ensuring high-speed read-write performance; taking the SSD as a three-level storage medium and a high-speed storage pool for persistent storage, and taking the HDD as a four-level storage medium and a low-speed storage pool for persistent storage.
The implementation principle of the anti-tampering function comprises the following steps:
the method comprises the following steps: a bucket is created.
Step two: tamper-resistant and related attributes are set for the bucket by the object storage client.
Step three: after the barrel is set with the anti-tampering attribute, the barrel is opened with the anti-tampering function, when the system time enters the protection period, a common user cannot upload the object into the barrel any more, cannot delete or modify the object in the barrel, and only can check or download the object. And the data in the barrel after the anti-tampering attribute is set is according to the size of the access amount counted. For data stored on the HDD, if the access amount is greater than an access amount threshold value N1 within a settable time T, judging whether the nonvolatile memory exceeds a storage alarm limit, and if not, migrating the data to the nonvolatile memory; and if the nonvolatile storage exceeds the storage alarm limit, inquiring whether the storage capacity in the SSD reaches an alarm value, if not, migrating to the SSD, otherwise, not migrating the data. For data on the nonvolatile memory, if the access amount is less than the adjustable set threshold N2 within the set time T2, migrating to the HDD; for data on the SSD, if the amount of access is less than the adjustable set threshold N3 within a set time T3, migrating to the HDD;
step four: in the delayed protection period and the non-protection period of the bucket, each user has the permission of deletion and modification. The data at this time is migrated in accordance with a general data migration method of the multi-plane system.
Step five: after the protection period of the barrel is finished, namely when the non-protection period is entered, the anti-tampering function is closed.
In the following, a data migration apparatus provided in an embodiment of the present application is introduced, and a data migration apparatus described below and a data migration method described above may be referred to each other.
Referring to fig. 6, a block diagram of a data migration apparatus according to an exemplary embodiment is shown, as shown in fig. 6, including:
a first determining module 601, configured to determine, based on access heat of all buckets, a target bucket that needs to perform data migration; wherein the target bucket comprises a nonvolatile multi-level cache medium and a hard disk drive;
a second determining module 602, configured to determine data to be migrated based on the read-write speed of the hard disk drive and the data heat of the data in the hard disk drive;
the migration module 603 is configured to migrate the data to be migrated to the multi-level cache medium when the system time enters the protected period time period of the target bucket.
The data migration device provided by the embodiment of the application is a data migration device in a target storage bucket, and the target storage bucket is of a multi-plane storage structure, namely comprises a nonvolatile multi-level cache medium and a hard disk drive. Because the capacity of the hard disk drive is large but the read-write speed is slow, in the data migration device provided in the embodiment of the present application, the data to be migrated is determined based on the read-write speed of the hard disk drive and the data heat of the data, and is migrated to the multi-level cache medium within the protection period time period, that is, the data with higher data heat is migrated to the multi-level cache medium with higher read-write speed, so as to improve the read-write efficiency of the data. Therefore, the data migration device provided by the embodiment of the application gives full play to the performance of various storage media in the multi-plane storage structure, and more reasonably distributes data on different media in the protection period, so that the utilization rate of various storage media in the multi-plane storage structure is improved, all heterogeneous hybrid storage pooling technologies of cluster nodes are realized, unified management and data distribution are realized, the access delay of the whole system is reduced, and the optimal performance is achieved.
On the basis of the foregoing embodiment, as a preferred implementation manner, the first determining module 601 is specifically a module that calculates a visit heat of each bucket, and determines a bucket with the visit heat greater than a heat threshold as a target bucket.
On the basis of the foregoing embodiment, as a preferred implementation, the second determining module 602 includes:
the first determining unit is used for determining an access amount threshold corresponding to the hard disk drive according to the read-write speed of the hard disk drive and determining the data heat of data in the hard disk drive;
the first judging unit is used for judging whether first target data with the data heat degree larger than an access amount threshold value corresponding to the hard disk drive exists in the hard disk drive or not; if yes, determining the data to be migrated as the first target data.
On the basis of the foregoing embodiment, as a preferred implementation manner, the first determining unit is specifically a unit that determines an access amount threshold corresponding to the hard disk drive according to a read-write speed of the hard disk drive, and determines a total download amount of data in the hard disk drive in a first time window as a data hot degree of the data in the hard disk drive.
On the basis of the above embodiment, as a preferred implementation, the first determining unit includes:
the first determining subunit is used for determining an access amount threshold corresponding to the hard disk drive according to the read-write speed of the hard disk drive;
the second determining subunit is used for determining the total download amount of the data in the hard disk drive in a plurality of time windows and determining a time attenuation coefficient corresponding to each time window;
and the first calculating subunit is used for calculating the data heat of the data in the hard disk drive based on the total download amount and the time attenuation coefficient in each time window.
On the basis of the above embodiment, as a preferred implementation, the method further includes:
the judging module is used for judging whether the hard disk drive needs data migration or not based on the current occupied capacity of the hard disk drive; if yes, the workflow of the second determining module 602 is started.
On the basis of the foregoing embodiment, as a preferred implementation manner, the determining module specifically determines a first capacity threshold based on the total capacity of the hard disk drive, and determines whether the currently occupied capacity is greater than the first capacity threshold; and if so, judging the module of the hard disk drive needing data migration.
On the basis of the foregoing embodiment, as a preferred implementation, the migration module 603 includes:
a second determining unit configured to determine a highest-level storage medium among the multi-level storage media as a target storage medium; wherein, in the multi-stage storage medium, the read-write speed of the storage medium is positively correlated with the level;
a third determining unit, configured to determine a target capacity threshold corresponding to the target storage medium based on the total capacity of the target storage medium;
a second judging unit, configured to judge whether a current occupied capacity of the target storage medium is greater than the target capacity threshold; if yes, re-determining the next-level storage medium of the target storage medium as the target storage medium, and re-starting the work flow of the third determination unit; if not, starting the working process of the migration unit;
the migration unit is configured to migrate the data to be migrated to the target storage medium.
On the basis of the above embodiment, as a preferred implementation, the method further includes:
the acquisition module is used for acquiring the anti-tampering attributes of all the buckets; wherein the tamper-resistant attribute comprises at least a guard period time period for the bucket;
and the detection module is used for detecting whether the system time enters the protection period time period or not after the anti-tampering attribute takes effect.
On the basis of the above embodiment, as a preferred implementation, the method further includes:
a third determining module, configured to determine migration data based on the read-write speed and the data heat of each storage medium in the multiple storage media;
and the migration module is used for migrating the migration data to the hard disk drive when the system time enters the protection period time period of the target storage bucket.
On the basis of the foregoing embodiment, as a preferred implementation, the third determining module includes:
a fourth determining unit, configured to determine, according to the read-write speed of each storage medium in the multiple storage media, an access amount threshold corresponding to each storage medium, and determine the data heat of data in the multiple storage media;
a second judging unit, configured to judge whether second target data whose data heat is smaller than an access amount threshold corresponding to the storage medium exists in each storage medium; and if so, determining the backlogged data as the second target data.
On the basis of the foregoing embodiment, as a preferred implementation manner, the fourth determining unit specifically determines, according to the read-write speed of each storage medium in the multi-level storage medium, an access amount threshold corresponding to each storage medium, and determines a total download amount of data in the multi-level storage medium in a second time window as a data heat degree of the data in the multi-level storage medium.
On the basis of the above embodiment, as a preferred implementation, the fourth determining unit includes:
the third determining subunit is used for determining the total download amount of the data in the multi-level storage medium in a plurality of time windows and determining a time attenuation coefficient corresponding to each time window;
and the second calculating subunit is used for calculating the data heat of the data in the multi-level storage medium based on the total download amount and the time attenuation coefficient in each time window.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
The present application further provides an electronic device, and referring to fig. 7, a structure diagram of an electronic device 700 provided in an embodiment of the present application may include a processor 11 and a memory 12, as shown in fig. 7. The electronic device 700 may also include one or more of a multimedia component 13, an input/output (I/O) interface 14, and a communication component 15.
The processor 11 is configured to control the overall operation of the electronic device 700, so as to complete all or part of the steps in the data migration method. The memory 12 is used to store various types of data to support operation at the electronic device 700, such as instructions for any application or method operating on the electronic device 700 and application-related data, such as contact data, transmitted and received messages, pictures, audio, video, and so forth. The Memory 12 may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk or optical disk. The multimedia component 13 may include a screen and an audio component. Wherein the screen may be, for example, a touch screen and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may further be stored in the memory 12 or transmitted via the communication component 15. The audio assembly also includes at least one speaker for outputting audio signals. The I/O interface 14 provides an interface between the processor 11 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication module 15 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless Communication, such as Wi-Fi, bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so that the corresponding Communication component 15 may include: Wi-Fi module, bluetooth module, NFC module.
In an exemplary embodiment, the electronic Device 700 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the data migration method described above.
In another exemplary embodiment, a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the above-described data migration method is also provided. For example, the computer readable storage medium may be the memory 12 described above including program instructions that are executable by the processor 11 of the electronic device 700 to perform the data migration method described above.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Claims (13)
1. A method of data migration, comprising:
determining a target bucket needing data migration based on the access heat of all buckets; wherein the target bucket comprises a nonvolatile multi-level storage medium and a hard disk drive;
determining data to be migrated based on the read-write speed of the hard disk drive and the data heat of the data in the hard disk drive;
when the system time enters the protection period time period of the target storage bucket, migrating the data to be migrated to the multi-level storage medium;
wherein, the determining a target bucket needing to be migrated based on the access heat of all buckets includes:
calculating the access heat of each bucket, and determining the bucket with the access heat larger than a heat threshold value as a target bucket;
wherein the determining data to be migrated based on the read-write speed of the hard disk drive and the data heat of the data in the hard disk drive comprises:
determining an access amount threshold corresponding to the hard disk drive according to the read-write speed of the hard disk drive, and determining the data heat of data in the hard disk drive;
judging whether first target data with the data heat degree larger than an access amount threshold corresponding to the hard disk drive exists in the hard disk drive;
if yes, determining the data to be migrated as the first target data;
if not, re-entering the step of determining the access amount threshold corresponding to the hard disk drive according to the read-write speed of the hard disk drive;
wherein the determining the data heat of the data in the hard disk drive comprises:
and determining the total download amount of the data in the hard disk drive in the first time window as the data heat of the data in the hard disk drive.
2. The data migration method of claim 1, wherein said determining the data heat of the data in the hard disk drive comprises:
determining the total download amount of data in the hard disk drive in a plurality of time windows, and determining a time attenuation coefficient corresponding to each time window;
and calculating the data heat of the data in the hard disk drive based on the total download amount in each time window and the time attenuation coefficient.
3. The data migration method according to claim 1, wherein before determining the data to be migrated based on the read/write speed of the hard disk drive and the access information of the data in the hard disk drive, the method further comprises:
determining whether the hard disk drive requires data migration based on a current occupied capacity of the hard disk drive;
if yes, executing the step of determining the data to be migrated based on the read-write speed of the hard disk drive and the access information of the data in the hard disk drive;
and if not, re-entering the step of determining the target bucket needing data migration based on the access heat of all the buckets.
4. The data migration method of claim 3, wherein said determining whether the hard disk drive requires data migration based on the current occupied capacity of the hard disk drive comprises:
determining a first capacity threshold based on a total capacity of the hard disk drive, determining whether the currently occupied capacity is greater than the first capacity threshold;
if yes, determining that the hard disk drive needs data migration.
5. The data migration method according to claim 1, wherein migrating the data to be migrated to the multi-level storage medium comprises:
determining a highest-level storage medium in the multi-level storage media as a target storage medium; wherein, in the multi-stage storage medium, the read-write speed of the storage medium is positively correlated with the level;
determining a target capacity threshold corresponding to the target storage medium based on the total capacity of the target storage medium;
judging whether the current occupied capacity of the target storage medium is larger than the target capacity threshold value;
if yes, re-determining a next-stage storage medium of the target storage medium as the target storage medium, and re-entering the step of determining a target capacity threshold corresponding to the target storage medium based on the total capacity of the target storage medium;
and if not, migrating the data to be migrated to the target storage medium.
6. The data migration method according to claim 1, further comprising:
acquiring the anti-tampering attributes of all the buckets; wherein the tamper-resistant attribute comprises at least a guard period time period for the bucket;
and after the anti-tampering attribute takes effect, detecting whether the system time enters the protection period time period.
7. The data migration method according to any one of claims 1 to 6, further comprising:
determining the return data based on the read-write speed and the data heat of each storage medium in the multi-stage storage media;
and migrating the backlogged data to the hard disk drive when the system time enters the protection period of the target bucket.
8. The data migration method according to claim 7, wherein the determining the migrated data based on the read-write speed and the data heat of each storage medium in the multi-level storage medium comprises:
determining an access amount threshold corresponding to each storage medium according to the read-write speed of each storage medium in the multi-level storage media, and determining the data heat of data in the multi-level storage media;
judging whether second target data with the data heat degree smaller than the access quantity threshold corresponding to the storage medium exists in each storage medium or not;
if yes, determining the backlogged data as the second target data;
and if not, re-entering the step of determining the access amount threshold corresponding to each storage medium according to the read-write speed of each storage medium in the multi-stage storage media.
9. The data migration method of claim 8, wherein said determining the data heat of the data in the multi-level storage medium comprises:
and determining the total download amount of the data in the multi-level storage medium in a second time window as the data heat of the data in the multi-level storage medium.
10. The data migration method of claim 8, wherein said determining the data heat of the data in the multi-level storage medium comprises:
determining the total download amount of data in the multi-level storage medium in a plurality of time windows, and determining a time attenuation coefficient corresponding to each time window;
and calculating the data heat of the data in the multi-stage storage medium based on the total download amount and the time attenuation coefficient in each time window.
11. A data migration apparatus, comprising:
the first determining module is used for determining a target bucket needing data migration based on the access heat of all buckets; wherein the target bucket comprises a nonvolatile multi-level storage medium and a hard disk drive;
the second determining module is used for determining data to be migrated based on the read-write speed of the hard disk drive and the data heat of the data in the hard disk drive;
the migration module is used for migrating the data to be migrated to the multi-level storage medium when the system time enters the protection period time period of the target storage bucket;
the first determining module is specifically a module for calculating the access heat of each bucket, and determining the bucket with the access heat greater than a heat threshold value as a target bucket;
wherein the second determining module comprises:
the first determining unit is used for determining an access amount threshold corresponding to the hard disk drive according to the read-write speed of the hard disk drive and determining the data heat of data in the hard disk drive;
the first judging unit is used for judging whether first target data with the data heat degree larger than an access amount threshold value corresponding to the hard disk drive exists in the hard disk drive or not; if yes, determining the data to be migrated as the first target data; if not, restarting the working process of the first determination unit;
the first determining unit is specifically a unit that determines an access amount threshold corresponding to the hard disk drive according to the read-write speed of the hard disk drive, and determines a total download amount of data in the hard disk drive in a first time window as a data heat degree of the data in the hard disk drive.
12. An electronic device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data migration method according to any one of claims 1 to 10 when executing the computer program.
13. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the data migration method according to any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911318556.4A CN111124304B (en) | 2019-12-19 | 2019-12-19 | Data migration method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911318556.4A CN111124304B (en) | 2019-12-19 | 2019-12-19 | Data migration method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111124304A CN111124304A (en) | 2020-05-08 |
| CN111124304B true CN111124304B (en) | 2022-04-22 |
Family
ID=70500193
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911318556.4A Active CN111124304B (en) | 2019-12-19 | 2019-12-19 | Data migration method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111124304B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112860188A (en) * | 2021-02-09 | 2021-05-28 | 山东英信计算机技术有限公司 | Data migration method, system, device and medium |
| CN113377291B (en) * | 2021-06-09 | 2023-07-04 | 北京天融信网络安全技术有限公司 | Data processing method, device, equipment and medium of cache equipment |
| CN114138183B (en) * | 2021-10-29 | 2023-09-12 | 郑州云海信息技术有限公司 | Data layering method, device and equipment of storage system and storage medium |
| CN115543965A (en) * | 2022-10-21 | 2022-12-30 | 北京火山引擎科技有限公司 | Cross-machine-room data processing method, device, storage medium, and program product |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102081559B (en) * | 2011-01-11 | 2013-06-05 | 华为数字技术（成都）有限公司 | Data recovery method and device for redundant array of independent disks |
| CN103389948A (en) * | 2012-05-10 | 2013-11-13 | 索尼公司 | Device and method for managing memorizer and electronic device |
| JP6381529B2 (en) * | 2013-07-19 | 2018-08-29 | 株式会社日立製作所 | Storage apparatus and storage control method |
| CN108089814B (en) * | 2016-11-23 | 2021-04-06 | 中移(苏州)软件技术有限公司 | Data storage method and device |
| US20180285383A1 (en) * | 2017-03-29 | 2018-10-04 | Commvault Systems, Inc. | Managing pod cells using a global repository cell |
| CN107193500A (en) * | 2017-05-26 | 2017-09-22 | 郑州云海信息技术有限公司 | A kind of distributed file system Bedding storage method and system |
| CN108205424B (en) * | 2017-12-29 | 2021-09-28 | 北京奇虎科技有限公司 | Data migration method and device based on disk and electronic equipment |
-
2019
- 2019-12-19 CN CN201911318556.4A patent/CN111124304B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111124304A (en) | 2020-05-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111124304B (en) | Data migration method and device, electronic equipment and storage medium | |
| US10380035B2 (en) | Using an access increment number to control a duration during which tracks remain in cache | |
| US9952970B2 (en) | Cache allocation for disk array | |
| CN110018788A (en) | It is classified storage method, device, electronic equipment and computer readable storage medium | |
| CN111159436B (en) | Method, device and computing equipment for recommending multimedia content | |
| CN108228649B (en) | Method and apparatus for data access | |
| US9658967B2 (en) | Evicting cached stores | |
| CN110226158B (en) | Data prefetching method and device and storage equipment | |
| EP4020153A1 (en) | Cache space management method and device | |
| KR100690804B1 (en) | Method for executing garbage collection of mobile terminal | |
| CN110413228B (en) | Mapping table management method and system, electronic equipment and storage medium | |
| CN101763433B (en) | Data storage system and method | |
| CN109471843A (en) | A kind of metadata cache method, system and relevant apparatus | |
| CN115686385B (en) | Data storage method and device, computer equipment and storage medium | |
| KR20170052441A (en) | Centralized distributed systems and methods for managing operations | |
| US9542330B2 (en) | Systems and methods for background destaging storage tracks | |
| CN112631504A (en) | Method and device for realizing local cache by using off-heap memory | |
| WO2017128641A1 (en) | Multi-tenant buffer management method and server | |
| CN109582649A (en) | A kind of metadata storing method, device, equipment and readable storage medium storing program for executing | |
| CN103336670A (en) | Method and device for automatic distribution of data blocks based on data temperature | |
| US9785562B2 (en) | Adjusting allocation of storage devices | |
| CN115102908A (en) | Method for generating network message based on bandwidth control and related device | |
| CN113742131B (en) | Method, electronic device and computer program product for storage management | |
| US10719243B2 (en) | Techniques for preserving an expected lifespan of a non-volatile memory | |
| WO2023165543A1 (en) | Shared cache management method and apparatus, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |