CN116578247A - Metadata-based data access method and system - Google Patents
Metadata-based data access method and system Download PDFInfo
- Publication number
- CN116578247A CN116578247A CN202310833094.XA CN202310833094A CN116578247A CN 116578247 A CN116578247 A CN 116578247A CN 202310833094 A CN202310833094 A CN 202310833094A CN 116578247 A CN116578247 A CN 116578247A
- Authority
- CN
- China
- Prior art keywords
- data
- area
- metadata
- description
- region
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000012986 modification Methods 0.000 claims description 20
- 230000004048 modification Effects 0.000 claims description 20
- 239000006185 dispersion Substances 0.000 claims description 16
- 238000012217 deletion Methods 0.000 claims description 11
- 230000037430 deletion Effects 0.000 claims description 11
- 238000012423 maintenance Methods 0.000 abstract description 6
- 230000007547 defect Effects 0.000 abstract description 5
- 230000006870 function Effects 0.000 description 5
- 238000013500 data storage Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 239000012634 fragment Substances 0.000 description 4
- 239000002699 waste material Substances 0.000 description 4
- 238000013459 approach Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000013439 planning Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
- G06F12/023—Free address space management
- G06F12/0238—Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/164—File meta data generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a metadata-based data access method and system, relates to the technical field of data access, and solves the defects that in the prior art, the maintenance cost is high in a mode of manually adjusting the capacity of a storage area, and capacity resources are wasted due to storage in a preset file format to a certain extent. The method comprises the following steps: acquiring the capacity of a memory, and dividing the space of the memory into a first storage area and a second storage area; wherein the first storage area is configured to store description metadata and description data, the description metadata including validity of the metadata and the number of areas in which the data is stored; the second storage area is configured to store actual data; and constructing description metadata and description data based on the actual data to be accessed, storing the description metadata and the description data into the first access area, and storing, reading, deleting and/or modifying the actual data based on the constructed description metadata and description data.
Description
Technical Field
The present application relates to the field of data access technologies, and in particular, to a metadata-based data access method and system.
Background
The unmanned aerial vehicle has the advantages of small volume, low cost and convenient use, and is separated from the category of the traditional aircraft for executing a single task, for example, the civil disaster relief unmanned aerial vehicle has the functions of disaster feedback and material delivery, and can be used as a network relay; the military reconnaissance and battle integrated unmanned aerial vehicle not only can execute tasks such as terrain reconnaissance, tactical decoy and the like, but also can realize remote battle. Along with the increase of the complexity of the tasks executed by the unmanned aerial vehicle, the corresponding data volume of the flight path, the task planning, the log and the like which need to be accessed is multiplied, and the complexity of data management is also increased.
The unmanned aerial vehicle data access method disclosed in the prior art mainly comprises two methods. One approach is to artificially divide the storage device into fixed-size areas, each with its own storage function, such as a first area for storing mission data, a second area for storing flight paths, and a third area for storing logs. Another method is to add a file system to the software, create files with different names from different types of data, and then access the files in a preset file format.
When unmanned aerial vehicle data access is performed by manually dividing a fixed-size area, when the data volume of a certain area is increased, the situation that the capacity of the current area is insufficient and the capacity of other areas remains large occurs, and the maintenance cost is high by manually adjusting the capacity of a storage area; and the data storage process has the characteristic of block storage in a preset file format, so that the capacity resource waste is caused.
Disclosure of Invention
The embodiment of the application solves the defects that the maintenance cost is high by manually adjusting the capacity of the storage area in the prior art, and the storage in the preset file format can cause the waste of capacity resources to a certain extent by providing the metadata-based data access method and system.
In a first aspect, an embodiment of the present application provides a metadata-based data access method, where the method includes:
acquiring the capacity of a memory, and dividing the space of the memory into a first storage area and a second storage area; wherein the first storage area is configured to store description metadata and description data, the description metadata including validity of the metadata and the number of areas in which the data is stored; the description data comprises a region code, a region start address, a region end address and a scattered region start address of the storage data; the second storage area is configured to store actual data;
and constructing description metadata and description data based on the actual data to be accessed, storing the description metadata and the description data into the first storage area, and storing, reading, deleting and/or modifying the actual data based on the constructed description metadata and description data.
With reference to the first aspect, in one possible implementation manner, the storing actual data based on the constructed description metadata and the description data includes:
viewing description metadata based on the storage command, and judging the validity of the metadata and the number of areas for storing the data;
when the metadata is effective and the number of the areas for storing the data is larger than zero, searching the area codes of the storage areas corresponding to the actual data to be stored in the first storage area;
when the first storage area does not have the corresponding area code, constructing description data of the corresponding area, and writing actual data to be stored into a second storage area corresponding to the constructed description data; and when the corresponding area codes exist in the first storage area, searching the scattered area of the corresponding area and executing the corresponding steps on the scattered area.
With reference to the first aspect, in one possible implementation manner, the searching for the dispersed area of the corresponding area and performing the corresponding step on the dispersed area includes:
when the actual data to be stored belongs to the current scattered area, writing the actual data to be stored into the scattered area; when the actual data to be stored does not belong to the current dispersion area, expanding the current dispersion area, and writing the actual data to be stored into the expanded dispersion area.
With reference to the first aspect, in one possible implementation manner, the expanding the current dispersion area includes:
searching an expandable region with the largest space, wherein the expandable region comprises an idle storage region which can be partitioned or does not have a scattered region;
dividing the searched expandable region, wherein a part of the divided expandable region is used as a dispersion region;
and acquiring the area information of the expandable area serving as the scattered area, and storing the corresponding area information into the first storage area, wherein the area information comprises an area code, an area starting address and an area ending address.
With reference to the first aspect, in one possible implementation manner, the reading actual data based on the constructed description metadata and the description data includes:
constructing description metadata based on the read command, and judging the validity of the metadata and the number of areas for storing the data;
when the metadata is effective and the number of the areas for storing the data is larger than zero, searching an area code of a storage area corresponding to the data to be read in the read command in the first storage area;
according to the region code, obtaining a region start address and a region end address, and judging whether the sum of the length of the data to be read in the read command and the length of the data to be read is smaller than the region end address;
and when the sum of the start address of the data to be read and the length of the data to be read in the read command is smaller than the end address of the area, reading the actual data with the corresponding length in the corresponding storage area.
With reference to the first aspect, in one possible implementation manner, the determining whether the sum of the start address of the data to be read and the length of the data to be read in the read command is smaller than the end address of the area includes:
when the sum of the start address of the data to be read and the length of the data to be read in the read command is greater than or equal to the end address of the area, acquiring the value of the scattered area;
when the value of the scattered area is not zero, an area end address is acquired in the scattered area, and actual data of corresponding length in the corresponding storage area is read based on the area start address and the area end address.
With reference to the first aspect, in one possible implementation manner, the deleting actual data based on the constructed description metadata and the description data includes:
constructing description metadata based on the deleting command, and judging the validity of the metadata and the number of areas for storing the data;
when the metadata is effective and the number of the areas for storing the data is larger than zero, searching the area codes of the storage areas corresponding to the deleting command in the first storage area;
judging the validity of the item of the deleting command, and deleting the actual data of the corresponding item according to the deleting command when the item of the deleting command is legal; when the item of the deletion command is illegal, the range of the scattered area is acquired again and the item of the deletion command is judged again.
With reference to the first aspect, in one possible implementation manner, the re-acquiring the range of the scattered area and re-judging the entry of the deletion command includes:
and acquiring a dispersed area starting address, a dispersed area ending address and a next dispersed area address, repeating the judging of the item of the deleting command, and deleting the actual data of the corresponding item according to the deleting command.
With reference to the first aspect, in one possible implementation manner, the modifying the actual data based on the constructed description metadata and the description data includes:
constructing description metadata based on the modification command, and judging the validity of the metadata;
when the metadata is valid, searching a region code of a storage region corresponding to the modification command in the first storage region;
acquiring an area start address and an area end address according to the area code, and judging the validity of the entry of the modification command;
when the entry of the modification command is legal, the actual data of the corresponding storage area is modified according to the modification command.
In a second aspect, an embodiment of the present application provides a metadata-based data access system, the system including:
a storage module configured to: acquiring the capacity of a memory, and dividing the space of the memory into a first storage area and a second storage area; wherein the first storage area is configured to store description metadata and description data, the description metadata including validity of the metadata and the number of areas in which the data is stored; the description data comprises a region code, a region start address, a region end address and a scattered region start address of the storage data; the second storage area is configured to store actual data;
an execution module configured to: and constructing description metadata and description data based on the actual data to be accessed, storing the description metadata and the description data into the first storage area, and storing, reading, deleting and/or modifying the actual data based on the constructed description metadata and description data.
One or more technical solutions provided in the embodiments of the present application at least have the following technical effects or advantages:
the capacity space of the memory is divided into a first memory area and a second memory area, the actual data stored in the second memory area is managed through the description metadata and the description data in the first memory area, the second memory area is expanded in real time according to the memory space required by the actual data, and the probability of generating unoccupied fragments which are difficult to use in the memory space is reduced; meanwhile, the first storage area and the second storage area are managed through metadata, and the actually occupied capacity space is managed through a virtual mode of expanding space, so that the defect of high maintenance difficulty caused by manually dividing the capacity space can be overcome, the probability of large occupied capacity caused by storing data by pre-stored file names can be reduced, and the utilization rate of the capacity space in the memory is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a metadata-based data access method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a first storage area and a second storage area divided by a metadata-based data access method according to an embodiment of the present application;
FIG. 3 is a flowchart of storing actual data according to the metadata-based data access method according to the embodiment of the present application;
FIG. 4 is a flowchart of a metadata-based data access method according to an embodiment of the present application for expanding a current decentralized region;
FIG. 5 is a flowchart of a metadata-based data access method according to an embodiment of the present application for reading actual data;
FIG. 6 is a flowchart of a metadata-based data access method according to an embodiment of the present application for deleting actual data;
FIG. 7 is a flowchart of a metadata-based data access method according to an embodiment of the present application for modifying actual data;
fig. 8 is a schematic structural diagram of an execution module in the metadata-based data access system according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. It will be apparent that the described embodiments are some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Referring to fig. 1 and fig. 2, fig. 1 is a flowchart of a metadata-based data access method according to an embodiment of the present application; fig. 2 is a schematic structural diagram of a first storage area and a second storage area divided by a metadata-based data access method according to an embodiment of the present application.
In some embodiments, the present application provides a metadata-based data access method, where the data access method includes step 101 and step 102.
Step 101, obtaining the capacity of a memory, and dividing the space of the memory into a first memory area and a second memory area; wherein the first storage area is configured to store description metadata and description data, the description metadata including validity of the metadata and the number of areas in which the data is stored; the description data comprises a region code, a region start address, a region end address and a scattered region start address of the storage data; the second storage area is configured to store actual data.
And 102, constructing description metadata and description data based on the actual data to be accessed, storing the description metadata and the description data into a first storage area, and storing, reading, deleting and/or modifying the actual data based on the constructed description metadata and description data.
The capacity space of the memory is divided into a first memory area and a second memory area, the actual data stored in the second memory area is managed through the description metadata and the description data in the first memory area, the second memory area is expanded in real time according to the memory space required by the actual data, and the probability of generating unoccupied fragments which are difficult to use in the memory space is reduced; meanwhile, the first storage area and the second storage area are managed through metadata, and the actually occupied capacity space is managed through a virtual mode of expanding space, so that the defect of high maintenance difficulty caused by manually dividing the capacity space can be overcome, the probability of large occupied capacity caused by storing data by pre-stored file names can be reduced, and the utilization rate of the capacity space in the memory is improved.
Referring to fig. 3, fig. 3 is a flowchart illustrating a metadata-based data access method according to an embodiment of the present application for storing actual data.
In some embodiments, the data access method provided by the present application includes, in a process of executing storing actual data based on constructed description metadata and description data: viewing description metadata based on the storage command, and judging the validity of the metadata and the number of areas for storing the data; when the metadata is effective and the number of the areas for storing the data is larger than zero, searching an area code of a storage area corresponding to the actual data to be stored in the first storage area; when the first storage area does not have the corresponding area code, constructing description data of the corresponding area, and writing actual data to be stored into a second storage area corresponding to the constructed description data; and when the corresponding area codes exist in the first storage area, searching the scattered area of the corresponding area and executing the corresponding steps on the scattered area.
And when the fact that the corresponding area is unoccupied is detected, the actual data is stored into the corresponding area, and the corresponding description data is formed and stored in the first storage area.
In some embodiments, the data access method provided by the present application includes, in a process of performing searching for a scattered area of a corresponding area and performing a corresponding step on the scattered area: when the actual data to be stored belongs to the current scattered area, writing the actual data to be stored into the scattered area; when the actual data to be stored does not belong to the current dispersion area, expanding the current dispersion area, and writing the actual data to be stored into the expanded dispersion area.
By checking the corresponding relation between the capacity of the current scattered area and the storage capacity required by the actual data, the actual data is conveniently stored in the corresponding scattered area, and when the actual data does not belong to the current scattered area, the actual data is stored in a manner of expanding the current scattered area, so that the purpose of storing the actual data to be stored currently in the second storage area can be realized.
Referring to fig. 4, fig. 4 is a flowchart illustrating a metadata-based data access method according to an embodiment of the present application to expand a current scattered area.
In some embodiments, when actual data to be stored does not belong to a current dispersion area, the data access method provided by the present application includes: searching the expandable area with the largest space, wherein the expandable area comprises an idle storage area which can be divided or does not have a scattered area; dividing the searched expandable region, wherein a part of the divided expandable region is used as a dispersion region; the method comprises the steps of obtaining area information of an expandable area serving as a scattered area, and storing the corresponding area information into a first storage area, wherein the area information comprises an area code, an area starting address and an area ending address.
Illustratively, when the searched expandable region is segmented, the searched expandable region is segmented in half, the first half of the storage region is unchanged, the second half of the storage region is used as a scattered region, and expandable region information serving as the scattered region is stored in the first storage region.
The expandable area with larger space is divided into two parts, the size of the needed storage area is increased by reducing the large expandable space area, the problem that the capacity of the current area is insufficient and the capacity of other areas is large due to the increase of the data volume of a certain area in the traditional data storage process can be solved to a certain extent, and compared with the traditional data storage mode, the data storage method provided by the application has high flexibility and expandability. Meanwhile, the whole space of the memory is managed and allocated through metadata, when the storage space for storing actual data is insufficient, the region expansion is carried out, each storage region in the memory is fully utilized, the probability of occurrence of storage idle fragments is reduced, and the purpose of improving the effective utilization rate of the memory can be achieved.
Referring to fig. 5, fig. 5 is a flowchart illustrating a metadata-based data access method according to an embodiment of the present application for reading actual data.
In some embodiments, the data access method provided by the present application includes, in executing a process of reading actual data based on constructed description metadata and description data: constructing description metadata based on the read command, and judging the validity of the metadata and the number of areas for storing the data; when the metadata is effective and the number of the areas for storing the data is larger than zero, searching an area code of a storage area corresponding to the data to be read in the read command in the first storage area; according to the region code, obtaining a region start address and a region end address, and judging whether the sum of the length of the data to be read in the read command and the length of the data to be read is smaller than the region end address; and when the sum of the start address of the data to be read and the length of the data to be read in the read command is smaller than the end address of the area, reading the actual data with the corresponding length in the corresponding storage area.
In the process of judging whether the sum of the length of the data to be read in the data to be read starting address and the length of the data to be read in the read command is smaller than the area end address, when the sum of the length of the data to be read in the data to be read starting address and the length of the data to be read in the read command is larger than or equal to the area end address, acquiring the value of the scattered area; ending the current step when the value of the dispersed area is zero; and when the value of the scattered area is not zero, acquiring an area end address in the scattered area, and reading actual data with corresponding length in the corresponding storage area based on the area start address and the area end address.
The validity of the metadata is judged by acquiring the description metadata based on the reading command, so that the probability of time waste caused by executing subsequent steps when the metadata is invalid is reduced, and the convenience of reading the actual data is improved by acquiring the region starting address and the region ending address based on the region code and judging the integrity of the actual data stored in the current region access region according to the data starting address to be read and the data length to be read.
Referring to fig. 6, fig. 6 is a flowchart illustrating a metadata-based data access method according to an embodiment of the present application to perform deletion of actual data.
In some embodiments, the data access method provided by the present application includes, when performing deletion of actual data based on constructed description metadata and description data: constructing description metadata based on the deleting command, and judging the validity of the metadata and the number of areas for storing the data; when the metadata is effective and the number of the areas for storing the data is larger than zero, searching the area codes of the storage areas corresponding to the deleting command in the first storage area, and when the metadata is invalid or the number of the areas for storing the data is equal to zero, stopping executing the current command; judging the validity of the item of the deleting command, and deleting the actual data of the corresponding item according to the deleting command when the item of the deleting command is legal; when the item of the deletion command is illegal, the range of the scattered area is acquired again and the item of the deletion command is judged again.
Wherein the step of re-acquiring the range of the dispersed area and re-judging the item of the deletion command includes: and acquiring a dispersed area starting address, a dispersed area ending address and a next dispersed area address, repeating the judging of the item of the deleting command, and deleting the actual data of the corresponding item according to the deleting command.
The actual data in the second storage area is judged by sequentially judging the validity of the metadata and the number of the areas for storing the data, so that the probability of time waste caused by repeated operation when the actual data does not exist is reduced.
Referring to fig. 7, fig. 7 is a flowchart illustrating a modification of actual data according to the metadata-based data access method according to an embodiment of the present application.
In some embodiments, the data access method provided by the present application includes, when performing modification of actual data based on constructed description metadata and description data: constructing description metadata based on the modification command, and judging the validity of the metadata; when the metadata is valid, searching a region code of a storage region corresponding to the modification command in the first storage region; acquiring an area start address and an area end address according to the area code, and judging the validity of the entry of the modification command; when the entry of the modification command is legal, the actual data of the corresponding storage area is modified according to the modification command. When the value of the current dispersion area is found to be zero in the execution process, the current command is directly ended.
And if the sum of the area starting address and the length of the data to be modified in the modification command is smaller than the area ending address, indicating that the actual data to be modified is in the current storage area, and modifying the corresponding actual data according to the modification command.
Referring to fig. 8, fig. 8 is a schematic structural diagram of an execution module in the metadata-based data access system according to an embodiment of the present application.
Based on the data access method disclosed by the technical scheme, the application also provides a data access system, which comprises a storage module and an execution module, wherein the storage module is in communication connection with the execution module.
The storage module is configured to: acquiring the capacity of a memory, and dividing the space of the memory into a first storage area and a second storage area; wherein the first storage area is configured to store description metadata and description data, the description metadata including validity of the metadata and the number of areas in which the data is stored; the description data comprises a region code, a region start address, a region end address and a scattered region start address of the storage data; the second storage area is configured to store actual data;
the execution module is configured to: and constructing description metadata and description data based on the actual data to be accessed, storing the description metadata and the description data into the first storage area, and storing, reading, deleting and/or modifying the actual data based on the constructed description metadata and description data.
Illustratively, the execution module includes a partition into a DAO layer, a logic layer, and a service layer. The DAO layer is generally called Data Access Object, and is configured to engage a read-write interface of the storage module to perform data reading, writing and attribute acquisition, and provide data services for the logic layer. The logic layer is connected between the DAO layer and the service layer and is configured to assemble and operate data according to the actual requirements of external commands and realize specific execution commands. The service layer is configured to link the logic layer with the application logic, and unlock, mutex and/or synchronize the operation logic.
According to the embodiment of the application, the capacity space of the memory is divided into the first memory area and the second memory area, the actual data stored in the second memory area is managed through the description metadata and the description data in the first memory area, the second memory area is expanded in real time according to the memory space required by the actual data, and the probability of generating unoccupied fragments which are difficult to utilize in the memory space is reduced; meanwhile, the first storage area and the second storage area are managed through metadata, and the actually occupied capacity space is managed through a virtual mode of expanding space, so that the defect of high maintenance difficulty caused by manually dividing the capacity space can be overcome, the probability of large occupied capacity caused by storing data by pre-stored file names can be reduced, and the utilization rate of the capacity space in the memory is improved.
Although the application provides method operational steps as described in the examples or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive labor. The order of steps recited in the present embodiment is only one way of performing the steps in a plurality of steps, and does not represent a unique order of execution. When implemented by an actual device or client product, the method of the present embodiment or the accompanying drawings may be performed sequentially or in parallel (e.g., in a parallel processor or a multithreaded environment).
The apparatus or module set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. For convenience of description, the above devices are described as being functionally divided into various modules, respectively. The functions of the various modules may be implemented in the same piece or pieces of software and/or hardware when implementing the present application. Of course, a module that implements a certain function may be implemented by a plurality of sub-modules or a combination of sub-units.
The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the present application; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced with equivalents; such modifications and substitutions do not depart from the spirit of the application.
Claims (10)
1. A metadata-based data access method, the method comprising:
acquiring the capacity of a memory, and dividing the space of the memory into a first storage area and a second storage area; wherein the first storage area is configured to store description metadata and description data, the description metadata including validity of the metadata and the number of areas in which the data is stored; the description data comprises a region code, a region start address, a region end address and a scattered region start address of the storage data; the second storage area is configured to store actual data;
and constructing description metadata and description data based on the actual data to be accessed, storing the description metadata and the description data into the first storage area, and storing, reading, deleting and/or modifying the actual data based on the constructed description metadata and description data.
2. The metadata-based data access method according to claim 1, wherein the storing of actual data based on the constructed description metadata and the description data comprises:
viewing description metadata based on the storage command, and judging the validity of the metadata and the number of areas for storing the data;
when the metadata is effective and the number of the areas for storing the data is larger than zero, searching the area codes of the storage areas corresponding to the actual data to be stored in the first storage area;
when the first storage area does not have the corresponding area code, constructing description data of the corresponding area, and writing actual data to be stored into a second storage area corresponding to the constructed description data; and when the corresponding area codes exist in the first storage area, searching the scattered area of the corresponding area and executing the corresponding steps on the scattered area.
3. The metadata-based data access method of claim 2, wherein the searching for the scattered areas of the corresponding areas and performing the corresponding steps on the scattered areas comprises:
when the actual data to be stored belongs to the current scattered area, writing the actual data to be stored into the scattered area; when the actual data to be stored does not belong to the current dispersion area, expanding the current dispersion area, and writing the actual data to be stored into the expanded dispersion area.
4. The metadata-based data access method of claim 3, wherein expanding the current dispersion area comprises:
searching an expandable region with the largest space, wherein the expandable region comprises an idle storage region which can be partitioned or does not have a scattered region;
dividing the searched expandable region, wherein a part of the divided expandable region is used as a dispersion region;
and acquiring the area information of the expandable area serving as the scattered area, and storing the corresponding area information into the first storage area, wherein the area information comprises an area code, an area starting address and an area ending address.
5. The metadata-based data access method according to claim 1, wherein the reading of actual data based on the constructed description metadata and the description data comprises:
constructing description metadata based on the read command, and judging the validity of the metadata and the number of areas for storing the data;
when the metadata is effective and the number of the areas for storing the data is larger than zero, searching an area code of a storage area corresponding to the data to be read in the read command in the first storage area;
according to the region code, obtaining a region start address and a region end address, and judging whether the sum of the length of the data to be read in the read command and the length of the data to be read is smaller than the region end address;
and when the sum of the start address of the data to be read and the length of the data to be read in the read command is smaller than the end address of the area, reading the actual data with the corresponding length in the corresponding storage area.
6. The metadata-based data access method according to claim 5, wherein the determining whether the sum of the start address of the data to be read and the length of the data to be read in the read command is smaller than the end address of the area comprises:
when the sum of the start address of the data to be read and the length of the data to be read in the read command is greater than or equal to the end address of the area, acquiring the value of the scattered area;
when the value of the scattered area is not zero, an area end address is acquired in the scattered area, and actual data of corresponding length in the corresponding storage area is read based on the area start address and the area end address.
7. The metadata-based data access method according to claim 1, wherein the deleting of the actual data based on the constructed description metadata and the description data comprises:
constructing description metadata based on the deleting command, and judging the validity of the metadata and the number of areas for storing the data;
when the metadata is effective and the number of the areas for storing the data is larger than zero, searching the area codes of the storage areas corresponding to the deleting command in the first storage area;
judging the validity of the item of the deleting command, and deleting the actual data of the corresponding item according to the deleting command when the item of the deleting command is legal; when the item of the deletion command is illegal, the range of the scattered area is acquired again and the item of the deletion command is judged again.
8. The metadata-based data access method of claim 7, wherein the re-acquiring the range of the scattered area and re-judging the entry of the deletion command comprises:
and acquiring a dispersed area starting address, a dispersed area ending address and a next dispersed area address, repeating the judging of the item of the deleting command, and deleting the actual data of the corresponding item according to the deleting command.
9. The metadata-based data access method according to claim 1, wherein the modifying actual data based on the constructed description metadata and the description data comprises:
constructing description metadata based on the modification command, and judging the validity of the metadata;
when the metadata is valid, searching a region code of a storage region corresponding to the modification command in the first storage region;
acquiring an area start address and an area end address according to the area code, and judging the validity of the entry of the modification command;
when the entry of the modification command is legal, the actual data of the corresponding storage area is modified according to the modification command.
10. A metadata-based data access system, comprising:
a storage module configured to: acquiring the capacity of a memory, and dividing the space of the memory into a first storage area and a second storage area; wherein the first storage area is configured to store description metadata and description data, the description metadata including validity of the metadata and the number of areas in which the data is stored; the description data comprises a region code, a region start address, a region end address and a scattered region start address of the storage data; the second storage area is configured to store actual data;
an execution module configured to: and constructing description metadata and description data based on the actual data to be accessed, storing the description metadata and the description data into the first storage area, and storing, reading, deleting and/or modifying the actual data based on the constructed description metadata and description data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310833094.XA CN116578247B (en) | 2023-07-10 | 2023-07-10 | Metadata-based data access method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310833094.XA CN116578247B (en) | 2023-07-10 | 2023-07-10 | Metadata-based data access method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116578247A true CN116578247A (en) | 2023-08-11 |
| CN116578247B CN116578247B (en) | 2023-10-13 |
Family
ID=87543467
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310833094.XA Active CN116578247B (en) | 2023-07-10 | 2023-07-10 | Metadata-based data access method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116578247B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108089821A (en) * | 2017-12-20 | 2018-05-29 | 福建星海通信科技有限公司 | A kind of method of micro controller data storage management |
| US20190324669A1 (en) * | 2018-04-23 | 2019-10-24 | Western Digital Technologies, Inc. | Block reuse for memory operations |
| WO2020102064A1 (en) * | 2018-11-12 | 2020-05-22 | Dover Microsystems, Inc. | Systems and methods for metadata encoding |
| CN112269540A (en) * | 2020-10-16 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Data storage method, system, equipment and computer readable storage medium |
| CN113311993A (en) * | 2021-03-26 | 2021-08-27 | 阿里巴巴新加坡控股有限公司 | Data storage method and data reading method |
| CN114327246A (en) * | 2020-10-12 | 2022-04-12 | 深圳市爱图仕影像器材有限公司 | Data storage method in storage medium, storage medium and computer equipment |
| CN115576484A (en) * | 2022-09-08 | 2023-01-06 | 深圳市正浩创新科技股份有限公司 | Data reading and writing method and device, electronic equipment and storage medium |
| CN115756317A (en) * | 2022-11-11 | 2023-03-07 | 浙江大华技术股份有限公司 | Storage method, storage device, electronic equipment and storage medium of shingled magnetic disk |
| CN115827511A (en) * | 2022-10-21 | 2023-03-21 | 阿里巴巴（中国）有限公司 | Method and device for setting cache |
| CN116149552A (en) * | 2023-01-30 | 2023-05-23 | 苏州浪潮智能科技有限公司 | Storage system optimization method, system, equipment and storage medium |
| CN116301614A (en) * | 2023-02-22 | 2023-06-23 | 深圳市立创普电源技术有限公司 | Memory data access method, system, device and storage medium |
-
2023
- 2023-07-10 CN CN202310833094.XA patent/CN116578247B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108089821A (en) * | 2017-12-20 | 2018-05-29 | 福建星海通信科技有限公司 | A kind of method of micro controller data storage management |
| US20190324669A1 (en) * | 2018-04-23 | 2019-10-24 | Western Digital Technologies, Inc. | Block reuse for memory operations |
| WO2020102064A1 (en) * | 2018-11-12 | 2020-05-22 | Dover Microsystems, Inc. | Systems and methods for metadata encoding |
| CN114327246A (en) * | 2020-10-12 | 2022-04-12 | 深圳市爱图仕影像器材有限公司 | Data storage method in storage medium, storage medium and computer equipment |
| CN112269540A (en) * | 2020-10-16 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Data storage method, system, equipment and computer readable storage medium |
| CN113311993A (en) * | 2021-03-26 | 2021-08-27 | 阿里巴巴新加坡控股有限公司 | Data storage method and data reading method |
| CN115576484A (en) * | 2022-09-08 | 2023-01-06 | 深圳市正浩创新科技股份有限公司 | Data reading and writing method and device, electronic equipment and storage medium |
| CN115827511A (en) * | 2022-10-21 | 2023-03-21 | 阿里巴巴（中国）有限公司 | Method and device for setting cache |
| CN115756317A (en) * | 2022-11-11 | 2023-03-07 | 浙江大华技术股份有限公司 | Storage method, storage device, electronic equipment and storage medium of shingled magnetic disk |
| CN116149552A (en) * | 2023-01-30 | 2023-05-23 | 苏州浪潮智能科技有限公司 | Storage system optimization method, system, equipment and storage medium |
| CN116301614A (en) * | 2023-02-22 | 2023-06-23 | 深圳市立创普电源技术有限公司 | Memory data access method, system, device and storage medium |
Non-Patent Citations (3)
| Title |
|---|
| YINGLONG XIONG等: "ECCH: Erasure Coded Consistent Hashing for Distributed Storage Systems", 《2021 IEEE INTL CONF ON PARALLEL & DISTRIBUTED PROCESSING WITH APPLICATIONS, BIG DATA & CLOUD COMPUTING, SUSTAINABLE COMPUTING & COMMUNICATIONS, SOCIAL COMPUTING & NETWORKING (ISPA/BDCLOUD/SOCIALCOM/SUSTAINCOM)》, pages 177 - 184 * |
| 刘昕林;邓巍;黄萍;刘睿臻;: "基于Hadoop和Spark的可扩展性大数据分析系统设计", 自动化与仪器仪表, no. 03 * |
| 王雪丽;罗文广;覃永新;: "基于有限状态机的SRAM分区地址发生器的设计与实现", 广西科技大学学报, no. 04 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116578247B (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107273455B (en) | Block chain data access method and device | |
| CN107391628B (en) | Data synchronization method and device | |
| CN106648994B (en) | Method, equipment and system for backing up operation log | |
| WO2017050064A1 (en) | Memory management method and device for shared memory database | |
| CN106796588B (en) | The update method and equipment of concordance list | |
| CN110532123A (en) | The failover method and device of HBase system | |
| CN112579595A (en) | Data processing method and device, electronic equipment and readable storage medium | |
| CN113220573A (en) | Test method and device for micro-service architecture and electronic equipment | |
| CN106776795B (en) | Data writing method and device based on Hbase database | |
| CN113961510A (en) | File processing method, device, equipment and storage medium | |
| CN108140022B (en) | Data query method and database system | |
| CN111209061B (en) | User information filling method, device, computer equipment and storage medium | |
| CN110806942B (en) | Data processing method and device | |
| JP2023546818A (en) | Transaction processing method, device, electronic device, and computer program for database system | |
| CN116578247B (en) | Metadata-based data access method and system | |
| CN113342647A (en) | Test data generation method and device | |
| CN112711606A (en) | Database access method and device, computer equipment and storage medium | |
| CN113806309B (en) | Metadata deleting method, system, terminal and storage medium based on distributed lock | |
| CN111786808A (en) | Cloud system migration method and device and mixed cloud system | |
| CN110716814B (en) | Performance optimization method and device for inter-process large-data-volume communication | |
| CN115756760A (en) | File synchronization method and device | |
| CN111209304B (en) | Data processing method, device and system | |
| CN115599295A (en) | Node capacity expansion method and device of storage system | |
| CN108376104B (en) | Node scheduling method and device and computer readable storage medium | |
| CN114722261A (en) | Resource processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |