CN111143293B - Metadata acquisition method, device, equipment and computer readable storage medium - Google Patents

Abstract

The invention discloses a metadata acquisition method, device, equipment and computer-readable storage medium; in this solution, since omap only stores metadata of files in a directory, or metadata of sub-directories in a directory, this The metadata storage method allows the client to only obtain the metadata of the file in the directory or the metadata of the subdirectory when obtaining the metadata, thereby reducing the processing time of the metadata obtaining request and reducing the disk load. Improve metadata acquisition efficiency.

Description

Metadata acquisition method, apparatus, device, and computer-readable storage medium

technical field

The present invention relates to the technical field of information storage, and more particularly, to a method, apparatus, device and computer-readable storage medium for obtaining metadata.

Background technique

At present, when the client needs to operate a certain directory, if the metadata service checks that the metadata information of the Cdir (directory shard) corresponding to the directory is not in the cache, it needs to retrieve the data from rados (Reliable, Autonomic Distributed ObjectStore, distributed data storage system) Crawling CDir information and directory entries. However, since the metadata information of all subdirectories and files in the directory are stored in the same omap (object map, the data structure for storing metadata information), when the directory information is not in the cache, no matter whether the directory information is obtained To obtain the metadata information of the subdirectory, or to obtain the metadata information of the files in the directory, it is necessary to capture the entire omap data structure of the directory from rados, that is, the omap data structure to be captured contains all the subdirectories in the directory. Metadata information of directories and files, it can be seen that this acquisition method captures some unnecessary metadata information invisibly. When there are a large number of grabbing operations, it will give osd (ObjectStorage Device) It brings a large read and write load, resulting in low crawling efficiency and timeout of client request processing time.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method, apparatus, device and computer-readable storage medium for obtaining metadata, so as to improve the efficiency of obtaining metadata.

In order to achieve the above object, the present invention provides a method for obtaining metadata, including:

Receive a metadata acquisition request sent by the client; the metadata acquisition request is an acquisition request for acquiring a file under a directory, or an acquisition request for acquiring a subdirectory under the directory;

Determine the target omap corresponding to the metadata acquisition request in the disk; wherein, the omap only stores the metadata of the file under the directory, or the metadata of the subdirectory under the directory;

Obtain metadata information from the target omap and send it to the client.

Wherein, before receiving the metadata acquisition request sent by the client, the method further includes:

Receive a request to store the target directory;

The metadata of the files under the target directory and the metadata of the subdirectories under the target directory are stored in different omaps of the disk.

Wherein, the metadata of the file under the target directory and the metadata of the subdirectory under the target directory are stored in different omaps of the disk, including:

Traverse each target content under the target directory in turn, and determine the index node of each target content according to the directory entry of each target content;

Using the inode of each target content to determine the type of each target content, the type including files and subdirectories;

Store the metadata of the target content of type file and the metadata of the target content of type subdirectory to different omaps on disk.

Wherein, the metadata of the target content whose type is a file and the metadata of the target content whose type is a subdirectory are stored in different omaps of the disk, including:

Cache the metadata of the target content of type file and the metadata of target content of type subdirectory into different data structures;

Store metadata in different data structures in different omaps on disk.

In order to achieve the above object, the present invention further provides a device for obtaining metadata, comprising:

a first receiving module, configured to receive a metadata acquisition request sent by a client; the metadata acquisition request is an acquisition request for acquiring a file in a directory, or an acquisition request for acquiring a subdirectory under the directory;

A determination module for determining the target omap corresponding to the metadata acquisition request in the disk; wherein, the omap only stores the metadata of the file under the directory, or the metadata of the subdirectory under the directory;

A sending module, configured to obtain metadata information from the target omap and send it to the client.

Among them, this program also includes:

a second receiving module, configured to receive a request for storing the target directory;

The storage module is configured to store the metadata of the files under the target directory and the metadata of the subdirectories under the target directory in different omaps of the disk.

Wherein, the storage module includes:

The first determining unit is used to traverse each target content under the target directory in turn, and determine the index node of each target content according to the directory entry of each target content;

a second determining unit, configured to determine the type of each target content by using the index node of each target content, and the type includes files and subdirectories;

The storage unit is used to store the metadata of the target content whose type is a file and the metadata of the target content whose type is a subdirectory into different omaps on the disk.

The storage unit is specifically configured to: cache the metadata of the target content whose type is a file and the metadata of the target content whose type is a subdirectory into different data structures; store the metadata in the different data structures respectively to different omaps on disk.

To achieve the above purpose, the present invention further provides a device for obtaining metadata, including:

memory for storing computer programs;

The processor is configured to implement the steps of the above metadata acquisition method when executing the computer program.

To achieve the above object, the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above metadata acquisition method are implemented.

As can be seen from the above solutions, a method for obtaining metadata provided by an embodiment of the present invention includes: receiving a request for obtaining metadata sent by a client; the request for obtaining metadata is a request for obtaining a file in a directory, or a request for obtaining a file in a directory. The acquisition request of the subdirectory; determine the target omap corresponding to the metadata acquisition request in the disk; wherein, the omap only stores the metadata of the file under the directory, or the metadata of the subdirectory under the directory; from the target omap obtains metadata information and sends it to the client.

It can be seen that, because the omap in this application only stores the metadata of the files under the directory, or the metadata of the subdirectories under the directory, this way of classifying and storing the metadata enables the client to only obtain the file when obtaining the metadata. metadata or sub-directory metadata, thereby reducing the processing time of the metadata acquisition request, reducing the disk load, and improving the metadata acquisition efficiency; the invention also discloses a metadata acquisition device, equipment and a computer-readable The storage medium can also achieve the above technical effects.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a schematic flowchart of a method for obtaining metadata disclosed in an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of a device for obtaining metadata disclosed in an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a device for obtaining metadata according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Embodiments of the present invention disclose a method, apparatus, device, and computer-readable storage medium for obtaining metadata, so as to improve the efficiency of obtaining metadata.

Referring to FIG. 1, a method for obtaining metadata provided by an embodiment of the present invention includes:

S101. Receive a metadata acquisition request sent by a client; the metadata acquisition request is an acquisition request for acquiring a file under a directory, or an acquisition request for acquiring a subdirectory under the directory;

It should be noted that the metadata acquisition method described in this embodiment is applied when the metadata is not in the cache. If the metadata has been stored in the cache, the metadata acquisition request sent by the client is received at this time. You can directly look up the corresponding metadata from the cache and return it, and you don't need to look up the corresponding metadata from the disk and return it. In the case where the metadata is not in the cache, because whether to obtain the metadata information of the subdirectories under the directory or the metadata information of the files under the directory, it is necessary to obtain all the subdirectories and files under the directory from the disk. In this way, some unnecessary metadata information is captured invisibly. When there are a large number of fetching operations, it will bring a large read and write load to the disk, resulting in low fetching efficiency. The client request processing time timed out.

In the present application, before receiving the metadata acquisition request sent by the client, the method further includes: receiving a request for storing a target directory; Data, stored in different omaps on disk.

That is to say, in the present application, when a client stores a directory, its metadata needs to be stored in different omaps on the disk according to the types of files and subdirectories, thereby realizing classified storage of metadata. Therefore, when the client in this application obtains metadata, it can only send a request for obtaining files in a directory, or only a request for obtaining sub-directories in a directory, and of course, it can also send a request for obtaining files and sub-directories in a directory. The request for obtaining the directory is not specifically limited here.

S102, determine the target omap corresponding to the metadata acquisition request in the disk; wherein, the omap only stores the metadata of the file under the directory, or the metadata of the subdirectory under the directory;

S103. Obtain metadata information from the target omap, and send it to the client.

It should be noted that after receiving the metadata acquisition request sent by the client, the target omap for storing metadata can be determined according to the metadata type obtained by the request; that is, if the metadata acquisition request is to acquire files in the directory , you need to use the omap that stores the metadata of the files in the directory as the target omap; if the metadata acquisition request is an acquisition request to acquire a subdirectory under the directory, you need to store the subdirectory in the directory. The omap of the metadata is used as the target omap; of course, if the metadata acquisition request is a request for acquiring files and subdirectories in a directory, you need to find the omap that stores the metadata of the files in the directory, and store the metadata in the directory. The omap of the metadata of the subdirectory of , uses both omaps as the target omap. After determining the target omap that stores the corresponding metadata, the corresponding metadata can be directly obtained from the target omap and sent to the client, thereby realizing the acquisition of the metadata.

It can be seen that this scheme classifies and stores the metadata of the content in the directory according to the difference between files and directories. Therefore, when the client obtains the metadata, whether it is to obtain the metadata of the file under the directory or the metadata of the subdirectory under the directory, it can directly obtain the corresponding metadata without the need to obtain the entire metadata of the directory. The metadata acquisition time is reduced, thereby reducing the processing time of the metadata acquisition request, reducing the disk load, and improving the metadata acquisition efficiency.

Based on the above embodiment, in this embodiment, the metadata of the files under the target directory and the metadata of the subdirectories under the target directory are stored in different omaps of the disk, including:

Traverse each target content under the target directory in turn, and determine the index node of each target content according to the directory entry of each target content;

Using the inode of each target content to determine the type of each target content, the type including files and subdirectories;

Cache the metadata of the target content of type file and the metadata of target content of type subdirectory into different data structures;

Store metadata in different data structures in different omaps on disk.

It should be noted that, in this application, when submitting a directory to a disk, it is necessary to traverse the content in the directory. In this solution, for the convenience of description, the content that is performing the operation is called the target content, and the target content can be is a file or subdirectory. Among them, the files or subdirectories under the directory have corresponding directory entries, and the directory entry is a structure including a file name and an inode number. Therefore, through the directory entry of the target content, the inode of the target content can be determined. Because Attributes and other related information are stored in the index node, so you can directly determine whether the type of the target content is a file or a subdirectory according to the index node of the target content, and convert the metadata of the target content of type file and the target content of type subdirectory. The metadata is cached in different omaps on disk.

It should be noted that, because the application needs to classify the metadata before storing it, in order to avoid the metadata classification process being too slow and the problem of the duration of metadata writing to the disk, the metadata corresponding to the file and the subdirectory can be stored first. The data information is stored in different data structures to realize the pre-classification and temporary storage of the metadata. After the metadata is classified, different OSD operation data structures can be created, and the metadata temporarily stored in the data structures can be stored separately. The information is filled into the OSD operation data structure, and the metadata storage operation is performed, so as to realize the storage of the metadata information of files and subdirectories in different omaps of rados respectively. In this way, when crawling the content in the directory, only the metadata information of the file or directory can be crawled according to the client's needs.

To sum up, it can be seen that the method for classifying and storing metadata information proposed in this application can reduce the processing time of metadata requests and the read and write load of osd, and improve the efficiency in business usage scenarios; First, the metadata of the target content of type file and the metadata of target content of type subdirectory are cached in different data structures, and then the cached metadata in different data structures are stored in different omaps of the disk respectively. In this way, the disk writing time can be reduced and the metadata writing efficiency can be improved.

The following describes the obtaining apparatus provided by the embodiments of the present invention, and the obtaining apparatus described below and the obtaining method described above may refer to each other.

Referring to FIG. 2, a schematic structural diagram of an apparatus for obtaining metadata provided by an embodiment of the present invention; the apparatus includes:

The first receiving module 100 is configured to receive a metadata acquisition request sent by a client; the metadata acquisition request is an acquisition request for acquiring a file under a directory, or an acquisition request for acquiring a subdirectory under the directory;

A determination module 200 is used to determine the target omap corresponding to the metadata acquisition request in the disk; wherein, the omap only stores the metadata of the file under the directory, or the metadata of the subdirectory under the directory;

The sending module 300 is configured to obtain metadata information from the target omap and send it to the client.

Among them, this program also includes:

a second receiving module, configured to receive a request for storing the target directory;

The storage module is configured to store the metadata of the files under the target directory and the metadata of the subdirectories under the target directory in different omaps of the disk.

Wherein, the storage module includes:

The first determining unit is used to traverse each target content under the target directory in turn, and determine the index node of each target content according to the directory entry of each target content;

a second determining unit, configured to determine the type of each target content by using the index node of each target content, and the type includes files and subdirectories;

The storage unit is used to store the metadata of the target content whose type is a file and the metadata of the target content whose type is a subdirectory into different omaps on the disk.

The storage unit is specifically configured to: cache the metadata of the target content whose type is a file and the metadata of the target content whose type is a subdirectory into different data structures; store the metadata in the different data structures respectively to different omaps on disk.

Referring to FIG. 3 , an embodiment of the present invention further discloses a schematic structural diagram of a device for obtaining metadata; the device includes:

memory 11 for storing computer programs;

The processor 12 is configured to implement the steps of the metadata acquisition method described in the above method embodiments when the computer program is executed.

In this embodiment, the device may be a PC (Personal Computer, personal computer), or may be a terminal device such as a smart phone, a tablet computer, a palmtop computer, and a portable computer.

The device may include a memory 11 , a processor 12 and a bus 13 .

The memory 11 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (eg, SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, and the like. The memory 11 may in some embodiments be an internal storage unit of the device, such as a hard disk of the device. In other embodiments, the memory 11 may also be an external storage device of the device, such as a plug-in hard disk equipped on the device, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card ( Flash Card), etc. Further, the memory 11 may also include both an internal storage unit of the device and an external storage device. The memory 11 can be used not only to store application software installed in the device and various types of data, such as program codes for executing the metadata acquisition method, etc., but also to temporarily store data that has been output or will be output.

The processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chips in some embodiments, for running program codes or processing stored in the memory 11 Data, such as program code that executes metadata acquisition methods, etc.

The bus 13 may be a peripheral component interconnect (PCI for short) bus or an extended industry standard architecture (Extended industry standard architecture, EISA for short) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in FIG. 3, but it does not mean that there is only one bus or one type of bus.

Further, the device may also include a network interface 14, and the network interface 14 may optionally include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used between the device and other electronic devices Establish a communication connection.

Optionally, the device may further include a user interface, and the user interface may include a display (Display), an input unit such as a keyboard (Keyboard), and an optional user interface may further include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like. Among them, the display may also be appropriately referred to as a display screen or a display unit, for displaying information processed in the device and for displaying a visual user interface.

FIG. 3 only shows the device with components 11-14. Those skilled in the art can understand that the structure shown in FIG. 3 does not constitute a limitation on the device, and may include fewer or more components than those shown. Either some components are combined, or different component arrangements.

An embodiment of the present invention further discloses a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the metadata acquisition method according to the above method embodiment is implemented A step of.

Wherein, the storage medium may include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other various storage media that can store program codes medium.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (6)

1. A metadata acquisition method, comprising:

receiving a metadata acquisition request sent by a client; the metadata acquisition request is an acquisition request for acquiring files under a directory or acquisition requests for acquiring sub-directories under the directory;

determining a target omap corresponding to the metadata acquisition request in a disk; wherein, the omap only stores the metadata of the files under the directory or the metadata of the subdirectories under the directory;

acquiring metadata information from the target omap and sending the metadata information to the client;

before receiving a metadata obtaining request sent by a client, the method further includes:

receiving a request to store a target directory;

storing the metadata of the files under the target directory and the metadata of the subdirectories under the target directory into different omaps of a disk; caching metadata of target content with the type of a file and metadata of target content with the type of a subdirectory into different data structures; and respectively storing the metadata in different data structures into different omaps of the disk.

2. The method according to claim 1, wherein the storing the metadata of the file in the target directory and the metadata of the subdirectory in the target directory to different omaps of a disk comprises:

sequentially traversing each target content under the target directory, and determining an index node of each target content according to a directory entry of each target content;

determining the type of each target content by using the index node of each target content, wherein the type comprises files and subdirectories;

and storing the metadata of the target content with the type of the file and the metadata of the target content with the type of the subdirectory into different omaps of the disk.

3. A metadata acquisition apparatus, comprising:

the first receiving module is used for receiving a metadata acquisition request sent by a client; the metadata acquisition request is an acquisition request for acquiring files under a directory or an acquisition request for acquiring sub-directories under the directory;

a determining module, configured to determine a target omap corresponding to the metadata obtaining request in a disk; wherein, the omap only stores the metadata of the files under the directory or the metadata of the subdirectories under the directory;

the sending module is used for obtaining metadata information from the target omap and sending the metadata information to the client;

a second receiving module, configured to receive a request for storing a target directory;

the storage module is used for storing the metadata of the files under the target directory and the metadata of the subdirectories under the target directory into different omaps of a disk; caching metadata of target content with the type of a file and metadata of target content with the type of a subdirectory into different data structures; and respectively storing the metadata in different data structures into different omaps of the disk.

4. The metadata acquisition apparatus according to claim 3, wherein the storage module comprises:

the first determining unit is used for sequentially traversing each target content under the target directory and determining an index node of each target content according to a directory entry of each target content;

a second determining unit, configured to determine a type of each target content using the inode of each target content, where the type includes a file and a subdirectory;

and the storage unit is used for storing the metadata of the target content with the type of the file and the metadata of the target content with the type of the subdirectory into different omaps of the disk.

5. A metadata acquisition apparatus characterized by comprising:

a memory for storing a computer program;

a processor for implementing the steps of the metadata acquisition method as claimed in claim 1 or 2 when executing said computer program.

6. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of the metadata acquisition method according to claim 1 or 2.

Images