CN110109865A - A kind of date storage method, device, equipment and readable storage medium storing program for executing - Google Patents

Abstract

The invention discloses a kind of date storage methods, this method first creates certain file amount before initiating file creation date write-in application in storage client according to unalterable rules, when storage client creation write-in application is sent, file created is directly returned into user, greatly reduce the time of document creation, entirely the response process of document creation request is not needed and Metadata Service communicates, determine that the empty file pre-created is written in the data flow being directly written into after data to be written, pre-creating mechanism by file reduces communication in data storage procedure between Metadata Service, reduce I/O path therein, improve the concurrency of document creation, shorten time delay, improve service response efficiency.The present invention also provides a kind of data storage device, equipment and a kind of readable storage medium storing program for executing, have above-mentioned beneficial effect.

Description

A data storage method, device, equipment and readable storage medium

technical field

The present invention relates to the technical field of data processing, in particular to a data storage method, device, equipment and a readable storage medium.

Background technique

At present, with the increase in the amount of interactive data, the system has higher requirements for data storage efficiency, especially for some application scenarios that store a large amount of data and require high timeliness. For example, there are often scenarios where files are created at a fixed time and are instantaneously high in concurrency, such as Video surveillance business scenario. Video surveillance services have extremely high requirements for video stream storage. In order to meet the requirement of no frame loss, the maximum delay for creating and writing stored files cannot exceed 500ms, and there are also requirements for the average delay.

In the current data storage method, when the storage client receives a file creation and writing request, it needs to control the background to create a new file, and write the data after the creation is successful. Among them, since the file creation process requires background metadata services to control multiple parties to participate in the interaction, it takes a long time for a certain time file creation data to be written to high-concurrency scenarios, which will cause front-end users to respond slowly or even stagnate, affecting user experience.

Therefore, how to optimize the time occupied by the creation of the writing process for the data storage process is a technical problem to be solved by those skilled in the art.

Contents of the invention

The purpose of the present invention is to provide a data storage method, which can optimize the time occupied by the creation and writing process for the data storage process, and reduce the pressure on instantaneous metadata services; another purpose of the present invention is to provide a data storage device, equipment and A readable storage medium has the above beneficial effects.

In order to solve the above technical problems, the present invention provides a data storage method, including:

Create an empty file when the storage service mounts;

Create a maintenance linked list for each of the empty files to obtain an empty file linked list;

When a file creation write request is received, determine the target data stream to be written;

Determine the pre-created empty file as the data receiving end according to the empty file linked list, and obtain the target file;

calling the target file to receive and store the target data stream.

Optionally, determining a pre-created empty file as a data receiving end according to the empty file linked list includes:

Obtain the first empty file in the empty file linked list;

Perform name replacement on the obtained first empty file according to the user name specified by the user, and use the obtained file as the target file.

Optionally, after determining the pre-created empty file as the data receiving end according to the empty file linked list, the method further includes:

Output the prompt message that the file is successfully created.

Optionally, after determining the pre-created empty file as the data receiving end according to the empty file linked list, the method further includes:

Create empty files identical to the target number; wherein, the target number corresponds to the number of occupied files in the empty file linked list;

Add the newly created empty files of the target number to the empty file linked list.

To achieve the above purpose, the present application also discloses a data storage device, comprising:

The first empty file creation unit is used to create an empty file when the storage service is mounted;

A linked list creation unit, configured to create a maintenance linked list for each of the empty files to obtain an empty file linked list;

A data stream determining unit, configured to determine a target data stream to be written when a file creation and writing request is received;

A target file determining unit, configured to determine a pre-created empty file as a data receiving end according to the empty file linked list, and obtain the target file;

The target file calling unit is used to call the target file to receive and store the target data stream.

Optionally, the target file determination unit includes:

A file acquisition subunit, configured to acquire the first empty file in the empty file linked list;

The name replacement subunit is configured to perform name replacement on the acquired first empty file according to the user name specified by the user, and use the obtained file as the target file.

Optionally, the data storage device further includes: a prompt unit connected to the output terminal of the target file determining unit, and the prompt unit is configured to output prompt information indicating that the file creation is successful.

Optionally, the data storage device further includes: a second empty file creation unit, the second empty file creation unit is connected to the output end of the target file determination unit, and the second empty file creation unit includes:

Create a subunit for creating a target number of empty files; wherein, the target number corresponds to the number of occupied files in the empty file linked list;

The adding subunit is used to add the newly created empty files of the target number to the empty file linked list.

In order to achieve the above purpose, this application also discloses a data storage device, including:

memory for storing computer programs;

A processor configured to implement the steps of the above data storage method when executing the computer program.

To achieve the above purpose, the present application also discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor, the steps of the above-mentioned data storage method are realized.

In the data storage method provided by the present invention, the storage client first creates a certain amount of files according to fixed rules before initiating the application for file creation and data writing, and when the storage client sends the application for creation and writing, the created files are directly The file is returned to the user, which greatly reduces the time of file creation. The entire response process to the file creation request does not need to communicate with the metadata service. After confirming the data to be written, the data stream to be written is directly written into the pre-created space. Files are enough. Through the file pre-creation mechanism, the communication between the data storage process and the metadata service is reduced, the IO path is reduced, the concurrency of file creation is increased, the delay is shortened, and the business response efficiency is improved.

The present invention also provides a data storage device, equipment and a readable storage medium, which have the above-mentioned beneficial effects, and will not be repeated here.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

FIG. 1 is a flow chart of a data storage method provided by an embodiment of the present invention;

FIG. 2 is a structural block diagram of a data storage device provided by an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a data storage device provided by an embodiment of the present invention.

Detailed ways

The core of the present invention is to provide a data storage method, which can optimize the time occupation of the creation and writing process for the data storage process, and reduce the pressure on instantaneous metadata services; another core of the present invention is to provide a data storage device, equipment and A readable storage medium.

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Embodiment one:

Please refer to Fig. 1, Fig. 1 is the flowchart of the data storage method that the present embodiment provides; This method mainly comprises the following steps:

Step s110: Create an empty file when the storage service is mounted.

The number of created empty files can be set according to the actual application scenario. For example, in an application scenario where data streams of 1000 files need to be stored at a fixed time, 1000 empty files can be created before the fixed time. The number of created empty files can be fixed or not. For scenarios with a fixed number of files and high concurrency instantaneously, you can first create a specified number of empty files in a fixed directory when the storage service is mounted. For a large number of variable The number of created files may not be fixed in the file creation scenario. For example, empty files may be created at any time when the metadata service pressure is low. In this embodiment, the number of created empty files is not limited. The number of empty files can be set via the configuration file. This function can be provided by the client directly accessing the storage cluster in the form of mounting.

The creation of empty files needs to be completed before the high concurrency of data storage services occurs. In this embodiment, the specific creation time is not limited. In order to ensure the optimal processing of storage services in various time periods under uncertain times, it can be mounted on the data storage service. Start the creation of empty files at the same time. Of course, you can also consider other factors to create empty files in other time periods when the storage service is mounted. For example, if the metadata service is under high pressure while the data storage service is mounted, you can Empty files are created when the pressure on the metadata service is low (the overall need to call the metadata service is less). Here we only take the creation time of the above two kinds of empty files as examples to introduce. The creation time settings in other specific application scenarios are not limited here, and the empty file can be created as early as possible according to the actual situation.

Step s120: Create a maintenance linked list for each empty file to obtain an empty file linked list.

The local client maintains a linked list including all empty files. The empty file linked list mainly includes information about the pre-created empty files to be called, and may also include all pre-created empty file calls. The empty file linked list is mainly used for calling the empty file, and the calling of the empty file linked list for other purposes is not limited here.

The empty file linked list can be created after the empty file is created, or the empty file linked list can be created first, and then the newly created empty file can be added to the linked list for empty file management, or the empty file linked list can be created at the same time as the empty file is created. Here, there is no limitation on the execution sequence of creating an empty file and creating an empty file maintenance linked list. For the process of creating the crouching tiger linked list of empty files, please refer to the introduction in the relevant technology, and will not repeat them here.

Step s130: When a file creation and writing request is received, determine the target data stream to be written.

Step s140: Determine the pre-created empty file as the data receiving end according to the empty file link list, and obtain the target file.

The file creation and writing request means that there is new data to be written into the newly created file, the target data stream to be written is determined, and the current target file for data writing is determined from the pre-created empty files. Using the pre-created empty file as the target file for currently receiving data can not only avoid the service pressure on the metadata service during the new file creation process, but also reduce the service delay caused by the file creation process. In the system, the performance of file creation is improved through the pre-creation mechanism. After pre-creation, it waits for the application of client file creation, and returns the created file without delay. For high-concurrency file creation scenarios, when a high-concurrency file creation request is sent to the storage, the created empty file can be returned to the user in an instant, which improves the performance of the user's file creation and meets the client's high-concurrency file creation needs. , the business operation is smoother and more stable. For example, in some video streaming scenarios, the size of the file written by each camera is fixed, and a new file will be created after reaching the fixed size, so thousands of cameras will create files at the same time. The pre-creation mechanism provided in this embodiment can significantly improve the data storage efficiency in this scenario and improve the competitiveness of the product.

It should be noted that, when a file creation and writing request is received, the execution sequence of determining the target data stream to be written and determining the target file is not limited in this embodiment. The target data stream can be determined first and then the target file can be determined. The target file can be determined first and then the target data flow can be determined, or can be executed at the same time, and can be set according to the actual application scenario.

The process of determining the target file according to the empty file link list is not limited here, and any pre-created empty file can be selected as the target file. Among them, in order to reduce the resources and time occupied in the file selection process, the first empty file in the empty file list can be obtained; the name of the first empty file obtained is replaced according to the user name specified by the user, and the obtained file is used as the target file . When the storage client sends a request to create a file, rename the first empty file in the linked list. Rename is the file name created by the user, which reduces the communication with the metadata service and reduces the IO path. Reduce latency and improve file creation performance.

Step s150: calling the target file to receive and store the target data stream.

For the process of invoking the target file to receive and store the target data stream, refer to the data writing method in the related art, which will not be repeated here.

In addition, after the target file is determined from the pre-created empty file in the above step s140, a prompt message that the file is successfully created can be further output, which can prompt the user to execute the steps and also feel the improvement of the response speed from the user end . Of course, it is also possible not to output the prompt information, or to output the prompt information at other time points, which is not limited in this embodiment.

Based on the above introduction, the data storage method disclosed in the embodiment of the present invention creates a certain amount of files according to fixed rules before the storage client initiates the file creation and data writing application, and when the storage client sends the creation and writing application, Directly return the created file to the user, greatly reducing the time for creating the file. The entire response process to the file creation request does not need to communicate with the metadata service. After confirming the data to be written, directly write the data stream to be written Just enter the pre-created empty file. Through the file pre-creation mechanism, the communication between the data storage process and the metadata service is reduced, the IO path in it is reduced, the concurrency of file creation is increased, the delay is shortened, and the Business responsiveness.

Embodiment two:

Based on the above-mentioned embodiments, in order to ensure the continuous response speed of the background while the client continuously receives data storage requests, it is possible to create empty files with the same number as the target after determining the pre-created empty files as the data receiving end according to the empty file link list. files; wherein, the target number corresponds to the number of occupied files in the empty file linked list; and the newly created empty files of the target number are added to the empty file linked list. That is, create empty files here after the number of empty files is reduced, maintain the number of pre-created empty files in the system, and avoid the situation of efficiency decline caused by insufficient number of empty files in the case of continuous large amount of data.

Among them, the process of creating a new file can be carried out after each rename, that is, a new empty file is created immediately after calling an empty file, so as to ensure that the number of empty files in the fixed directory remains unchanged; it can also be used when the number of empty files is less than a certain Create a large number of empty files after the threshold. For example, when the number of empty files is less than 100, start the creation of 900 empty files to prepare for the next high-concurrency creation request. This reduces pressure peaks on the metadata service. In this embodiment, there is no limitation on the execution time period for supplementing new files, which can be set according to actual needs. Of course, only one new file creation process can be performed at the same time of a single storage service mount, that is, no empty file supplement is performed, and there is no limitation.

Embodiment three:

Please refer to Fig. 2, Fig. 2 is the structural block diagram of the data storage device that this embodiment provides; The object file calls unit 250 . The data storage device provided in this embodiment can be compared with the above data storage method.

Wherein, the first empty file creation unit 210 is mainly used to create an empty file when the storage service is mounted;

Linked list creation unit 220 is mainly used for creating maintenance linked list for each empty file, obtains empty file linked list;

The data stream determination unit 230 is mainly used to determine the target data stream to be written when a file creation write request is received;

The target file determination unit 240 is mainly used to determine the pre-created empty file as the data receiving end according to the empty file linked list to obtain the target file;

The target file calling unit 250 is mainly used to call the target file to receive and store the target data stream.

The data storage device provided in this embodiment can optimize the time occupied by the creation and writing process for the data storage process, and reduce the pressure on the instantaneous metadata service.

Optionally, the target file determining unit may specifically include:

The file acquisition subunit is used to acquire the first empty file in the empty file linked list;

The name replacement subunit is used to replace the name of the first empty file obtained according to the user name specified by the user, and use the obtained file as the target file.

Optionally, the data storage device provided in this embodiment may further include: a prompt unit connected to the output terminal of the target file determination unit, and the prompt unit is configured to output prompt information indicating that the file is created successfully.

Optionally, the data storage device provided in this embodiment may further include: a second empty file creation unit, the second empty file creation unit is connected to the output end of the target file determination unit, and the second empty file creation unit includes:

Create a subunit for creating a target number of empty files; wherein, the target number corresponds to the number of occupied files in the empty file linked list;

The adding subunit is used for adding the newly created empty files of the target number to the empty file linked list.

Embodiment four:

The following is an introduction to the data storage device provided by this application. The specific introduction to the data storage device can refer to the steps of the above data storage method. The device mainly includes:

memory for storing computer programs;

The processor is configured to implement the steps of the above data storage method when executing the computer program.

Please refer to FIG. 3 , which is a schematic structural diagram of a data storage device provided by an embodiment of the present application. The data storage device may have relatively large differences due to different configurations or performances, and may include one or more than one processor (central processing units, CPU) 322 (eg, one or more processors) and memory 332, one or more storage media 330 (eg, one or more mass storage devices) for storing application programs 342 or data 344. Wherein, the memory 332 and the storage medium 330 may be temporary storage or persistent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the data processing device. Furthermore, the central processing unit 322 may be configured to communicate with the storage medium 330 , and execute a series of instruction operations in the storage medium 330 on the data storage device 301 .

The data storage device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and/or, one or more operating systems 341, such as Windows Server™, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

The steps in the data storage method described in FIG. 1 above can be realized by the structure of the data storage device.

Embodiment five:

The readable storage medium provided by the embodiment of the present application is introduced below, and the readable storage medium described below and the data storage method described above may be referred to in correspondence.

A readable storage medium disclosed in the present application stores a program on it, and when the program is executed by a processor, the steps of the data storage method are realized.

Specifically, the readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like that can store program codes. readable storage media.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described devices, equipment, storage media and units can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here. .

In the several embodiments provided in this application, it should be understood that the disclosed device, system, storage medium and method may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, for example, multiple units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

A unit described as a separate component may or may not be physically separated, and a component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

An integrated unit may be stored in a mobile terminal if it is realized in the form of a software function unit and sold or used as an independent product. Based on this understanding, the essence of the technical solution of this application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including Several instructions are used to make a mobile terminal (which may be a mobile phone, or a tablet computer, etc.) execute all or part of the steps of the methods in various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other various media that can store program codes. .

Each embodiment in the description is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the related information, please refer to the description of the method part.

Professionals can further realize that the units and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, terminals, or a combination of the two. In order to clearly illustrate the interoperability of hardware and software Alternatively, in the above description, the components and steps of each example have been generally described according to their functions. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be directly implemented by hardware, software modules executed by a processor, or a combination of both. Software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

The data storage method, device, equipment and readable storage medium provided by the present application have been introduced in detail above. In this paper, specific examples are used to illustrate the principles and implementation methods of the present application, and the descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application. It should be pointed out that those skilled in the art can make some improvements and modifications to the application without departing from the principles of the application, and these improvements and modifications also fall within the protection scope of the claims of the application.

Claims (10)

1. a kind of date storage method characterized by comprising

Empty file is created when storage service carry；

For each empty file creation maintenance chained list, empty file chained list is obtained；

When receiving document creation write request, target data stream to be written is determined；

Empty file is pre-created according to what the empty file chained list was determined as data receiver, obtains file destination；

The file destination is called to receive and store the target data stream.

2. date storage method as described in claim 1, which is characterized in that be determined as data according to the empty file chained list Receiving end pre-creates empty file, comprising:

Obtain the first empty file in the empty file chained list；

The user name specified according to user carries out title replacement to the first empty file of acquisition, using obtained file as institute State file destination.

3. date storage method as described in claim 1, which is characterized in that be determined as number according to the empty file chained list According to receiving end pre-create empty file after, further includes:

The output creation successful prompt information of file.

4. the date storage method stated as described in any one of claims 1 to 3, which is characterized in that according to the empty file chain Table be determined as data receiver pre-create empty file after, further includes:

Create empty file identical with target number；Wherein, number of files is occupied in the target number and the empty file chained list Amount corresponds to；

The empty file of the newly created target number is added in the empty file chained list.

5. a kind of data storage device characterized by comprising

First empty file creating unit, for creating empty file when storage service carry；

Chained list creating unit obtains empty file chained list for safeguarding chained list for each empty file creation；

Data flow determination unit, for determining target data stream to be written when receiving document creation write request；

File destination determination unit pre-creates empty file for be determined as data receiver according to the empty file chained list, Obtain file destination；

File destination call unit, for calling the file destination to receive and store the target data stream.

6. data storage device as claimed in claim 5, which is characterized in that the file destination determination unit includes:

File acquisition subelement, for obtaining the first empty file in the empty file chained list；

Title replaces subelement, and the user name for being specified according to user carries out title to the first empty file of acquisition and replaces It changes, using obtained file as the file destination.

7. data storage device as claimed in claim 5, which is characterized in that further include: prompt unit, the prompt unit with The output end of the file destination determination unit connects, and the prompt unit is for exporting the creation successful prompt information of file.

8. such as the described in any item data storage devices of claim 5 to 7, which is characterized in that further include: the creation of the second empty file Unit, the second empty file creating unit are connect with the output end of the file destination determination unit, second empty file Creating unit includes:

Subelement is created, for creating the empty file of target number；Wherein, in the target number and the empty file chained list It is corresponding to occupy quantity of documents；

Subelement is added, for the empty file of the newly created target number to be added in the empty file chained list.

9. a kind of data storage device characterized by comprising

Memory, for storing computer program；

Processor realizes the data storage side as described in Claims 1-4 any one when for executing the computer program The step of method.

10. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium Program realizes the data storage as described in Claims 1-4 any one when the computer program is executed by the processor The step of method.