CN114564149B - Data storage method, device, equipment and storage medium - Google Patents
Data storage method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114564149B CN114564149B CN202210175833.6A CN202210175833A CN114564149B CN 114564149 B CN114564149 B CN 114564149B CN 202210175833 A CN202210175833 A CN 202210175833A CN 114564149 B CN114564149 B CN 114564149B
- Authority
- CN
- China
- Prior art keywords
- current
- memory block
- storage memory
- storage
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 68
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000004590 computer program Methods 0.000 claims description 16
- 238000007726 management method Methods 0.000 claims description 9
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage method, a data storage device, data storage equipment and a data storage medium. Acquiring a current small file to be stored, and determining the current occupied space of the current small file; determining a current storage memory block of the current small file according to the current occupied space, wherein the current storage memory block is a target storage memory block of a previous small file for storing the current small file or a next data storage memory block associated with the target storage memory block; and storing the current small file into the current storage memory block. According to the invention, the plurality of small files are combined and stored in the same data storage memory block, so that the disk space is effectively utilized, and the data reading and writing speed is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data storage method, apparatus, device, and storage medium.
Background
In order to make the data transmission speed close to the hard disk transmission speed, the HDFS (Hadoop distributed file system) is designed to relatively minimize Seek time (Seek) and set the size of a data Block (Block) to be larger, so that the time for reading and writing the data Block is far longer than the Seek time and is close to the hard disk transmission speed, but when the file is too small, the Seek time is longer than the data read-write time.
A small file refers to a file that has a file size that is significantly smaller than the HDFS data block size. At present, any file in the HDFS occupies a Block data Block, when there are too many small files, not only the Block data Block is wasted, but also each file, directory or data Block is represented by metadata in the form of an object in the memory of the NameNode node. When processing files, each small file occupies one slot, so most of time is spent on a start task and a release task.
Disclosure of Invention
The invention provides a data storage method, a device, equipment and a storage medium, which are used for realizing the effects of effectively utilizing disk space and improving the read-write speed of data.
According to an aspect of the present invention, there is provided a data storage method comprising:
acquiring a current small file to be stored, and determining the current occupied space of the current small file;
determining a current storage memory block of the current small file according to the current occupied space, wherein the current storage memory block is a target storage memory block of a previous small file for storing the current small file or a next data storage memory block associated with the target storage memory block;
and storing the current small file into the current storage memory block.
Optionally, the determining, according to the current occupied space, a current storage memory block of the current small file includes:
acquiring a target residual storage space of a target storage memory block;
if the target storage space is larger than or equal to the current occupied space, determining the target storage memory block as a current storage memory block; otherwise the first set of parameters is selected,
and determining the next data storage memory block associated with the target storage memory block as the current storage memory block.
Optionally, while storing the current small file in the current storage memory block, the method further includes:
forming current file metadata of the current small file, and storing the current file metadata into a file directory management memory block, wherein the current file metadata comprises a file name of the current small file, storage block information of the current storage memory block and storage position information of the current small file in the current storage memory block.
Optionally, the method further comprises:
monitoring memory free spaces of all non-empty memory blocks, wherein the non-empty memory blocks are data memory blocks for storing at least one small file;
determining the non-empty storage memory blocks with the memory empty space larger than or equal to a space empty threshold value as memory blocks to be adjusted;
and when the number of the memory blocks to be adjusted is equal to an adjustment threshold value, adjusting the storage positions of the small files in the memory blocks to be adjusted.
According to another aspect of the present invention, there is provided a data storage device comprising:
the current occupation space determining module is used for acquiring a current small file to be stored and determining the current occupation space of the current small file;
the current storage memory block determining module is used for determining a current storage memory block of the current small file according to the current occupied space, wherein the current storage memory block is a target storage memory block of a previous small file of the current small file or a next data storage memory block associated with the target storage memory block;
and the file data storage module is used for storing the current small file into the current storage memory block.
Optionally, the current storage memory block determining module includes:
the target residual storage space acquisition unit is used for acquiring the target residual storage space of the target storage memory block;
a current storage memory block determining unit, configured to determine that the target storage memory block is a current storage memory block if the target storage space is greater than or equal to the current occupied space; otherwise, determining the next data storage memory block associated with the target storage memory block as the current storage memory block.
Optionally, the apparatus further comprises a file directory storage module for:
and forming current file metadata of the current small file while storing the current small file into the current storage memory block, and storing the current file metadata into a file directory management memory block, wherein the current file metadata comprises a file name of the current small file, storage block information of the current storage memory block and storage position information of the current small file in the current storage memory block.
Optionally, the apparatus further comprises a storage location adjustment module for:
monitoring memory free spaces of all non-empty memory blocks, wherein the non-empty memory blocks are data memory blocks for storing at least one small file;
determining the non-empty storage memory blocks with the memory empty space larger than or equal to a space empty threshold value as memory blocks to be adjusted;
and when the number of the memory blocks to be adjusted is equal to an adjustment threshold value, adjusting the storage positions of the small files in the memory blocks to be adjusted.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data storage method of any one of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to execute a data storage method according to any embodiment of the present invention.
According to the technical scheme, the current small file to be stored is obtained, and the current occupied space of the current small file is determined; determining a current storage memory block of a current small file according to the current occupied space, wherein the current storage memory block is a target storage memory block of a previous small file for storing the current small file or a next data storage memory block associated with the target storage memory block; and storing the current small file into the current storage memory block. According to the invention, the plurality of small files are combined and stored in the same data storage memory block, so that the disk space is effectively utilized, and the data reading and writing speed is improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a data storage method according to a first embodiment of the present invention;
FIG. 2 is a schematic diagram of a data storage device according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device implementing a data storage method according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "current," "previous," "target," and the like in the description and claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
Fig. 1 is a flowchart of a data storage method according to an embodiment of the present invention, where the method may be performed by a data storage device, and the data storage device may be implemented in hardware and/or software, and the data storage device may be configured in a computer device. As shown in fig. 1, the method includes:
s110, acquiring a current small file to be stored, and determining the current occupied space of the current small file.
In the prior art, one HDFS small file occupies one data storage memory block, and in this embodiment, a plurality of small files are stored in one data storage memory block, so as to save the data storage memory block.
In this embodiment, the data storage memory block may be understood as a DataNode memory block. A small file refers to a file that has a file size that is significantly smaller than the memory size of the memory blocks in the HDFS. The current doclet may be understood as the doclet to be stored at hand. Determining the current footprint of the current doclet may be understood as the memory space required to store the current doclet.
S120, determining a current storage memory block of the current small file according to the current occupied space, wherein the current storage memory block is a target storage memory block of a previous small file for storing the current small file or a next data storage memory block associated with the target storage memory block.
The current storage memory block may be understood as a data storage memory block in which a current small file is to be stored. The previous doclet of the current doclet may be understood as the doclet stored immediately before the current doclet. A target storage memory block may be understood as a data storage memory block storing a previous doclet.
In this embodiment, the small files may be stored continuously from the first data storage memory block, that is, one data storage memory block may store a plurality of small files. When a small file needs to be stored, the data storage memory block where the small file is to be stored needs to be determined first.
Alternatively, S120 may be specifically implemented by the following steps:
s1201, acquiring a target residual storage space of a target storage memory block.
In this embodiment, after storing the previous small file of the current small file, the data storage memory block storing the previous small file may be used as the target storage memory block, and when storing the current small file, the spare storage space of the target storage memory block, that is, the target remaining storage space, may be determined first, so as to determine whether the target storage memory block may store the current small file.
S1202, judging whether the target storage space is larger than or equal to the current occupied space.
Specifically, when the current occupied space of the current small file and the target remaining storage space of the target storage memory block are obtained, the current occupied space and the target remaining storage space can be compared, and when the target storage space is greater than or equal to the current occupied space, S1203 can be performed; otherwise, S1204 is performed.
S1203, determining the target storage memory block as the current storage memory block.
Specifically, when the target storage space is greater than or equal to the current occupied space, it can be understood that the spare storage space of the target storage memory block satisfies the requirement of storing the current small file, and the current small file can be stored in the target storage memory block, so that the target storage memory block can be determined as the current storage memory block.
S1204, determining the next data storage memory block associated with the target storage memory block as the current storage memory block.
Specifically, when the target storage space is smaller than the current occupied space, it may be understood that the free storage space of the target storage memory block does not satisfy the storage of the current small file, and the current small file may be stored in a new data storage memory block, so that the next data storage memory block associated with the target storage memory block may be determined as the current storage memory block. In this embodiment, the association relationship of the data storage memory blocks may be a numbering sequence relationship of the memory blocks.
S130, storing the current small file into a current storage memory block.
Specifically, a current storage memory block in which the current small file is to be stored is determined, and the current small file is stored in the current storage memory block.
According to the embodiment of the invention, the current occupied space of the current small file is determined by acquiring the current small file to be stored; determining a current storage memory block of a current small file according to the current occupied space, wherein the current storage memory block is a target storage memory block of a previous small file for storing the current small file or a next data storage memory block associated with the target storage memory block; and storing the current small file into the current storage memory block. According to the embodiment of the invention, the plurality of small files are combined and stored in the same data storage memory block, so that the disk space is effectively utilized, and the data reading and writing speed is improved.
On the basis of the above technical solution, the data storage method provided in this embodiment may further include the following steps:
and S140, forming current file metadata of the current small file, and storing the current file metadata into a file directory management memory block, wherein the current file metadata comprises the file name of the current small file, the storage block information of the current storage memory block and the storage position information of the current small file in the current storage memory block.
The file directory management memory block may be understood as a NameNode memory block.
In this embodiment, when the current small file is stored, the current file metadata of the current small file may be formed according to the file name of the current small file, the storage block information of the current storage memory block, and the storage position information of the current small file in the current storage memory block, and the current file metadata may be stored in the file directory management memory block NameNode, and when the small file is accessed, the search may be performed according to the file information in the NameNode.
S150, monitoring memory free spaces of all non-empty memory blocks; determining a non-empty memory block with the memory empty space being greater than or equal to a space empty threshold value as a memory block to be adjusted; and when the number of the memory blocks to be adjusted is equal to the adjustment threshold value, adjusting the storage positions of the small files in the memory blocks to be adjusted.
A non-empty storage memory block is understood to be a data storage memory block in which at least one small file is stored.
In this embodiment, the memory space of each non-empty memory block may be monitored in real time, and when the memory space of a certain non-empty memory block is greater than or equal to a preset space free threshold due to the file deletion, the non-empty memory block may be determined as a memory block to be adjusted. When the number of the memory blocks to be adjusted in the magnetic disk is large, for example, when the number of the memory blocks to be adjusted reaches a preset adjustment threshold, the storage position of the small file in the memory blocks to be adjusted can be adjusted. For example, when the memory space of a certain non-empty memory block is greater than or equal to 100MB, determining the non-empty memory block as a memory block to be adjusted; when the number of the memory blocks to be adjusted reaches 1000, the storage positions of the small files in the 1000 memory blocks to be adjusted are adjusted, and the small files can be continuously stored in the memory blocks to be adjusted with the front numbers according to the data storage method so as to release the idle memory blocks to be adjusted.
Example two
Fig. 2 is a schematic structural diagram of a data storage device according to a second embodiment of the present invention. As shown in fig. 2, the apparatus includes:
the current occupation space determining module 210 is configured to obtain a current small file to be stored, and determine a current occupation space of the current small file.
The current storage memory block determining module 220 is configured to determine, according to the current occupied space, a current storage memory block of the current small file, where the current storage memory block is a target storage memory block of a previous small file storing the current small file or a next data storage memory block associated with the target storage memory block.
A file data storage module 230 for storing the current small file into the current storage memory block
Optionally, the current storage memory block determining module 220 includes:
the target residual storage space acquisition unit is used for acquiring the target residual storage space of the target storage memory block;
a current storage memory block determining unit, configured to determine that the target storage memory block is a current storage memory block if the target storage space is greater than or equal to the current occupied space; otherwise, determining the next data storage memory block associated with the target storage memory block as the current storage memory block.
Optionally, the device further includes a file directory storage module, configured to:
and forming current file metadata of the current small file while storing the current small file into the current storage memory block, and storing the current file metadata into a file directory management memory block, wherein the current file metadata comprises a file name of the current small file, storage block information of the current storage memory block and storage position information of the current small file in the current storage memory block.
Optionally, the apparatus further includes a storage location adjustment module configured to:
monitoring memory free spaces of all non-empty memory blocks, wherein the non-empty memory blocks are data memory blocks for storing at least one small file;
determining the non-empty storage memory blocks with the memory empty space larger than or equal to a space empty threshold value as memory blocks to be adjusted;
and when the number of the memory blocks to be adjusted is equal to an adjustment threshold value, adjusting the storage positions of the small files in the memory blocks to be adjusted.
The data storage device provided by the embodiment of the invention can execute the data storage method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Example III
Fig. 3 shows a schematic diagram of the structure of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 3, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as data storage methods.
In some embodiments, the data storage method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. One or more of the steps of the data storage method described above may be performed when the computer program is loaded into RAM 13 and executed by processor 11. Alternatively, in other embodiments, the processor 11 may be configured to perform the data storage method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (6)
1. A method of data storage, comprising:
acquiring a current small file to be stored, and determining the current occupied space of the current small file;
determining a current storage memory block of the current small file according to the current occupied space, wherein the current storage memory block is a target storage memory block of a previous small file for storing the current small file or a next data storage memory block associated with the target storage memory block;
storing the current small file into the current storage memory block;
monitoring memory free spaces of all non-empty memory blocks, wherein the non-empty memory blocks are data memory blocks for storing at least one small file;
determining the non-empty storage memory blocks with the memory empty space larger than or equal to a space empty threshold value as memory blocks to be adjusted;
when the number of the memory blocks to be adjusted is equal to an adjustment threshold value, adjusting the storage positions of the small files in the memory blocks to be adjusted;
the determining, according to the current occupied space, a current storage memory block of the current small file includes:
acquiring a target residual storage space of a target storage memory block;
if the target remaining storage space is larger than or equal to the current occupied space, determining the target storage memory block as a current storage memory block; otherwise the first set of parameters is selected,
and determining the next data storage memory block associated with the target storage memory block as the current storage memory block.
2. The method of claim 1, wherein storing the current doclet into the current storage memory block, simultaneously with storing the current doclet into the current storage memory block, further comprises:
forming current file metadata of the current small file, and storing the current file metadata into a file directory management memory block, wherein the current file metadata comprises a file name of the current small file, storage block information of the current storage memory block and storage position information of the current small file in the current storage memory block.
3. A data storage device, comprising:
the current occupation space determining module is used for acquiring a current small file to be stored and determining the current occupation space of the current small file;
the current storage memory block determining module is used for determining a current storage memory block of the current small file according to the current occupied space, wherein the current storage memory block is a target storage memory block of a previous small file of the current small file or a next data storage memory block associated with the target storage memory block;
the file data storage module is used for storing the current small file into the current storage memory block;
the storage position adjusting module is used for monitoring the memory space of all non-empty storage memory blocks, wherein the non-empty storage memory blocks are data storage memory blocks for storing at least one small file; determining the non-empty storage memory blocks with the memory empty space larger than or equal to a space empty threshold value as memory blocks to be adjusted; when the number of the memory blocks to be adjusted is equal to an adjustment threshold value, adjusting the storage positions of the small files in the memory blocks to be adjusted;
the current storage memory block determining module comprises:
the target residual storage space acquisition unit is used for acquiring the target residual storage space of the target storage memory block;
a current storage memory block determining unit, configured to determine that the target storage memory block is a current storage memory block if the target remaining storage space is greater than or equal to the current occupied space; otherwise, determining the next data storage memory block associated with the target storage memory block as the current storage memory block.
4. The apparatus of claim 3, further comprising a file directory storage module for:
and forming current file metadata of the current small file while storing the current small file into the current storage memory block, and storing the current file metadata into a file directory management memory block, wherein the current file metadata comprises a file name of the current small file, storage block information of the current storage memory block and storage position information of the current small file in the current storage memory block.
5. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data storage method of any one of claims 1-2.
6. A computer readable storage medium storing computer instructions for causing a processor to perform the data storage method of any one of claims 1-2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210175833.6A CN114564149B (en) | 2022-02-25 | 2022-02-25 | Data storage method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210175833.6A CN114564149B (en) | 2022-02-25 | 2022-02-25 | Data storage method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114564149A CN114564149A (en) | 2022-05-31 |
| CN114564149B true CN114564149B (en) | 2024-03-26 |
Family
ID=81715424
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210175833.6A Active CN114564149B (en) | 2022-02-25 | 2022-02-25 | Data storage method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114564149B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116107763B (en) * | 2023-04-12 | 2023-06-20 | 北京燧原智能科技有限公司 | Data transmission method, device, equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101866359A (en) * | 2010-06-24 | 2010-10-20 | 北京航空航天大学 | Small file storage and visit method in avicade file system |
| CN107391280A (en) * | 2017-07-31 | 2017-11-24 | 郑州云海信息技术有限公司 | A kind of reception of small documents and storage method and device |
| CN112434000A (en) * | 2020-11-20 | 2021-03-02 | 苏州浪潮智能科技有限公司 | Small file merging method, device and equipment based on HDFS |
| CN113111038A (en) * | 2021-03-31 | 2021-07-13 | 北京达佳互联信息技术有限公司 | File storage method, device, server and storage medium |
| CN113448938A (en) * | 2021-07-20 | 2021-09-28 | 恒安嘉新（北京）科技股份公司 | Data processing method and device, electronic equipment and storage medium |
| CN113971163A (en) * | 2020-07-23 | 2022-01-25 | 深圳市杉岩数据技术有限公司 | Small file merging storage method, small file reading method and server |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2867589A1 (en) * | 2013-10-15 | 2015-04-15 | Coho Data Inc. | Systems, methods and devices for implementing data management in a distributed data storage system |
-
2022
- 2022-02-25 CN CN202210175833.6A patent/CN114564149B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101866359A (en) * | 2010-06-24 | 2010-10-20 | 北京航空航天大学 | Small file storage and visit method in avicade file system |
| CN107391280A (en) * | 2017-07-31 | 2017-11-24 | 郑州云海信息技术有限公司 | A kind of reception of small documents and storage method and device |
| CN113971163A (en) * | 2020-07-23 | 2022-01-25 | 深圳市杉岩数据技术有限公司 | Small file merging storage method, small file reading method and server |
| CN112434000A (en) * | 2020-11-20 | 2021-03-02 | 苏州浪潮智能科技有限公司 | Small file merging method, device and equipment based on HDFS |
| CN113111038A (en) * | 2021-03-31 | 2021-07-13 | 北京达佳互联信息技术有限公司 | File storage method, device, server and storage medium |
| CN113448938A (en) * | 2021-07-20 | 2021-09-28 | 恒安嘉新（北京）科技股份公司 | Data processing method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114564149A (en) | 2022-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114564149B (en) | Data storage method, device, equipment and storage medium | |
| CN116796085A (en) | File processing method and device, electronic equipment and storage medium | |
| CN116594563A (en) | Distributed storage capacity expansion method and device, electronic equipment and storage medium | |
| CN115438007A (en) | File merging method and device, electronic equipment and medium | |
| CN112887426B (en) | Information stream pushing method and device, electronic equipment and storage medium | |
| CN115905322A (en) | Service processing method and device, electronic equipment and storage medium | |
| CN115617800A (en) | Data reading method and device, electronic equipment and storage medium | |
| CN116226251A (en) | Data export method and device, electronic equipment and storage medium | |
| CN116579914B (en) | Execution method and device of graphic processor engine, electronic equipment and storage medium | |
| CN113326890B (en) | Labeling data processing method, related device and computer program product | |
| CN116107763B (en) | Data transmission method, device, equipment and storage medium | |
| CN118567870B (en) | Batch data processing method, device, equipment and storage medium | |
| CN117931082A (en) | Data processing method and device based on data temperature, electronic equipment and storage medium | |
| CN118519574A (en) | Data processing method, device, equipment and medium | |
| CN116595110A (en) | Data storage method and device, electronic equipment and storage medium | |
| CN116126249A (en) | Data reading method and device, electronic equipment and storage medium | |
| CN117610512A (en) | Method, device and equipment for removing header and footer and storage medium | |
| CN116860826A (en) | Data processing method, device, equipment and medium based on time sequence database | |
| CN117687992A (en) | Data migration method, device, equipment and storage medium | |
| CN117573267A (en) | Application program data display method, system, electronic equipment and storage medium | |
| CN115599828A (en) | Information processing method, device, equipment and storage medium | |
| CN117453628A (en) | Application file management method, device equipment and storage medium | |
| CN115421665A (en) | Data storage method, device, equipment and storage medium | |
| CN115017145A (en) | Data expansion method, device and storage medium | |
| CN115617811A (en) | Data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |