CN113900990A - File fragment storage method, device, equipment and storage medium - Google Patents
File fragment storage method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113900990A CN113900990A CN202111181263.3A CN202111181263A CN113900990A CN 113900990 A CN113900990 A CN 113900990A CN 202111181263 A CN202111181263 A CN 202111181263A CN 113900990 A CN113900990 A CN 113900990A
- Authority
- CN
- China
- Prior art keywords
- file
- storage
- fragment
- information
- fragmented
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000012634 fragment Substances 0.000 title claims abstract description 179
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000013467 fragmentation Methods 0.000 claims description 45
- 238000006062 fragmentation reaction Methods 0.000 claims description 45
- 238000012217 deletion Methods 0.000 claims description 16
- 230000037430 deletion Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 6
- 238000007906 compression Methods 0.000 claims description 4
- 230000006835 compression Effects 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 238000013500 data storage Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a file fragment storage method, a device, equipment and a storage medium. The method comprises the following steps: the method comprises the steps that a created file created by a user is fragmented through a client to obtain a fragmented file, file storage information is obtained, and the fragmented file and the file storage information are sent to a gateway storage server; processing the received fragment files based on the file storage information through a gateway storage server, and determining a linked list relationship among the fragment files; sending the linked list relation and the file storage information to a database for storage through a gateway storage server; and sending each processed fragment file to a network hard disk for storage through a gateway storage server. By the technical scheme, the file fragments can be stored in the network disk, so that the whole file does not need to be downloaded when the fragment file needs to be modified, and file transmission resources are saved.
Description
Technical Field
The embodiment of the invention relates to the technical field of information interaction, in particular to a file fragment storage method, device, equipment and storage medium.
Background
The network disk is an object-oriented storage method, a network disk user can access files at any time and any place through the Internet, and the uploading size of a single file of the network disk is unlimited and has the characteristics of openness, low space and the like.
The file storage by adopting the network disk in the prior art has the following problems: if a user wants to read the data of the network disk, the whole file must be downloaded, and partial data cannot be flexibly read according to the user requirements; if a user wants to modify the data of the network disk, the user can only upload the modified complete file again to cover the old data, which causes waste of file uploading and downloading resources, and especially for large files, the problem is particularly prominent.
Disclosure of Invention
Embodiments of the present invention provide a file fragment storage method, apparatus, device, and storage medium, so as to implement storage of a file fragment in a network disk, so that when a fragment file needs to be modified, the entire file does not need to be downloaded, and file transmission resources are saved.
In a first aspect, an embodiment of the present invention provides a file fragment storage method, including:
the method comprises the steps that a created file created by a user is fragmented through a client to obtain a fragmented file, file storage information is obtained, and the fragmented file and the file storage information are sent to a gateway storage server;
processing the received fragment files based on the file storage information through a gateway storage server, and determining a linked list relationship among the fragment files;
storing the linked list relationship and the file storage information through the gateway storage server; and sending each processed fragment file to a network hard disk for storage through the gateway storage server.
In a second aspect, an embodiment of the present invention further provides a file fragment storage device, where the device includes:
the system comprises a fragmentation module, a gateway storage server and a server, wherein the fragmentation module is used for fragmenting a created file created by a user through a client to obtain a fragmentation file, acquiring file storage information and sending the fragmentation file and the file storage information to the gateway storage server;
the processing module is used for processing the received fragment files based on the file storage information through the gateway storage server and determining the linked list relationship among the fragment files;
the information storage module is used for storing the linked list relation and the file storage information through the gateway storage server;
and the file storage module is used for sending the processed fragment files to a network hard disk for storage through the gateway storage server.
In a third aspect, an embodiment of the present invention further provides a terminal device, including one or more processors;
a memory for storing one or more programs;
the system comprises a client, a gateway storage server and a server, wherein the client is used for fragmenting a created file created by a user to obtain a fragmented file, acquiring file storage information and sending the fragmented file and the file storage information to the gateway storage server;
the gateway storage server is used for processing the received fragment files based on the file storage information and determining the linked list relationship among the fragment files; storing the linked list relationship and the file storage information and sending each processed fragment file to a network hard disk for storage;
when the one or more programs are executed by the one or more processors, the one or more processors implement the file fragment storage method according to any of the embodiments of the present invention.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the file fragment storage method according to any of the embodiments of the present invention.
The embodiment of the invention fragments a created file created by a user through a client to obtain a fragmented file, acquires file storage information, and sends the fragmented file and the file storage information to a gateway storage server; processing each received fragment file based on file storage information through a gateway storage server, and determining a linked list relationship among the fragment files; storing the linked list relationship and the file storage information through a gateway storage server; the processed fragment files are sent to the network hard disk for storage through the gateway storage server, the problem that if a user wants to modify the data of the network disk, the modified complete files can only be uploaded again to cover old data, so that file uploading and downloading resources are wasted is solved, the whole files do not need to be downloaded when the fragment files are modified, and file transmission resources are saved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a flowchart of a file fragment storage method according to a first embodiment of the present invention;
FIG. 2A is a schematic diagram of a deployment of a client and a gateway storage server;
FIG. 2B is a schematic diagram of another client and gateway storage server deployment;
fig. 2C is a flowchart of a file fragment storage method according to a second embodiment of the present invention;
fig. 2D is a schematic diagram of a file fragment storage manner in the second embodiment of the present invention;
FIG. 2E is a diagram illustrating a linked list relationship in the second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a file fragment storage device in a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a terminal device in the fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
Example one
Fig. 1 is a flowchart of a file fragment storage method according to an embodiment of the present invention, where this embodiment is applicable to a case where file fragments created by a user are stored in a network hard disk, and the method may be executed by a file fragment storage device in an embodiment of the present invention, where the device may be implemented in a software and/or hardware manner, as shown in fig. 1, the method specifically includes the following steps:
s110, fragmenting a created file created by a user through a client to obtain a fragmented file, acquiring file storage information, and sending the fragmented file and the file storage information to a gateway storage server.
The client and the gateway storage server are installed in the terminal device, the client provides a service for storing files in the network hard disk for a user, and the gateway storage server processes the fragmented files based on the file storage information. The client may obtain the created file created by the user in the form of a web page or software, which is not limited in this embodiment of the present invention.
It should be noted that there are two deployment manners of the client and the gateway storage server: one is that for a single client, as shown in fig. 2A, the client and the gateway storage server may be deployed in the same terminal device; for the other, for clients of multiple different terminals, as shown in fig. 2B, the gateway storage service may deploy a single-node server as a distributed cluster server to provide services to different clients; the single-node gateway storage server or the distributed cluster gateway storage server may provide an application API interface for the client to call the gateway storage service, and may further interact with the network hard disk through an API provided by the network hard disk to store a file, for example, a standard RESETful API may be provided by using a flash component.
Specifically, a created file created by a user and file storage information are acquired through a client, and the file storage information may include storage information of the created file, such as whether compression and encryption are required; information of the file itself, such as a file name, a file creation path, and a file attribute, may also be included. The client fragments the created file created by the user to obtain a fragment file, and sends the fragment file and the file storage information to the gateway storage server through an API (application program interface) of the gateway storage server.
Optionally, the client may further calculate a hash value of each fragmented file, add the hash value of each fragmented file to the file storage information, and send the file storage information to the gateway storage server, so that the gateway storage server performs deduplication on the fragmented files based on the hash values of the fragmented files.
For example, the method for slicing the created file created by the user to obtain the sliced file may be:
before the file created by the user and the file storage information are obtained through the client, the method further comprises the following steps: the user inputs account authentication information of the network hard disk in the terminal equipment, the client initiates a file storage request of which the header file comprises the account authentication information, and the file storage request is sent to the gateway storage service through the API interface. Because the account authentication information of the user is managed by the client, the gateway storage service does not need to record and manage the authentication information of the user, and the account security is further ensured.
And S120, processing the received fragment files through the gateway storage server based on the file storage information, and determining the linked list relationship among the fragment files.
The linked list relationship is used to indicate a storage relationship between the fragmented files, and a storage relationship between each fragmented file and the associated created file, for example, a link relationship between the identification codes of each fragmented file, and a link relationship between the identification code of each fragmented file and the identification code of the associated created file. The file identification code can be used as a unique identification code of the created file and is used for distinguishing each created file, and the identification code of the created file cannot be changed; the fragment identification code can be used as a unique identification code of the fragment file to distinguish each fragment file, and the file identification code can be changed only when the content of the fragment file is changed.
The gateway storage server comprises a metadata function and a data storage function, wherein the metadata function is used for determining file identification information or storing file storage information; the data storage function is used for processing the fragmented files based on the file storage information.
Specifically, after receiving the fragmented files and the file storage information sent by the client, the gateway storage server processes the fragmented files based on the file storage information to determine the fragment identification information of each fragmented file.
For example, the way of processing each received fragmented file based on the file storage information may be: carrying out duplicate removal on each received fragment file based on the hash value of the fragment file; and processing the de-duplicated fragmented files based on the file processing information.
S130, storing the linked list relation and the file storage information through the gateway storage server.
Specifically, the linked list relationship and the file storage information are both stored in the gateway storage server, and when a user accesses the fragmented file stored in the network hard disk, the fragmented file can be determined based on the linked list relationship and the file storage information stored in the gateway storage server, and the fragmented file can be acquired from the network hard disk.
And S140, sending the processed fragment files to a network hard disk for storage through a gateway storage server.
The network hard disk is an information data storage space in which a user can log in a website based on the internet to perform operations such as uploading, downloading and sharing of information data, and may be a cloud disk or a network disk, for example.
Specifically, the gateway storage server sends the processed fragmented files to the network hard disk, so that the network hard disk stores the processed fragmented files according to preset storage rules. The preset storage rule may be set by a user according to a requirement, and for example, the preset storage rule may be stored according to operation time, file size, file type, or file name, which is not limited in this embodiment of the present invention.
According to the technical scheme of the embodiment, a created file created by a user is fragmented through a client to obtain a fragmented file, file storage information is obtained, and the fragmented file and the file storage information are sent to a gateway storage server; processing each received fragment file based on file storage information through a gateway storage server, and determining a linked list relationship among the fragment files; storing the linked list relationship and the file storage information through a gateway storage server; the processed fragment files are sent to the network hard disk for storage through the gateway storage server, and the file fragments can be stored in the network disk, so that the whole file does not need to be downloaded when the fragment files need to be modified, and file transmission resources are saved.
Optionally, the method further includes:
and setting a user space file system in the client, and accessing the gateway storage server through the user space file system.
Specifically, the client can use the storage directory in the user space of the operating system through a user space file system (FUSE), the method is cross-system platform, for example, the method can be used in a system platform such as linux/mac, and the FUSE performs data interaction with a gateway storage server through network connection modes such as http protocol, so that the gateway storage service can perform data interaction with the fragmented files in the network disk according to the file storage information.
Example two
Fig. 2C is a flowchart of a file fragment storage method in the second embodiment of the present invention, which is optimized based on the foregoing embodiment in this embodiment, and in this embodiment, a fragment file is obtained by fragmenting a created file created by a user, where the fragmenting file includes: acquiring the byte number of the created file; calculating a first ratio of the byte number of the created file to the byte number of the fragments, and determining the result of rounding the first ratio upwards as the fragment number of the created file; and fragmenting the created file based on the number of the fragmentation bytes to obtain the fragmentation file of the fragmentation number.
As shown in fig. 2C, the method of this embodiment specifically includes the following steps:
s210, acquiring the created file created by the user and the byte number of the created file through the client.
Specifically, a created file created by a user is obtained through a client, and the size of the created file, that is, the number of bytes of the created file, is identified through a preset program in the client.
It should be noted that, the fragmented file after the client fragment passes information to the gateway storage service in the form of API, and the actual file transmission is in units of bytes, but for convenience of representation, the size of the created file may also be in units of MiB, where 1MiB is 1024KiB is 1024 × 1024 Byte.
S220, fragmenting the created file based on a preset number of fragment bytes to obtain fragmented files of the number of fragments; the number of fragments is the result of rounding up the first ratio of the number of bytes of the created file and the number of bytes of the preset fragments.
The preset number of the slice bytes can be set by the client based on the user requirement, and can also be customized by the user.
Specifically, the number of fragments for creating a file is calculated as follows:
wherein N is the number of the fragments, BfTo create the number of bytes of the file, BsIs the number of bytes of the sliced file,
the rounding-up operator.
It can be understood that the client segments the created file based on a preset number of segment bytes to obtain a segmented file of the number of segments, and if the number of bytes of the created file and the number of bytes of the segmented file are divisible, the number of bytes of data stored in each segmented file is the preset number of segment bytes; for example, if the number of bytes of the created file is 10240KiB (10MiB) and the number of bytes of the created fragment is 2048KiB (2MiB), the file may be fragmented into 5 fragmented files, each of which is 2048 KiB.
If the byte number of the created file and the byte number of the sliced file cannot be divided, the byte number of the data which can be stored in the sliced file with the previous slicing number reduced by one is the preset slicing byte number, the byte number of the last sliced file is still the preset slicing byte number, but the byte number of the stored data is the remainder of the byte number of the created file and the byte number of the sliced file, and the rest byte numbers are filled by null bytes. For example, if the byte number of the created file is 10300KiB, and the preset byte number of the fragmentation is 2048KiB, the file may be fragmented into 6 fragmentation files, 2048KiB of each fragmentation file, the first 60KiB of the last fragmentation file is the file data, and the last 1988KiB is a null byte.
And S230, acquiring file storage information, and sending the fragment file and the file storage information to a gateway storage server.
Optionally, the file storage information includes: creating file information and file processing information; the creating file information includes: creating a file name, a path, a file attribute and a fragment file hash value of a file; the file processing information includes: slice file compression information and/or slice file encryption information.
The file name, the creating path, the creating file attribute and the file processing information of the creating file can be acquired by inputting from a client by a user; the fragmentation file hash value can be obtained by calculating each fragmentation file based on a hash algorithm preset in the client, and is a unique and extremely compact numerical representation form of the fragmentation file. The hash value calculation method of the fragmented file may be, for example, an MD5 message digest algorithm, which is not limited in this embodiment of the present invention.
And S240, processing each received fragment file through the gateway storage server based on the file storage information.
In a preferred embodiment, S240 may include: s241 to S242 are as follows:
s241, performing duplicate removal on each received fragmented file based on the fragmented file hash value.
Exemplarily, due to reasons such as network connection interruption or poor network connection state, the process of storing the file into the network hard disk is interrupted, so that the file uploading fails but part of the fragmented file is uploaded successfully; when the current user uploads the file again, the situation of repeatedly uploading the fragment file exists. In order to reduce repeated uploading of the fragmented files, it is necessary to perform deduplication on each received fragmented file. The method comprises the following specific steps: and comparing the hash value of the received fragment file corresponding to the current fragment file with the hash values of the fragment files corresponding to the fragment files stored in the network hard disk, and if two identical hash values of the fragment files exist, indicating that the fragment file identical to the current fragment file is stored in the network hard disk based on the uniqueness of the hash values of the fragment files, wherein the current fragment file is a repeatedly stored file, so that the current fragment file is abandoned, and the duplication elimination of the received fragment files is realized.
And S242, processing the duplicated fragmented files based on the file processing information.
Specifically, the way of processing the deduplicated sliced file based on the file processing information may be: and if the file storage information indicates that the fragmented files need to be compressed, the gateway storage server compresses the fragmented files, and if the file storage information indicates that the fragmented files need to be encrypted, the gateway storage server encrypts the fragmented files.
For example, data compression processing can be performed on the deduplicated fragmented files based on a Huffman algorithm, and the compressed Huffman has an encryption function, the compressed fragmented files are stored in a network hard disk, and the Huffman tree is stored in a database of the gateway storage server.
And S250, determining the linked list relation among all the fragment files.
Optionally, determining a linked list relationship between the fragmented files includes:
distributing fragment identification codes for all fragment files, and distributing file identification codes for the created files;
determining the sequence of each fragment file in the created file as the fragment serial number of each fragment file;
determining a linked list relationship between the fragmented files according to the fragmented serial numbers of the fragmented files, wherein the linked list relationship comprises: the file identifier comprises a current fragment identification code, a previous fragment identification code and a next fragment identification code of each fragment file, and a file identifier of a created file to which each fragment file belongs.
The identification code of the fragmented file is only related to the hash value of the fragmented file, and when the data of the fragmented file changes and the hash value of the fragmented file also changes, a new identification code needs to be allocated to the fragmented file with changed data. The identifier codes of the fragmented files may be randomly allocated or sequentially allocated in sequence by the gateway storage service, and for convenience, the fragmented identifier codes are typically allocated sequentially from 1 according to the generation order of the fragmented files. The linked list relationship may be a linked list formed by linked list elements of each fragmented file.
Specifically, the fragmentation sequence of the fragmentation file in the created file during fragmentation is determined as the fragmentation sequence number of the fragmentation file, and the relation between the fragmentation files, namely the previous fragmentation and the next fragmentation of each fragmentation file, can be determined based on the fragmentation sequence number, so as to determine the linked list relation formed by the linked list elements of each fragmentation file. The linked list elements of each shard file may include: the file identification code comprises a current fragment identification code, a previous fragment identification code, a next fragment identification code and a file identification code of an affiliated created file. It can be understood that the first fragment file is the first fragment file, and the linked list element of the first fragment file in the linked list relationship does not contain the previous fragment identification code; the linked list element of the last fragment file does not contain the following fragment identification code.
For example, if the byte number of the created file is 10240KiB and the preset number of the fragmentation byte is 2048KiB, the created file may be fragmented into 5 fragmentation files, each of which is 2048 KiB. The linked list relationship between the fragmented files determined according to the fragmented serial numbers of the fragmented files is shown in fig. 2D. The fragment identification codes of the fragment files are P1, P2, P3, P4 and P5 in sequence, the identification code of the created file to which each fragment file belongs is D1, wherein P represents the fragment identification, and D represents the file identification code.
For example, as shown in fig. 2E, the link list element corresponding to the second shard file in the link list relationship includes: the current fragmentation identification code P2, the previous fragmentation identification code P1, the next fragmentation identification code P3, and the file identification code D1 of the created file to which the second fragmentation file belongs.
S260, storing the linked list relation and the file storage information through the gateway storage server;
and S270, sending each processed fragment file to a network hard disk for storage through the gateway storage server.
According to the technical scheme of the embodiment, a created file created by a user is fragmented through a client to obtain a fragmented file, file storage information is obtained, and the fragmented file and the file storage information are sent to a gateway storage server; processing each received fragment file based on file storage information through a gateway storage server, and determining a linked list relationship among the fragment files; storing the linked list relationship and the file storage information through a gateway storage server; the processed fragment files are sent to the network hard disk for storage through the gateway storage server, and the file fragments can be stored in the network disk, so that the whole file does not need to be downloaded when the fragment files need to be modified, and file transmission resources are saved.
Optionally, storing the linked list relationship and the file storage information by the gateway storage server includes:
and storing the linked list relationship and the file storage information in a database cluster through the gateway storage server, wherein the database cluster is deployed in the gateway storage server.
Specifically, a database cluster may be deployed in the gateway storage server, and is used to store the linked list relationship and the file storage information. And determining the storage position of the fragmented file in the network hard disk based on the linked list relationship and the file storage information, thereby obtaining the fragmented file.
Optionally, the method further includes:
acquiring file deletion information through a client, and sending the file deletion information to a gateway storage server;
determining a file to be deleted based on the file deletion information through a gateway storage server, and deleting a linked list relation corresponding to the file to be deleted; asynchronously recovering the fragment identification codes corresponding to the fragment files in the files to be deleted;
judging whether the current identification code of the fragmented file to be deleted only exists in the linked list relation corresponding to the fragmented file to be deleted or not for the fragmented file to be deleted contained in the file to be deleted;
if yes, deleting the fragmented file to be deleted based on the file deletion information;
if not, the fragment file to be deleted is reserved.
The file deletion information is used to indicate information for deleting the created file, and the created file information may include: the file deletion method comprises the steps of storing a path of a file to be deleted, the name of the file to be deleted and a file identification code of the file to be deleted.
Specifically, file deletion information input by a user is acquired through a client, the file deletion information is sent to a gateway storage server, the gateway storage server determines files to be deleted based on storage paths and names of the files to be deleted, which are contained in the file deletion information, and deletes a linked list relation corresponding to the files to be deleted from the gateway storage server. Because the fragment identification codes of all the fragment files stored in the linked list relationship of the deleted file are discarded, the fragment identification codes corresponding to all the fragment files in the file to be deleted are asynchronously recycled, and the fragment identification codes are reused when the new fragment files are distributed. Preferably, the recycled fragment identification codes can be preferentially used.
And if so, deleting the fragmented files to be deleted based on the storage path and the identification code of the file deletion information. If not, the fragmented file to be deleted exists in other created files at the same time, and the fragmented file to be deleted needs to be reserved.
Optionally, the method further includes:
acquiring file query information input by a user through a client, and sending the file query information to a gateway storage server;
and acquiring file storage information from the network hard disk through a gateway storage server based on the file query information, and returning the file storage information to the client.
The file query information is used to indicate file storage information that needs to be queried, and the file query information may include: the storage path of the file to be queried and the name of the file to be queried.
Specifically, file query information input by a user is obtained through a client, the file query information is sent to a gateway storage server, the gateway storage server determines a file to be queried based on a storage path and a name of the file to be queried, the storage path and the name of the file are contained in the file query information, file storage information corresponding to the file to be queried is obtained from a network hard disk, and the file storage information is returned to the client.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a file fragmentation storage device according to a third embodiment of the present invention. The present embodiment may be applicable to a case where file segments created by a user are stored in a network hard disk, where the apparatus may be implemented in a software and/or hardware manner, and the apparatus may be integrated in any device that provides a function of storing file segments, as shown in fig. 3, where the apparatus for storing file segments specifically includes: a fragmentation module 310, a processing module 320, and a storage module 330.
The fragmentation module 310 fragments a created file created by a user through a client to obtain a fragmentation file, acquires file storage information, and sends the fragmentation file and the file storage information to a gateway storage server;
a processing module 320, configured to process, by the gateway storage server, each received fragmented file based on the file storage information, and determine a linked list relationship between the fragmented files;
an information storage module 330, configured to store the linked list relationship and the file storage information through the gateway storage server;
and the file storage module 340 is configured to send each processed fragmented file to a network hard disk for storage through the gateway storage server.
Optionally, the slicing module 310 is specifically configured to:
acquiring a created file created by a user and the byte number of the created file;
fragmenting the created file based on a preset number of fragment bytes to obtain fragmented files of the number of fragments; the number of fragments is the result of rounding up the first ratio of the number of bytes of the created file and the number of bytes of the preset fragments.
Optionally, the file storage information includes: creating file information and file processing information; the creating file information includes: creating a file name, a path, a file attribute and a fragment file hash value of a file; the file processing information includes: slice file compression information and/or slice file encryption information.
Optionally, the processing module 320 is specifically configured to:
carrying out duplicate removal on each received fragment file based on the hash value of the fragment file;
and processing the de-duplicated fragmented files based on the file processing information.
Optionally, the processing module 320 is further configured to:
distributing fragment identification codes for all fragment files, and distributing file identification codes for the created files;
determining the sequence of each fragment file in the created file as the fragment serial number of each fragment file;
determining a linked list relationship between the fragmented files according to the fragmented serial numbers of the fragmented files, wherein the linked list relationship comprises: the file identification code comprises a current fragment identification code, a previous fragment identification code and a next fragment identification code of each fragment file, and a file identification code of a created file to which each fragment file belongs.
Optionally, the information storage module 330 is specifically configured to:
and storing the linked list relationship and the file storage information in a database cluster through the gateway storage server, wherein the database cluster is deployed in the gateway storage server.
Optionally, the method further includes:
and the setting module is used for setting a user space file system in the client and accessing the gateway storage server through the user space file system.
Optionally, the method further includes:
the first acquisition module is used for acquiring file deletion information through a client and sending the file deletion information to a gateway storage server;
the first deleting module is used for determining a file to be deleted through the gateway storage server based on the file deleting information and deleting the linked list relation corresponding to the file to be deleted; asynchronously recovering the fragment identification codes corresponding to the fragment files in the files to be deleted;
the judging module is used for judging whether the current identification code of the fragmented file to be deleted only exists in the linked list relation corresponding to the fragmented file to be deleted or not for the fragmented file to be deleted contained in the file to be deleted;
the second deleting module is used for deleting the fragmented files to be deleted based on the file deleting information if the current identification codes of the fragmented files to be deleted only exist in the linked list relation corresponding to the fragmented files to be deleted;
and the reservation module is used for reserving the fragmented files to be deleted if the current identification codes of the fragmented files to be deleted do not exist only in the linked list relationship corresponding to the fragmented files to be deleted.
Optionally, the method further includes:
the second acquisition module is used for acquiring file query information input by a user through a client and sending the file query information to the gateway storage server;
and the query module is used for acquiring file storage information from the network hard disk through the gateway storage server based on the file query information and returning the file storage information to the client.
The product can execute the file fragment storage method provided by the first embodiment or the second embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 4 is a block diagram of a terminal device according to a fourth embodiment of the present invention, as shown in fig. 4, the terminal device includes a processor 410, a memory 420, a client 430, and a gateway storage server 440; the number of the processors 410 in the terminal device may be one or more, and one processor 410 is taken as an example in fig. 4; the processor 410, the memory 420, the client 430 and the gateway storage server 440 in the terminal device may be connected by a bus or other means, and the connection by the bus is exemplified in fig. 4.
The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 420 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 420 may further include memory located remotely from processor 410, which may be connected to a terminal device through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The client 430 is used for fragmenting a created file created by a user to obtain a fragmented file, acquiring file storage information, and sending the fragmented file and the file storage information to a gateway storage server;
the gateway storage server 440 is configured to process the received fragmented files based on the file storage information, and determine a linked list relationship between the fragmented files; storing the linked list relationship and the file storage information; and sending each processed fragment file to a network hard disk for storage.
EXAMPLE five
The fifth embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the file fragment storage method provided in the first or second embodiment of the present application: the method comprises the steps that a created file created by a user is fragmented through a client to obtain a fragmented file, file storage information is obtained, and the fragmented file and the file storage information are sent to a gateway storage server; processing the received fragment files based on the file storage information through a gateway storage server, and determining a linked list relationship among the fragment files; sending the linked list relation and the file storage information to a database for storage through a gateway storage server; and sending each processed fragment file to a network hard disk for storage through a gateway storage server.
Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (12)
1. A file fragment storage method is characterized by comprising the following steps:
the method comprises the steps that a created file created by a user is fragmented through a client to obtain a fragmented file, file storage information is obtained, and the fragmented file and the file storage information are sent to a gateway storage server;
processing the received fragment files based on the file storage information through a gateway storage server, and determining a linked list relationship among the fragment files;
storing the linked list relationship and the file storage information through the gateway storage server;
and sending each processed fragment file to a network hard disk for storage through the gateway storage server.
2. The method of claim 1, wherein slicing the created file created by the user to obtain a sliced file comprises:
acquiring a created file created by a user and the byte number of the created file;
fragmenting the created file based on a preset number of fragment bytes to obtain fragmented files of the number of fragments; the number of fragments is the result of rounding up the first ratio of the number of bytes of the created file and the number of bytes of the preset fragments.
3. The method of claim 1, wherein the file storage information comprises: creating file information and file processing information; the creating file information includes: creating a file name, a path, a file attribute and a fragment file hash value of a file; the file processing information includes: slice file compression information and/or slice file encryption information.
4. The method of claim 3, wherein processing each received fragmented file based on the file storage information comprises:
carrying out duplicate removal on each received fragment file based on the hash value of the fragment file;
and processing the de-duplicated fragmented files based on the file processing information.
5. The method of claim 1, wherein determining a linked list relationship between the sharded files comprises:
distributing fragment identification codes for all fragment files, and distributing file identification codes for the created files;
determining the fragmentation sequence of each fragmentation file in the created file as the fragmentation serial number of each fragmentation file;
determining a linked list relationship between the fragmented files according to the fragmented serial numbers of the fragmented files, wherein the linked list relationship comprises: the file identifier comprises a current fragment identification code, a previous fragment identification code and a next fragment identification code of each fragment file, and a file identifier of a created file to which each fragment file belongs.
6. The method of claim 1, wherein storing the linked list relationship and the file storage information by the gateway storage server comprises:
and storing the linked list relationship and the file storage information in a database cluster through the gateway storage server, wherein the database cluster is deployed in the gateway storage server.
7. The method of claim 1, further comprising:
and setting a user space file system in the client, and accessing the gateway storage server through the user space file system.
8. The method of claim 1, further comprising:
acquiring file deletion information through a client, and sending the file deletion information to a gateway storage server;
determining a file to be deleted based on the file deletion information through a gateway storage server, and deleting a linked list relation corresponding to the file to be deleted; asynchronously recovering the fragment identification codes corresponding to the fragment files in the files to be deleted;
judging whether the current identification code of the fragmented file to be deleted only exists in the linked list relation corresponding to the fragmented file to be deleted or not for the fragmented file to be deleted contained in the file to be deleted;
if yes, deleting the fragmented file to be deleted based on the file deletion information;
if not, the fragment file to be deleted is reserved.
9. The method of claim 1, further comprising:
acquiring file query information input by a user through a client, and sending the file query information to a gateway storage server;
and acquiring file storage information from the network hard disk through a gateway storage server based on the file query information, and returning the file storage information to the client.
10. A file shard storage device, comprising:
the system comprises a fragmentation module, a gateway storage server and a server, wherein the fragmentation module is used for fragmenting a created file created by a user through a client to obtain a fragmentation file, acquiring file storage information and sending the fragmentation file and the file storage information to the gateway storage server;
the processing module is used for processing the received fragment files based on the file storage information through the gateway storage server and determining the linked list relationship among the fragment files;
the information storage module is used for storing the linked list relation and the file storage information through the gateway storage server;
and the file storage module is used for sending the processed fragment files to a network hard disk for storage through the gateway storage server.
11. A terminal device, characterized in that the terminal device comprises:
one or more processors;
a memory for storing one or more programs;
the system comprises a client, a gateway storage server and a server, wherein the client is used for fragmenting a created file created by a user to obtain a fragmented file, acquiring file storage information and sending the fragmented file and the file storage information to the gateway storage server;
the gateway storage server is used for processing the received fragment files based on the file storage information and determining the linked list relationship among the fragment files; storing the linked list relationship and the file storage information; transmitting each processed fragment file to a network hard disk for storage;
when executed by the one or more processors, cause the one or more processors to implement the file shard storage method of any of claims 1-9.
12. A computer-readable storage medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the file fragment storage method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111181263.3A CN113900990A (en) | 2021-10-11 | 2021-10-11 | File fragment storage method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111181263.3A CN113900990A (en) | 2021-10-11 | 2021-10-11 | File fragment storage method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113900990A true CN113900990A (en) | 2022-01-07 |
Family
ID=79191393
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111181263.3A Pending CN113900990A (en) | 2021-10-11 | 2021-10-11 | File fragment storage method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113900990A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116383896A (en) * | 2023-06-07 | 2023-07-04 | 中航金网(北京)电子商务有限公司 | File integrity verification method, cloud platform starting method, device and equipment |
-
2021
- 2021-10-11 CN CN202111181263.3A patent/CN113900990A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116383896A (en) * | 2023-06-07 | 2023-07-04 | 中航金网(北京)电子商务有限公司 | File integrity verification method, cloud platform starting method, device and equipment |
| CN116383896B (en) * | 2023-06-07 | 2023-11-03 | 中航金网(北京)电子商务有限公司 | File integrity verification method, cloud platform starting method, device and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9984093B2 (en) | Technique selection in a deduplication aware client environment | |
| US9514209B2 (en) | Data processing method and data processing device | |
| US9990352B2 (en) | Chunk compression in a deduplication aware client environment | |
| US9917894B2 (en) | Accelerating transfer protocols | |
| US10459886B2 (en) | Client-side deduplication with local chunk caching | |
| CN109391673B (en) | Method, system and terminal equipment for managing update file | |
| WO2017215646A1 (en) | Data transmission method and apparatus | |
| CN106161633B (en) | Transmission method and system for packed files based on cloud computing environment | |
| US10862672B2 (en) | Witness blocks in blockchain applications | |
| CN111708743A (en) | File storage management method, file management client and file storage management system | |
| CN110572422B (en) | Data downloading method, device, equipment and medium | |
| US10339124B2 (en) | Data fingerprint strengthening | |
| WO2021237467A1 (en) | File uploading method, file downloading method and file management apparatus | |
| US20200065306A1 (en) | Bloom filter partitioning | |
| US20190377711A1 (en) | Reducing database fragmentation | |
| WO2013097812A1 (en) | Method and system for downloading font file | |
| CN111262822A (en) | File storage method, device, block link point and system | |
| US20160044077A1 (en) | Policy use in a data mover employing different channel protocols | |
| CN113900990A (en) | File fragment storage method, device, equipment and storage medium | |
| CN112969198A (en) | Data transmission method, terminal and storage medium | |
| CN111355793A (en) | Large file uploading method based on HTML5 and Python | |
| CN111245949A (en) | File filing and transmission method, device and equipment | |
| CN108897497B (en) | Centerless data management method and device | |
| US20170048303A1 (en) | On the fly statistical delta differencing engine | |
| CN109144991B (en) | Method and device for dynamic sub-metering, electronic equipment and computer-storable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100000 Unit 501A, Floor 5, 101, Building 1, Yard 5, Laiguangying West Road, Chaoyang District, Beijing Applicant after: Beijing Qingyun Science and Technology Co.,Ltd. Address before: 100020 601, 6 floor, 16 building, No. 36 Chuang Yuan Road, Chaoyang District, Beijing. Applicant before: Beijing Qingyun Science and Technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Country or region after: China Address after: 100000 Unit 501A, Floor 5, 101, Building 1, Yard 5, Laiguangying West Road, Chaoyang District, Beijing Applicant after: Beijing Qingyun Technology Group Co.,Ltd. Address before: 100000 Unit 501A, Floor 5, 101, Building 1, Yard 5, Laiguangying West Road, Chaoyang District, Beijing Applicant before: Beijing Qingyun Science and Technology Co.,Ltd. Country or region before: China |