CN102929789A - Record organizational method and record organizational structure - Google Patents
Record organizational method and record organizational structure Download PDFInfo
- Publication number
- CN102929789A CN102929789A CN2012103572253A CN201210357225A CN102929789A CN 102929789 A CN102929789 A CN 102929789A CN 2012103572253 A CN2012103572253 A CN 2012103572253A CN 201210357225 A CN201210357225 A CN 201210357225A CN 102929789 A CN102929789 A CN 102929789A
- Authority
- CN
- China
- Prior art keywords
- record
- data file
- file
- offsets
- records
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a record organizational method and a record organizational structure, wherein the record organizational method comprises the steps that an unique identifier is assigned for each type of records; a data file and a metadata file corresponding to each type of records are created in the corresponding disk file, and the data file of each type of records corresponds to one disk file; and only continuous re-appending write operation can be carried out to the data files. Through the technical scheme in the invention, the performance of a metadata server under high pressure can be improved, and the extendibility is good.
Description
Technical field
The present invention relates to the storage system of data server, more specifically, relate to record organization method and the record organization structure of record.
Background technology
In parallel memory system, often need some auxiliary records of record or index.For example when after the disk failures for the data above it are carried out fast quick-recovery, need all have which object on this disk of record, this just need to add record, delete corresponding record when deleting when document creation.In addition, because data server is many copies, so also need to record inconsistent object, deletion corresponding record after reparation is finished.In order to improve the metadata performance, asynchronous system has been adopted in deletion, that is, be the acknowledged client end after meta data server is put deleted marker, and the removing of real inode (index node) and data is by the garbage reclamation thread process.In addition, which file is the quota function also need to record, and the operations such as the owner revises have occured, and send the relevant quota information of message modification by background thread to data server.Lose and then cause part operation not finish in order to prevent outage record, these records need to be recorded to the bottom disk file, simultaneously in order further to improve reliability, these records also need to utilize the log mechanism log (also just to reach the principal and subordinate standby mutually for these records like this, has equal reliability with metadata) of metadata.
In order to take full advantage of the existing mechanism of metadata, existing implementation is every kind of corresponding catalogue inode of record, wherein every record manages as the dentry item of inode, so only need odd word seldom just can add record, when protocol failure, also can directly utilize dentry entry deletion interface to delete.
Yet there is following problem in this implementation:
1) is unfavorable for expansion: because the metadata of record is the inode of file system, and the reserved field of inode is limited, will be difficult to add when the record metadata attribute that need to increase exceeds restriction (the most important structure inode that revises as file system metadata for supplementary is not too desirable);
2) wasting space: because each record carries out record as a dentry item, so certainly can inherit all properties of dentry item, but be not all properties all be that this record is necessary;
3) also be the most serious defective of this mode, this implementation can affect the application performance of daily record, when the metadata pressure ratio is larger, can become the bottleneck of system, affect overall performance: the target that metadata profile hash pursues is that the dentry item is as far as possible even in each hash bucket, and the record amount of these auxiliary records is very large (because the file of each ParaStor file system can be used a plurality of disks), when expansion hash is larger, the daily record application can cause magnetic head to come flyback retrace, and then greatly reduce application efficiency, will temporarily refuse to a certain extent the metadata daily record when daily record pressure heap and submit request to, and then affect the performance of whole metadata.
For the problem in the correlation technique, effective solution is proposed not yet at present.
Summary of the invention
For the problem in the correlation technique, the present invention proposes a kind of record organization method and record organization structure, can improve the performance of meta data server and can well expand.
According to an aspect of the present invention, provide a kind of record organization method, having comprised: for every kind of record distributes a unique identification; Set up in the disk file of correspondence and every kind of data file and meta data file that record is corresponding, wherein, the data file of every kind of record is corresponding to a disk file; And make data file only append continuously write operation.
Preferably, metadata information in meta data file take unique identification as index.
Preferably, when a protocol failure in the data file, insert corresponding with an invalid record record that offsets at the end of data file.
Preferably, when needs use record, the data file is recycled, invalid record is offseted with offseting to record.
More preferably, utilize the recovery function of registered in advance to recycle.
More preferably, sequential scanning data file recycles.
More preferably, recycling comprises: all are offseted record add a ltsh chain table; The sequential scanning data file for each effective record, is searched in ltsh chain table and is offseted record; If find the corresponding record that offsets, then from ltsh chain table, extract this and offset record; If do not find the corresponding record that offsets, then this effectively recorded and write a new temporary file.
Preferably, after having scanned all effective records, replace data file with new temporary file.
Preferably, when generating daily record, only record logic log.
According to a further aspect in the invention, a kind of record organization structure is provided, this record organization structure comprises data file and the meta data file corresponding to a kind of record, wherein, the data file of distributing a unique identification and making every kind of record for each record is corresponding to a disk file, and the data file is only appended write operation continuously.
The present invention organizes mode by this simple metadata, eliminated the index demand of data file, realize record deletion by appending to write to offset to record, to be reduced to the access of objfile file and append write sequence and read, improved the disk read-write performance, and the unascertainable problem of some information when solving the concurrent daily record that brings of thread and generate with logic log.
Description of drawings
In order to be illustrated more clearly in the embodiment of the invention or technical scheme of the prior art, the below will do to introduce simply to the accompanying drawing of required use among the embodiment, apparently, accompanying drawing in the following describes only is some embodiments of the present invention, for those of ordinary skills, under the prerequisite of not paying creative work, can also obtain according to these accompanying drawings other accompanying drawing.
Fig. 1 is the process flow diagram according to record organization method of the present invention;
Fig. 2 is the schematic diagram according to the data organizational structure of the embodiment of the invention; And
Fig. 3 is the record organization mode and the schematic diagram that offsets the result that illustrates according to the embodiment of the invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the invention, the technical scheme in the embodiment of the invention is clearly and completely described, obviously, described embodiment only is the present invention's part embodiment, rather than whole embodiment.Based on the embodiment among the present invention, the every other embodiment that those of ordinary skills obtain belongs to the scope of protection of the invention.
For problems of the prior art, the present inventor has proposed a kind of simple record organization mode.Be described in detail referring to Fig. 1 to Fig. 3.
Fig. 1 is the process flow diagram according to record organization method of the present invention.
With reference to Fig. 1, record organization method according to the present invention comprises: S102, for every kind of record distributes a unique identification; S104 sets up in the disk file of correspondence and every kind of data file and meta data file that record is corresponding, and wherein, the data file of every kind of record is corresponding to a disk file; And S106, make data file only append continuously write operation.
Fig. 2 is the schematic diagram according to the data organizational structure of the embodiment of the invention.
With reference to Fig. 2, according to data organizational structure of the present invention be: (we are referred to as fid for every kind of record distributes a unique identification, since 1), every kind of corresponding disk file of record is used for depositing physical record, and (file is named with fid, we are called objfile file (data file)), a structure is used for depositing associated metadata information (such as current record number etc.), metadata information leave in 0 the name file in (being called the objfile meta data file), metadata information in the objfile meta data file take fid as index.
Only show two kinds of data files corresponding to record of " the asynchronous deletion of inode: 1 " and " inconsistent object: 2 " among Fig. 2, but it should be appreciated by those skilled in the art that to have other multiple records and corresponding data file thereof.
In order to simplify processing, when one in data file record A is invalid, can immediately it not deleted from the objfile file, offset record A ' but insert one in end of file.
Difficult point here is that to write disk file be to carry out according to filename and the side-play amount of log recording to log system, it might be concurrent that the objfile record produces thread, and each generation thread may once produce a plurality of objfile file records, hereof side-play amount of record is that uncertain (the objfile lock can not be added to whole daily record things life cycle when daily record generates, because being critical path, this can reduce the metadata performance, so the daily record that generates is first not necessarily submitted to first).For this problem, the present inventor has proposed a kind of solution, namely only records logic log when daily record generates, and logic log expansion thread is unique (log system is run into logic log meeting calling logic daily record call back function and launched).
When needs use record, at first the objfile file is reclaimed, A and A ' are offseted, only stay effective record for.
In order to improve extensibility, objfile supports to reclaim function registration, and all kinds of records can specifically reclaim function according to the specific needs registered in advance, uses the recovery function of registration when recycling.
In addition, also can use the acquiescence mode to reclaim.The acquiescence way of recycling is sequential scanning objfile file, to offset record and add a hash chained list, sequential scanning file again after scanning is finished, effective record is only paid close attention in this scanning, for each effective record, at first search at the hash chained list and offset record, extract from the hash chained list if find then will offset record, otherwise record is write a new temporary file, substitute the objfile file with temporary file after this scanning is finished.Objfile file record organizational form and to offset the result shown in Figure 3.
By adopting above-mentioned organizational form, can be achieved as follows advantage:
1) owing to only having continuous appending to write to the objfile file, and the record organization mode is simple, can not cause owing to the adding of new record the expansion of hash bucket, and then resettlement has write record.So greatly improved the daily record application efficiency, improved the performance of meta data server under the large pressure.
2) do not have useless attribute to write disk file, compare with original mode and saved the space, objfile metadata structure and interrecord structure are established for this purpose specially, so there is not redundant information.
3) have good extendability, not only be embodied in the objfile dependency structure member interpolation/deletion, and when having new auxiliary record type to produce, can use very easily this framework to expand, and can reach particular requirement by registration function.
In sum, by means of technique scheme of the present invention, organize mode by proposing simple objfile metadata, for every kind of record distributes specific fid, as the index of associated metadata in the objfile meta data file, eliminate the index demand of objfile file record with fid, realized record deletion by appending to write to offset to record, thereby will be reduced to the access of objfile file and append write sequence and read, improve the disk read-write performance.The unascertainable problem of some information when in addition, solving the concurrent daily record generation that brings of thread with logic log.
The above only is preferred embodiment of the present invention, and is in order to limit the present invention, within the spirit and principles in the present invention not all, any modification of doing, is equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. a record organization method is characterized in that, described method comprises:
For every kind of record distributes a unique identification;
Set up in the disk file of correspondence and every kind of data file and meta data file that record is corresponding, wherein, the data file of every kind of record is corresponding to a disk file; And
Make described data file only append continuously write operation.
2. method according to claim 1 is characterized in that, metadata information in described meta data file take described unique identification as index.
3. method according to claim 1 is characterized in that, when a protocol failure in the described data file, inserts corresponding with an invalid record record that offsets at the end of described data file.
4. method according to claim 3 is characterized in that, when needs use record, described data file is recycled, and described invalid record and the described record that offsets are offseted.
5. method according to claim 4 is characterized in that, utilizes the recovery function of registered in advance to carry out described recycling.
6. method according to claim 4 is characterized in that, the described data file of sequential scanning is carried out described recycling.
7. method according to claim 6 is characterized in that, described recycling comprises:
All are offseted record add a ltsh chain table;
The described data file of sequential scanning for each effective record, is searched in described ltsh chain table and is offseted record;
If find the corresponding record that offsets, then from described ltsh chain table, extract this and offset record;
If do not find the corresponding record that offsets, then this effectively recorded and write a new temporary file.
8. method according to claim 7 is characterized in that, after having scanned all effective records, replaces described data file with described new temporary file.
9. each described method in 8 according to claim 1 is characterized in that, only records logic log when generating daily record.
10. record organization structure, it is characterized in that, described record organization structure comprises data file and the meta data file corresponding to a kind of record, wherein, the data file of distributing a unique identification and making every kind of record for each record is corresponding to a disk file, and described data file is only appended write operation continuously.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210357225.3A CN102929789B (en) | 2012-09-21 | 2012-09-21 | Record organization method and record organization structure |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210357225.3A CN102929789B (en) | 2012-09-21 | 2012-09-21 | Record organization method and record organization structure |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102929789A true CN102929789A (en) | 2013-02-13 |
| CN102929789B CN102929789B (en) | 2016-06-08 |
Family
ID=47644592
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210357225.3A Active CN102929789B (en) | 2012-09-21 | 2012-09-21 | Record organization method and record organization structure |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102929789B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104615652A (en) * | 2014-12-29 | 2015-05-13 | 曙光云计算技术有限公司 | Organization method for auxiliary records |
| CN107704462A (en) * | 2016-08-08 | 2018-02-16 | 阿里巴巴集团控股有限公司 | Metadata maintaining method, equipment and the storage device of resource |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1521682A (en) * | 2003-02-10 | 2004-08-18 | 但汉雷 | Electronic medical record computer-management system and medical record information processing method |
| CN1670708A (en) * | 2004-03-17 | 2005-09-21 | 联想(北京)有限公司 | Management method for computer log |
| CN1936864A (en) * | 2005-09-22 | 2007-03-28 | 康佳集团股份有限公司 | Non-fixed-length record data recognition method |
| CN101295310A (en) * | 2008-06-12 | 2008-10-29 | 上海交通大学 | Method for storing data and metadata on magnetic disk |
| EP2053608A2 (en) * | 2001-11-29 | 2009-04-29 | Sharp Kabushiki Kaisha | Data recording method, data deletion method, data display method, recording apparatus, recording medium and program |
| CN102195781A (en) * | 2011-05-30 | 2011-09-21 | 武汉理工大学 | Electronic evidence obtaining system based on electronic record correlated signature |
-
2012
- 2012-09-21 CN CN201210357225.3A patent/CN102929789B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2053608A2 (en) * | 2001-11-29 | 2009-04-29 | Sharp Kabushiki Kaisha | Data recording method, data deletion method, data display method, recording apparatus, recording medium and program |
| CN1521682A (en) * | 2003-02-10 | 2004-08-18 | 但汉雷 | Electronic medical record computer-management system and medical record information processing method |
| CN1670708A (en) * | 2004-03-17 | 2005-09-21 | 联想(北京)有限公司 | Management method for computer log |
| CN1936864A (en) * | 2005-09-22 | 2007-03-28 | 康佳集团股份有限公司 | Non-fixed-length record data recognition method |
| CN101295310A (en) * | 2008-06-12 | 2008-10-29 | 上海交通大学 | Method for storing data and metadata on magnetic disk |
| CN102195781A (en) * | 2011-05-30 | 2011-09-21 | 武汉理工大学 | Electronic evidence obtaining system based on electronic record correlated signature |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104615652A (en) * | 2014-12-29 | 2015-05-13 | 曙光云计算技术有限公司 | Organization method for auxiliary records |
| CN104615652B (en) * | 2014-12-29 | 2018-05-29 | 曙光云计算集团有限公司 | A kind of method for organizing of auxiliary record |
| CN107704462A (en) * | 2016-08-08 | 2018-02-16 | 阿里巴巴集团控股有限公司 | Metadata maintaining method, equipment and the storage device of resource |
| CN107704462B (en) * | 2016-08-08 | 2021-07-06 | 阿里巴巴集团控股有限公司 | Resource metadata maintenance method, equipment and storage device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102929789B (en) | 2016-06-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102110146B (en) | Key-value storage-based distributed file system metadata management method | |
| CN104850598B (en) | A kind of real-time data base back-up restoring method | |
| CN101404567B (en) | For accessing method and the computer system of Backup Data | |
| CN102012933B (en) | Distributed file system and method for storing data and providing services by utilizing same | |
| CN102779180B (en) | The operation processing method of data-storage system, data-storage system | |
| US8606759B2 (en) | Apparatus, system and method for recovering meta data using fragmentary information | |
| KR100946986B1 (en) | File storage system and method for managing duplicated files in the file storage system | |
| CN104239443B (en) | A kind of storage method of serialized data operation log | |
| CN103473239A (en) | Method and device for updating data of non relational database | |
| CN104866435A (en) | Continuous data protection method | |
| CN103617277A (en) | Method for restoring data table content deleted mistakenly | |
| CN101577735A (en) | Method, device and system for taking over fault metadata server | |
| CN104239438A (en) | File information storage method and file information read-write method based on separate storage | |
| CN107229540A (en) | A kind of database restoring method and system based on time point | |
| CN102541982B (en) | Method for organizing and accessing metadata file log | |
| CN101408882B (en) | Method and system for searching authorization document | |
| CN107273449B (en) | Breakpoint processing method and system based on memory database | |
| CN104516945A (en) | Hadoop distributed file system metadata storage method based on relational data base | |
| CN104636218A (en) | Data recovery method and device | |
| CN104484402B (en) | A kind of method and device of deleting duplicated data | |
| CN102929789A (en) | Record organizational method and record organizational structure | |
| CN107291579A (en) | A kind of metadata cluster File Snapshot method based on static subtree subregion | |
| CN113821476B (en) | Data processing method and device | |
| CN106126555A (en) | A kind of file management method and file system | |
| CN103177026A (en) | Data management method and data management system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220727 Address after: 100089 building 36, courtyard 8, Dongbeiwang West Road, Haidian District, Beijing Patentee after: Dawning Information Industry (Beijing) Co.,Ltd. Patentee after: DAWNING INFORMATION INDUSTRY Co.,Ltd. Address before: 100193 No. 36 Building, No. 8 Hospital, Wangxi Road, Haidian District, Beijing Patentee before: Dawning Information Industry (Beijing) Co.,Ltd. |
|
| TR01 | Transfer of patent right |