CN101464890B - Mixed log generation method and analysis method - Google Patents
Mixed log generation method and analysis method Download PDFInfo
- Publication number
- CN101464890B CN101464890B CN2008102418990A CN200810241899A CN101464890B CN 101464890 B CN101464890 B CN 101464890B CN 2008102418990 A CN2008102418990 A CN 2008102418990A CN 200810241899 A CN200810241899 A CN 200810241899A CN 101464890 B CN101464890 B CN 101464890B
- Authority
- CN
- China
- Prior art keywords
- log
- daily record
- redo
- transaction
- statement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000004458 analytical method Methods 0.000 title claims abstract description 30
- 230000008569 process Effects 0.000 claims description 20
- 230000001360 synchronised effect Effects 0.000 claims description 20
- 210000001072 colon Anatomy 0.000 claims description 2
- 239000000976 ink Substances 0.000 claims description 2
- 238000011084 recovery Methods 0.000 description 17
- 230000015572 biosynthetic process Effects 0.000 description 13
- 238000005755 formation reaction Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000000977 initiatory effect Effects 0.000 description 3
- 230000001788 irregular Effects 0.000 description 3
- 238000002955 isolation Methods 0.000 description 3
- 230000008520 organization Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000012467 final product Substances 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 241000208340 Araliaceae Species 0.000 description 1
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 1
- 235000003140 Panax quinquefolius Nutrition 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 239000002253 acid Substances 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000000151 deposition Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002224 dissection Methods 0.000 description 1
- 235000008434 ginseng Nutrition 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004615 ingredient Substances 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a mixed log generation method and a mixed log analysis method. The mixed log generation method comprises the following steps: opening or creating a hybrid log file; receiving a user command and judging the type of the user command; and registering a corresponding command type log according to the judged type of the user command. The mixed log analysis method comprises the following steps: A. applying for a handle of a log parser, and opening a mixed log file; B. judging whether a closed REDO/UNDO transaction block node exists, if so, executing the step C, otherwise, executing the step D; C. assigning the REDO/UNDO transaction block endpoint pointer to an output variable, and returning to obtain a successful state value of the REDO/UNDO transaction block; D. and B, sequentially analyzing log lines of the mixed log file, executing the operation of generating the REDO/UNDO transaction block, and then executing the step B. The hybrid log integrates all information of the transaction log and the synchronization log, eliminating redundant information.
Description
Technical field
The present invention relates to the memory database field, relate in particular to a kind of mixing daily record generation method and analytic method of memory database.
Background technology
Modern communications is more and more stronger to the computer technology dependence; In the continuous development of telecom operators; Set up the various information system, comprise Operation Network system, comprehensive accounting system, charge system, office automation system or the like, these systems have accumulated the valuable data information of magnanimity; Through these data are carried out data mining, query analysis and on-line analysis, can discern the market opportunity, formulate market strategy.But these data informations are too huge, and the record strip number is easily in hundred million, and like note system, per second need be stored thousands of user profile.For huge data volume like this; Traditional data base management system (DBMS) based on disk more and more is difficult to deal with; So industry has proposed memory database technology and main memory DBM system (MainMemory Database Management System is hereinafter to be referred as MMDBMS).
Memory database is about to the database that data are placed on direct control in the internal memory.The main memory DBM system manages the part of entire database or database in internal memory, so direct visit data and need not visit disk.Thereby memory database has just had high performance transaction capabilities.
Yet because internal memory itself has volatibility, in case i.e. power down is perhaps broken down, data will all be lost, and cause the commercial system irretrievable loss.Therefore, MMDBMS can backup to the dirty data in the internal memory on the permanent storage in the certain hour interval.
MMDBMS supports concurrent operations, and the process of backup possibly be accompanied by the updating data process.What obtain after the backup possibly be the database image file that is in the inconsistency state.If use such image file to come restore database, will make database be absorbed in the inconsistency state.
In order to obtain an image file that is in coherency state, at present many systems adopt the mode of static backup, and promptly waiting does not have affairs when operation, to back up in the system.Do not allow during the backup database is upgraded.This backup method greatly reduces the availability of system.And backup operation can not be too frequent, otherwise the I/O of system can not bear.In addition, the time interval of backup is long more, in case system crash, the data volume of losing is also many more.At present general way is to adopt registration daily record (Logging) to ensure that data do not lose, lose less.
In addition, MMDBMS also can be absorbed in the inconsistency state in operational process.For example the system crash accident stops affairs, and user's active termination affairs etc. all possibly cause database inconsistent.Present most of system all adopts the method for registration transaction journal to come database recovery to up-to-date coherency state.
The plurality of applications system, like above-mentioned note system, often the node by a lot of band databases constitutes a distributed system, often needs synchrodata between the database of these nodes.Data synchronization process is carried out sometimes in real time, only needs regularly sometimes/an irregular synchronous secondary data.Often need not generate sync data file synchronously in real time.And for regularly/irregularly synchronously, what industry employing at present was maximum is the way that generates sync data file.
But, the transaction journal file adds sync data file, and the data volume of its generation is very huge.The researchist is through to the research of the two memory contents, finds that the two has deposited many identical or identical information, and redundance is quite big.This will cause the serious waste of storage space, also expend the computational resource when generating these redundant informations.
In addition, when carrying out system failure recovery, the common way of industry is to set up two formations at present: REDO formation and UNDO formation.Repeat REDO and UNDO operation during system failure recovery respectively, way is more loaded down with trivial details, and the multiple scanning journal file, and efficient is lower.
Summary of the invention
For existing simultaneously, the transaction journal file that solves above-mentioned Database Systems and sync data file cause waste of storage space; The inefficient problem of system failure recovery; Be necessary to provide a kind of storage space less, the mixing daily record generation method that system failure recovery efficient is higher.
In addition, also be necessary to provide a kind of mixing daily record analytic method of above-mentioned mixing journal format.
The present invention's mixing daily record generation method may further comprise the steps: open or create the hybrid log file that not only can serve as transaction journal but also can serve as synchronization log; Receive user command, and judge the type of said user command; The type of the user command that draws according to said judgement and the pattern register corresponding with the type are registered corresponding command type daily record.
The corresponding command type daily record of said registration may further comprise the steps: receive the daily record register request; Judge whether said log buffer district has enough spaces to hold said daily record; If there are not enough spaces, then write with a brush dipped in Chinese ink said hybrid log file to the data of said buffer area, and empty buffer area; Check the size that aims at said day side-play amount and said hybrid log file in the said buffer area, obtain the two addition said day to aim at the real offset in the said hybrid log file; Said daily record is write said buffer area, and return success byte number and the said real offset that writes.
The type that said judgement draws user command is the checkpoint order, then registers said checkpoint command log, and follows the execution of said checkpoint order, and the execution and the transaction concurrency of the order of said checkpoint are carried out.
The log lines beginning structure of said mixing daily record is lsn:len:sync:op_code, and said lsn representes the log lines sequence number, is used for explaining the order that the log lines order begins to carry out; Said len representes except except the shared length of said lsn and len; The length of said log lines, said sync representes whether said log lines is that needs produce synchronization log, and said op_code representes said log lines command code; Be used to discern the type of said log lines, said colon: be separator.
Said lsn is unique in the database overall situation, and said sync is the asynchronous daily record of 0 expression, is non-0 expression synchronization log, and the coding span of said op_code is 1 to 255.
The form of said mixing daily record comprises DML statement bracket journal format, DML basic operation journal format, affairs control journal format and checkpoint journal format, and the op_code of said DML statement bracket journal format, op are the DML order; Code is BEGIN, END or ABORT, and the op_code of said DML basic operation journal format is DELETE_VFIELD, INSERT_VFIELD; UPDATE_FIELD, DELETE_RECORD, INSERT_RECORD or FREE_PAGE; The lsn of said affairs control journal format is 0, and sync is 1, and op is TRANSACTION; Code is BEGIN, COMMIT or ROLLBACK, and the lsn of said checkpoint journal format is 0; Sync is 1, and op is CHECK_POINT, and code is BEGIN or END.
The present invention's mixing daily record analytic method may further comprise the steps:
Steps A, application date book resolver handle are opened hybrid log file;
Step B, judge whether to exist closed REDO/UNDO transaction block node; If, execution in step C, otherwise; Execution in step D; Said REDO/UNDO transaction block is represented the set of all daily records that affairs comprise, and said closed REDO/UNDO transaction block is initial by TRANSACTION_BEGIN, with TRANSACTION_COMMIT or TRANSACTION_ROLLBACK ending;
Step C, compose said REDO/UNDO transaction block node pointer to output variable, return and obtain REDO/UNDO transaction block success status value;
Step D, resolve the log lines of hybrid log file successively, carry out the operation that generates said REDO/UNDO transaction block, again execution in step B.
Carrying out the operation that generates said REDO/UNDO transaction block among the said step D may further comprise the steps: it is TRANSACTION_BEGIN that parsing runs into log lines; Generate REDO/UNDO transaction block node; Judge the affiliated command type of log lines more successively; Running into log lines is TRANSACTION_END, and puts said REDO/UNDO transaction block node and be closed sign.
Said REDO/UNDO transaction block comprises statement block, and said statement block begins with op_BEGIN, and closed statement block begins with op_BEGIN; The op_END ending, op is the DML order, the log lines that said step D resolves hybrid log file successively comprises: it is op_BEGIN that parsing runs into said log lines; Generated statement agllutination point; Judge command type under the log lines more successively, running into log lines is op_END, and puts said REDO/UNDO transaction block node and be closed sign.
Said step C replaces with: obtain current statement block; Generate the synchronous SQL statement of next bar according to current statement block, internal pointer is displaced to next bar be untreated log lines or next statement block node, and judges whether to generate synchronous SQL statement; If; Return and obtain synchronous SQL statement success status value, otherwise, concrete error status value returned.
Compared with prior art; The present invention's mixing daily record generation method and analytic method adopt the mixing daily record that mixes journal format to combine all information of transaction journal and synchronization log, thereby have guaranteed that the mixing daily record both can serve as transaction journal, can serve as the data sync daily record again; A lot of redundant informations have been eliminated; Make whole daily record scale descend greatly, not only saved file storage, and reduced memory cache space and the cpu resource that consumes when generating these daily records.
And the present invention's mixing daily record generation method and analytic method are supported the concurrent execution of affairs, improve the availability and the handling capacity of system greatly.And, only need be aided with hybrid log file and carry out the coherency state that affairs REDO and UNDO guarantee data.
Moreover the present invention's mixing daily record analytic method adopts scan edge journal file limit to generate the method for REDO/UNDO transaction block, a scanning is carried out in daily record just can be accomplished parsing and the processing to whole journal file, has improved resolution speed, has simplified treatment scheme.And; Integrate the construction process of REDO formation and UNDO formation; Closed when the REDO/UNDO transaction block, just carry out the REDO/UNDO operation or generate synchronous SQL statement, disposing discharges the shared memory source of this REDO/UNDO transaction block immediately.
After the detailed description that combines the advantages embodiment of the present invention, it is clearer that characteristics of the present invention and advantage will become.
Description of drawings
Fig. 1 is the schematic flow sheet of the present invention's mixing daily record generation method.
Fig. 2 is the idiographic flow synoptic diagram of Fig. 1.
Fig. 3 is the concrete realization flow figure of daily record registration among Fig. 1.
Fig. 4 is the process flow diagram that the present invention is used for the mixing daily record analytic method of system failure recovery.
Fig. 5 is the process flow diagram that the present invention is used for the synchronous mixing daily record analytic method of non-real-time data.
Embodiment
Below in conjunction with accompanying drawing the present invention's mixing daily record generation method and analytic method are described.
The present invention's mixing journal format is described with expansion BNF grammer metasymbol.
BNF grammer metasymbol is following:
[、]、|、*、+、::
Wherein, [with] be used to define the scope of syntactic element or represent optional part.| be illustrated in optional one of its right and left.* represent 0 or a plurality of.At least 1 of+expression.Square bracket represent that to independent appearance the syntactic element that brackets is optional part.Square bracket are to perhaps+together using with *, then respectively the syntactic element that brackets of expression occur 0 time to n time and at least once more than.:: the expression definition.BNF grammer metasymbol just is used for describing the instrument of journal format, is not the ingredient of described log lines.
Such as, field is used for describing a field value, and the BNF normal form of definition field is following:
field::name_len，name，type，prec，data_len，data
Wherein, name_len representes the string length of name, does not comprise NULL.Name representes field name.Type representes field type.Prec representes the precision of DECIMAL type.Data_len representes the byte stream length of data.Data representes the data word throttling.
For another example, field_desc is used for describing the definition of a field, except do not comprise the data and field just the same.The BNF normal form of definition field_desc is following:
field_desc::name_len，name，type，prec，data_len
Mix daily record and be divided into 4 types from function, it comprises: DML (Data ManipulationLanguage) statement bracket daily record, DML basic operation daily record, affairs control daily record and checkpoint daily record.Every type of corresponding respectively corresponding journal format of daily record, i.e. DML statement bracket daily record, DML basic operation daily record, affairs control daily record and the corresponding DML statement bracket of checkpoint daily record journal format, DML basic operation journal format, affairs control journal format and checkpoint journal format.
Every kind of daily record beginning all has following structure:
lsn:len:sync:op_code
Wherein, lsn representes log serial number, and is unique in the database overall situation.Lsn is used for explaining the order that this daily record order begins to carry out.Len representes except except the shared length of lsn and len, the length of this daily record.Sync representes whether this daily record is that needs produce synchronization log, the asynchronous daily record of 0 expression, non-0 expression synchronization log.Op_code representes the daily record command code, is used to discern the type of log lines.For the ease of reading; Op_code uses the underlined memonic symbol form of capitalization English letter to represent; Replace to separately by the implementor during concrete the realization that different numerical gets final product, like TRANSACTION_BEGIN, DELETE_BEGIN, INSERT_VFIELD, CHECK_POINT_END etc.And TRANSACTION_BEGIN, DELETE_BEGIN, INSERT_VFIELD, CHECK_POINT_END are memonic symbols, are not character strings, as the macro definition of C language.The coding span of op_code is 1～255.": " symbol that occurs for any position in the journal format all not necessarily, whether it only plays the effect of separating each field, and when resolving daily record, help verification log lines form correct.When daily record specifically realizes, can remove ": " symbol, also can keep ": " symbol, perhaps replace with other separators.
The DML statement refers to sql commands such as UPDATE, INSERT, DELETE, TRUNCATE_TABLE, DROP_TABLE.For every sql command, its corresponding mixing daily record comprises beginning and end mark at least, and these begin to be called the DML statement bracket with end mark.The mixing daily record of every sql command generation begins with an op_BEGIN daily record, and some basic operation daily records are being followed in the centre, finishes with an op_END or op_ABORT at last.Op replaces with the sql command name, like INSERT_BEGIN.
DML statement bracket journal format is following:
lsn:len:sync:op_BEGIN
lsn:len:sync:op_END
lsn:len:sync:op_ABORT
In DML statement bracket journal format, op can substitute with UPDATE, INSERT, DELETE, TRUNCATE_TABLE or DROP_TABLE.
The DML statement bracket can provide the information about relation table.Part also provides the keyword message of related record.As: lsn:len:sync:INSERT_BEGIN:tid:tn_len:table_name:other_in fo
Wherein, tn_len representes the length of table_name, is the byte type; The title that table_name then representes.Other_info representes the keyword message that writes down, like DESC:count: [(field_desc)]+, it is exactly the keyword message that related record is ordered in this daily record of INSERT_BEGIN.
The DML basic operation is meant the operation of accomplishing the required practical implementation change data of DML order.The DML order comprises SELECT, UPDATE, INSERT, DELETE.The corresponding daily record of DML basic operation mainly contains following 6 kinds:
1) DELETE_VFIELD, variable-length field of expression deletion;
2) INSERT_VFIELD, a variable-length field is inserted in expression;
3) UPDATE_FIELD, a fixed-length field is upgraded in expression;
4) DELETE_RECORD, record row of expression deletion;
5) INSERT_RECORD, a record row is inserted in expression;
6) FREE_PAGE, expression discharges several pages.
DML basic operation journal format:
lsn:len:sync:DELETE_VFIELD:(pno，rno)，(field)]*
lsn:len:sync:INSERT_VFIELD:(pno，rno)，(field)[，KEY:count:[(field)]*]
lsn:len:sync:UPDATE_FIELD:(pno，rno，offset)，OLD:(field)，NEW:(field)，KEY:count:[(field)]*
lsn:len:sync:DELETE_RECORD:(pno，rno)，(data_len，data)，KEY:count:[(field)]*
lsn:len:sync:INSERT_RECORD:(pno，rno)，(data_len，data)
lsn:len:sync:FREE_PAGE:count:[(pno)]*
The all DML basic operation daily records of DML statement bracket this sql command of parcel, the final execution result of indicating this sql command is successfully or fails.All DML basic operations all must be included within certain DML statement bracket daily record.So all daily records of a sql command generation are all defined by before and after the daily record of DML statement bracket, the centre comprises some DML basic operation daily records, and they are physically not necessarily being deposited continuously, but logically are an integral body.All daily records that produce sql command all be defined as sql command daily record, the perhaps daily record of DML sql command.
The journal format and the logical organization syntactic description thereof of DELETE order are as follows:
lsn:len:sync:DELETE_BEGIN:tid:tn_len:table_name
[
[lsn:len:sync:DELETE_VFIELD:(pno，rno)，(field)]*
lsn:len:sync:DELETE_RECORD:(pno，rno)，(data_len，data)，KEY:count:[(field)]*
]+
lsn:len:sync:DELETE_END|lsn:len:sync:DELETE_ABORT
Journal format and logical organization syntactic description thereof by above-mentioned sql command can know that the SQL statement daily record comprises to get over is engaged in recovering and the required full detail of data sync.
Affairs control daily record refers to an affairs initiation command and the corresponding daily record of the finish command.A complete transaction journal begins with a TRANSACTION_BEGIN daily record; The centre comprises some SQL statement daily records; End-state according to affairs are carried out ends up with a TRANSACTION_COMMIT daily record or TRANSACTION_ROLLBACK daily record at last.Each affairs control daily record has all intactly comprised all sql command daily records that belong to it.An affairs control daily record the inside can comprise 0 to a sql command daily record arbitrarily.Each affairs all has an affairs ID (Identity).Affairs ID is by affairs controller order assignment, affairs of unique expression in certain time period, the time sequencing of also having represented affairs to start basically.But,, reflect the sequencing that affairs are carried out fully so can not definitely guarantee affairs ID owing to allow transaction concurrency to carry out.But; Because affairs have the ACID attribute; Be atomicity (Atomicity), consistance (Consistency), isolation (Isolation), persistence (Durability); Perhaps exactly, because affairs have isolation, make that the effect of independent execution of affairs and concurrent execution is duplicate.
Affairs control journal format:
0:len:1:TRANSACTION_BEGIN:tid
0:len:1:TRANSACTION_COMMIT:tid
0:len:1:TRANSACTION_ROLLBACK:tid
In affairs control journal format, lsn is set to 0, and sync is set to 1.
Checkpoint daily record index knowledge checkpoint begins the daily record with the finish time.Registration CHECK_POINT_BEGIN daily record was registered the CHECK_POINT_END daily record after finishing when the checkpoint began.
The checkpoint journal format:
0:len:1:CHECK_POINT_BEGIN:cpn
0:len:1:CHECK_POINT_END:cpn
In the journal format of checkpoint, lsn is set to 0, and sync is set to 1.
Similar with the daily record of DML statement bracket, affairs control daily record and checkpoint daily record all are the daily record brackets, promptly all comprise a beginning daily record and a corresponding end log.These three kinds aim at writing down the op_BEGIN daily record before practical operation begins day; After practical operation is accomplished according to the daily record of execution result end of record (EOR), like INSERT_END, INSERT_ABORT, TRANSACTION_COMMIT, TRANSACTION_ROLLBACK, CHECK_POINT_END etc.
In addition, the checkpoint daily record only indicates the start-stop moment of checkpointed, does not have relation of inclusion in logic with affairs control daily record or sql command daily record.The checkpoint daily record mainly is used for doing system failure recovery.Affairs control daily record is used for supporting transaction failure to recover, and participation system fault recovery and media failure recover.So affairs control daily record needs when not only system moves, and all needs when fault recovery or system restart.
See also Fig. 1 and Fig. 2, wherein, Fig. 1 is the schematic flow sheet of the present invention's mixing daily record generation method, and Fig. 2 is the idiographic flow synoptic diagram of Fig. 1.
Mixing daily record generation method may further comprise the steps:
S1, open or create hybrid log file;
S2, receive user command, and judge the type of said user command;
The type of S3, the user command that draws according to said judgement is registered corresponding command type daily record.
The type that the user sends order comprises DML sql command, transaction commands or checkpoint order, its corresponding DML sql command daily record respectively, affairs control daily record and checkpoint daily record.
The idiographic flow that mixes daily record generation method may further comprise the steps:
101, open or create a hybrid log file;
102, receive user command;
103, judge whether it is the DML sql command, if, carry out 104, otherwise, carry out 109;
104, register the op_BEGIN daily record of this order;
105, registration DML basic operation daily record;
106, carry out the DML basic operation;
107, judge whether next bar DML basic operation in addition, if, carry out 105, otherwise, carry out 108;
108, register the op_END daily record of this order, carry out 115 again;
109, judge whether it is transaction commands, if, carry out 110, otherwise, carry out 111;
110, registration correspondent transaction command log carries out 115 again;
111, judge whether it is the checkpoint order, if carry out 112, otherwise carry out 115;
112, registration CHECK_POINT_BEGIN daily record;
113, carry out the checkpoint process;
114, registration CHECK_POINT_END daily record;
115, judge whether next command in addition, if carry out 103, otherwise carry out 116;
116, wait for user input commands.
Can know by above step, for the DML sql command, registration op_BEGIN daily record earlier before command execution; In the process of implementation, during DML basic operation of every beginning, its DML basic operation daily record of registration earlier; Execute after all DML basic operations, according to the net result of carrying out, if failure or abnormal ending; Registration op_ABORT daily record, otherwise success or normal termination, registration op_END daily record; Except the querying command of SELECT and so on is not asked to, the execution of all the other DML orders all will produce the sql command daily record.
For transaction commands, only need before carrying out corresponding order, to register corresponding daily record affairs initiation command TRANSACTION_BEGIN, affairs submiting command TRANSACTION_COMMIT or transaction retraction order TRANSACTION_ABORT.If when creating hybrid log file, the log buffer of distribution district is arranged, then after accomplishing registration TRANSACTION_COMMIT or TRANSACTION_ABORT, brush log buffer district data into hybrid log file.
For the checkpoint order, the checkpoint process is then carried out in registration CHECK_POINT_BEGIN daily record earlier, registers the CHECK_POINT_END daily record at last.
Said daily record generation method is followed and is write formula daily record (Write-Ahead Logging is called for short WAL) principle in advance.
In the memory database system operational process, system can produce the hybrid log file that a sequence numbers in order, and the checkpointed that checkpointed of manually initiating or system initiate automatically all will produce or updating memory database image file.Before each checkpointed, all do not accomplish the minimum journal file side-play amount offset of affairs and the journal file log_file_no of current use to write down current time.Wherein, offset is meant when article one affairs control daily record TRANSACTION_BEGIN that all ongoing offices produce writes journal file that with respect to the minimum value in the side-play amount of file header, it calculates with byte.By the time after checkpointed finishes, write these two values respectively in the header structure of image file, flush to the header structure of image file in the permanent storage simultaneously, perhaps remember in the affairs control documents.
Adopt the generation method of mixing daily record of the present invention, just do not carry out when checkpointed does not need issued transaction to remain static, it can be carried out with transaction concurrency.This mode belongs to the dynamic backup mode, and the image file of its generation not necessarily is in coherency state, so need assist with journal file, promptly is aided with hybrid log file and carries out that affairs are reformed and rollback, guarantees the coherency state of data with this.
See also Fig. 3, it is the concrete realization flow figure of daily record registration among Fig. 1.
The daily record registration is accomplished through daily record registration function.Daily record registration function comprises that at least two are gone into ginseng, i.e. daily record length and daily record volume data.If the log buffer of distribution district is arranged, the concrete realization of daily record registration may further comprise the steps:
201, receive the daily record register request;
202, whether the audit log buffer area has enough spaces to hold this daily record, if, carry out 204, otherwise, carry out 203;
203, the buffer area data are all outputed to hybrid log file, empty buffer area, carry out 204;
204, check the size that aims at side-play amount and current journal file in the buffer area this day, obtain the real offset that this daily record begins to locate with respect to file in the future to the two addition in hybrid log file;
205, write this daily record to the log buffer district;
206, return success the byte number that writes and aim at the real offset in the hybrid log file this day to caller.
If do not distribute the log buffer district; The concrete realization of daily record registration only need be checked the side-play amount that aims at this day in the hybrid log file and the size of current journal file, obtains the two addition to aim at said day the real offset that begins to locate with respect to hybrid log file in the hybrid log file; Write said daily record to hybrid log file; Return success the byte number that writes and aimed at the actual shifts in the hybrid log file in said day to caller.
Because transaction concurrency is carried out, so the sql command daily record all possibly produce registration at any time at any time with other daily records, so in journal file, these daily records can't be deposited according to its logical organization continuously, deposit but intersect.But every daily record is complete depositing, and can therefrom not got involved by other daily records and interrupt.
See also Fig. 4, it is the process flow diagram that the present invention is used for the mixing daily record analytic method of system failure recovery.
The present invention adopts scan edge journal file limit to generate the method for REDO/UNDO transaction block, only needs a scanning just can accomplish the parsing to whole journal file.The set of all daily records that comprise affairs calls a REDO/UNDO transaction block (REDO/UNDO Transaction Block).The set of the sql command statement that comprises affairs calls the sql command statement block, is called for short statement block.Closed REDO/UNDO transaction block is initial by TRANSACTION BEGIN, and with TRANSACTION COMMIT or TRANSACTION ROLLBACK ending, the centre comprises statement blocks such as several DELETE, UPDATE or INSERT.
In resolving, run into TRANSACTION COMMIT or TRANSACTIONROLLBACK, there is closed REDO/UNDO transaction block node in expression.
Be used for the mixing daily record analytic method of system failure recovery, it may further comprise the steps:
After a REDO/UNDO transaction block forms, then carry out the REDO/UNDO operation immediately.
301, daily record resolver handle hp of application, read-only mode is opened hybrid log file.Earlier be written into internal memory to image file; Find the deviation post of the daily record that first hybrid log file and article one will read then according to the journal file of aforesaid minimum skew offset and journal file log_file_no value, read-only mode is opened hybrid log file;
302, judge whether to exist closed REDO/UNDO transaction block node, if, carry out 303, otherwise, carry out 304;
303, compose REDO/UNDO transaction block node pointer to output variable, return and obtain REDO/UNDO transaction block success status value;
304, read the log lines data, and judge whether to run into and read document lending or end-of-file mark, if, then carry out 305, otherwise, carry out 306;
305, judge whether it is to run into end-of-file mark, if, return the state value that expression runs into end-of-file mark, otherwise, concrete error status value returned;
306, resolve this log lines, and judge whether to resolve successfully, if carry out 307, otherwise return concrete error status value;
307, judge whether this log lines is TRANSACTION_BEGIN, if, carry out 308, otherwise, carry out 309;
308, generate a REDO/UNDO transaction block node, add it to daily record resolver handle hp, and carry out 304;
309, judge whether this log lines is op_BEGIN, if, carry out 310, otherwise, carry out 311;
310, generate a statement block node, be inserted in the affiliated REDO/UNDO transaction block node, and carry out 304;
311, judge whether this log lines is TRANSACTION_END, if, carry out 312, otherwise, carry out 313;
312, find the REDO/UNDO transaction block node of coupling, this log lines is inserted, and put its state and be closed sign, and carry out 303;
313, judge whether to find the statement block node of coupling, if, carry out 315, otherwise, carry out 314;
314, return concrete error status value;
315, this log lines is inserted, and judged whether this log lines is op_END, if, carry out 316, otherwise, carry out 304;
316, this statement block is changed to closure state, and carries out 304.
Can know by above step, in the resolving, before returning the sign that runs into the end of file, be accompanied by and read journal file and the process of resolving log lines, up to returning a REDO/UNDO transaction block success status value.These REDO/UNDO transaction block constitute a formation, the REDO formation when being exactly system failure recovery.Described REDO formation possibly comprise the REDO/UNDO transaction block with the ROLLBACK ending, need handle according to UNDO, and all the other are all handled according to REDO.After all journal file dissection process finish; Possibly remain some among the daily record resolver hp in addition and not have closed REDO/UNDO transaction block; Use this moment and force taking-up mode backward to obtain each REDO/UNDO transaction block; UNDO formation when having constituted system failure recovery thus, and handle according to UNDO.In the system failure recovery process, handle the REDO formation, handle the UNDO formation again, this moment, database recovery was to up-to-date coherency state.
See also Fig. 5, it is the process flow diagram that the present invention is used for the synchronous mixing daily record analytic method of non-real-time data.
Non-real-time data is synchronous, promptly regular or irregular data sync.The mixing daily record analytic method that non-real-time data is synchronous, much the same with the analytic method of system failure recovery, its difference only is that the target of the mixing daily record analytic method that non-real-time data is synchronous is the synchronous SQL statement that obtains according to the statement block generation.The step difference is: step 403 is obtained current statement block, generates the synchronous SQL statement of next bar according to current statement block, and internal pointer is displaced to next bar be untreated log lines or next statement block node.Judge whether to generate synchronous SQL statement again, if, return and obtain synchronous SQL statement success status value, otherwise, concrete error status value returned.
For fault recovery, only need return and obtain REDO/UNDO transaction block success status value and get final product; And for data sync; At first obtain next bar log lines pointer of current statement block node; Generate the synchronous SQL statement of next bar, be displaced to the untreated log lines of next bar to internal pointer simultaneously, if run into op_END or op_ABORT log lines; Then be displaced to next statement block node, return the SQL statement text string of generation at last.
For the multinode database application system, the database of different nodes possibly carry out regular or irregular data sync, and these Database Systems are not necessarily isomorphism, possibly be Oracle or the such traditional database of Sybase such as goal systems.Use hybrid log file to resolve and generate the synchronous SQL statement text string that meets goal systems SQL grammer.
Compared with prior art; The present invention's mixing daily record generation method adopts the mixing daily record that mixes journal format to combine all information of transaction journal and synchronization log with analytic method; Each SQL statement daily record comprises to get over and be engaged in to recover and the required full detail of data sync, thereby has guaranteed that the mixing daily record both can serve as transaction journal, can serve as the data sync daily record again; A lot of redundant informations have been eliminated; Make whole daily record scale descend greatly, not only saved file storage, and reduced memory cache space and the cpu resource that consumes when generating these daily records.
And the present invention's mixing daily record generation method is supported the concurrent execution of affairs, and the concurrent execution of affairs and checkpoint process, improves the availability and the handling capacity of system greatly.And, only need be aided with hybrid log file and carry out affairs and reform and guarantee the coherency state of data with rollback.
Moreover the present invention's mixing daily record analytic method adopts scan edge journal file limit to generate the method for REDO/UNDO transaction block, a scanning is carried out in daily record just can be accomplished parsing and the processing to whole journal file, has improved resolution speed, has simplified treatment scheme.And, unified the construction process of REDO formation and UNDO formation, closed when the REDO/UNDO transaction block, just carry out the REDO/UNDO operation or generate synchronous SQL statement, disposing discharges the shared memory source of this REDO/UNDO transaction block immediately.
More than be merely preferred case study on implementation of the present invention, be not limited to the present invention, for a person skilled in the art, the present invention can have various changes and variation.All within spirit of the present invention and principle, any modification of being done, be equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (9)
1. one kind is mixed daily record generation method, it is characterized in that, may further comprise the steps:
Open or create hybrid log file;
Receive user command, and judge the type of said user command;
The type of the user command that draws according to said judgement is registered corresponding command type daily record;
The log lines beginning structure of said mixing daily record is lsn:len:sync:op_code, and said lsn representes the log lines sequence number, is used for explaining the order that the log lines order begins to carry out; Said len representes except except the shared length of said lsn and len; The length of said log lines, said sync representes whether said log lines is that needs produce synchronization log, and said op_code representes said log lines command code; Be used to discern the type of said log lines, said colon: be separator.
2. mixing daily record generation method according to claim 1 is characterized in that, the corresponding command type daily record of said registration may further comprise the steps:
Receive the daily record register request, judge whether said log buffer district has enough spaces to hold said daily record,, then write with a brush dipped in Chinese ink said hybrid log file to the data of said buffer area, and empty buffer area if there are not enough spaces;
Check the size that aims at said day side-play amount and said hybrid log file in the said buffer area, obtain the two addition said day to aim at the real offset in the said hybrid log file;
Said daily record is write said buffer area, and return success byte number and the said real offset that writes.
3. mixing daily record generation method according to claim 1; It is characterized in that the type that said judgement draws user command is the checkpoint order, then registers said checkpoint command log; And the execution of following said checkpoint to order, the execution and the transaction concurrency of the order of said checkpoint are carried out.
4. mixing daily record generation method according to claim 1 is characterized in that said lsn is unique in the database overall situation, and said sync is the asynchronous daily record of 0 expression, is non-0 expression synchronization log, and the coding span of said op_code is 1 to 255.
5. according to the described mixing daily record of claim 1,2 or 3 generation method, it is characterized in that the form of said mixing daily record comprises DML statement bracket journal format, DML basic operation journal format, affairs control journal format and checkpoint journal format; The op_code of said DML statement bracket journal format, op are the DML order, and code is BEGIN, END or ABORT, and the op_code of said DML basic operation journal format is DELETE_VFIELD; INSERT_VFIELD, UPDATE_FIELD, DELETE_RECORD, INSERT_RECORD or FREE_PAGE; The lsn of said affairs control journal format is 0, and sync is 1, and op is TRANSACTION; Code is BEGIN, COMMIT or ROLLBACK, and the lsn of said checkpoint journal format is 0; Sync is 1, and op is CHECK_POINT, and code is BEGIN or END.
6. one kind is mixed the daily record analytic method, it is characterized in that, may further comprise the steps:
Steps A, application date book resolver handle are opened hybrid log file;
Step B, judge whether to exist closed REDO/UNDO transaction block node; If, execution in step C, otherwise; Execution in step D; Said REDO/UNDO transaction block is represented the set of all daily records that affairs comprise, and said closed REDO/UNDO transaction block is initial by TRANSACTION_BEGIN, with TRANSACTION_COMMIT or TRANSACTION_ROLLBACK ending;
Step C, compose said REDO/UNDO transaction block node pointer to output variable, return and obtain REDO/UNDO transaction block success status value, end log process of analysis;
Step D, resolve the log lines of hybrid log file successively, carry out the operation that generates said REDO/UNDO transaction block, again execution in step B.
7. mixing daily record analytic method according to claim 6; It is characterized in that; Carrying out the operation that generates said REDO/UNDO transaction block among the said step D may further comprise the steps: it is TRANSACTION_BEGIN that parsing runs into log lines, generates REDO/UNDO transaction block node, judges the affiliated command type of log lines more successively; Running into log lines is TRANSACTION_END, and puts said REDO/UNDO transaction block node and be closed sign.
8. according to claim 6 or 7 described mixing daily record analytic methods, it is characterized in that said REDO/UNDO transaction block comprises statement block; Said statement block begins with op_BEGIN, and closed statement block begins with op_BEGIN, the op_END ending; Op is the DML order; The log lines that said step D resolves hybrid log file successively comprises: it is op_BEGIN that parsing runs into said log lines, and generated statement agllutination point is judged the affiliated command type of log lines more successively; Running into log lines is op_END, and puts said REDO/UNDO transaction block node and be closed sign.
9. mixing daily record analytic method according to claim 8 is characterized in that said step C replaces with: obtain current statement block; Generate the synchronous SQL statement of next bar according to current statement block, internal pointer is displaced to next bar be untreated log lines or next statement block node, and judges whether to generate synchronous SQL statement; If return and obtain synchronous SQL statement success status value, end log process of analysis; Otherwise, return concrete error status value, the end log process of analysis.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2008102418990A CN101464890B (en) | 2008-12-30 | 2008-12-30 | Mixed log generation method and analysis method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2008102418990A CN101464890B (en) | 2008-12-30 | 2008-12-30 | Mixed log generation method and analysis method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101464890A CN101464890A (en) | 2009-06-24 |
| CN101464890B true CN101464890B (en) | 2012-05-23 |
Family
ID=40805465
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2008102418990A Expired - Fee Related CN101464890B (en) | 2008-12-30 | 2008-12-30 | Mixed log generation method and analysis method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101464890B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102385624B (en) * | 2011-10-25 | 2014-01-01 | 曙光信息产业(北京)有限公司 | DFS (distributed file system)-oriented log data organization method |
| CN102970373B (en) * | 2012-12-03 | 2015-11-11 | 亚信科技（中国）有限公司 | A kind of method of data synchronization, Apparatus and system |
| CN106528607B (en) * | 2016-09-26 | 2019-10-29 | 武汉开目信息技术股份有限公司 | Attribute data separates the ensuring method of rear consistency with file data in PDM system |
| CN108073500A (en) * | 2016-11-14 | 2018-05-25 | 深圳联友科技有限公司 | A kind of log recording method and system |
| CN110019062A (en) * | 2017-08-14 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Method of data synchronization and system |
| CN107679195B (en) * | 2017-10-09 | 2021-01-22 | 杭州安恒信息技术股份有限公司 | Semi-master-slave database synchronization method and system based on HTTPS protocol transmission |
| CN107766490A (en) * | 2017-10-17 | 2018-03-06 | 链家网（北京）科技有限公司 | Automatic operation method, system, equipment and the storage medium of SQL files |
| EP3696658B1 (en) * | 2017-12-05 | 2023-10-04 | Huawei Technologies Co., Ltd. | Log management method, server and database system |
| CN108647112B (en) * | 2018-03-16 | 2022-03-01 | 创新先进技术有限公司 | Data backup method and device and distributed transaction processing system |
| CN110231909B (en) * | 2019-05-15 | 2021-03-05 | 广州视源电子科技股份有限公司 | Writing operation processing method and device |
| CN112256656B (en) * | 2019-07-22 | 2023-04-18 | 金篆信科有限责任公司 | Transaction rollback method and device, database, system and computer storage medium |
| CN110633258B (en) * | 2019-08-07 | 2022-08-12 | 平安科技（深圳）有限公司 | Log insertion method, device, computer device and storage medium |
| CN110780857B (en) * | 2019-10-23 | 2024-01-30 | 杭州涂鸦信息技术有限公司 | Unified log component |
-
2008
- 2008-12-30 CN CN2008102418990A patent/CN101464890B/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN101464890A (en) | 2009-06-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101464890B (en) | Mixed log generation method and analysis method | |
| DeWitt et al. | Implementation techniques for main memory database systems | |
| US7117229B2 (en) | Method and system for online reorganization of databases | |
| US5878410A (en) | File system sort order indexes | |
| US6411964B1 (en) | Methods for in-place online reorganization of a database | |
| CN100495400C (en) | Indexes on-line updating method of full text retrieval system | |
| US6263338B1 (en) | Method relating to databases | |
| US6298425B1 (en) | Computer disk management system using doublet A-B logging | |
| Graefe | A survey of B-tree logging and recovery techniques | |
| WO2007083371A1 (en) | Data integration device, method, and recording medium containing program | |
| US8560500B2 (en) | Method and system for removing rows from directory tables | |
| CN101853288A (en) | Configurable full-text retrieval service system based on document real-time monitoring | |
| CN101876996A (en) | Method and system for synchronizing data from main memory data bank to file data bank | |
| CN102955792A (en) | Method for implementing transaction processing for real-time full-text search engine | |
| CN107357920B (en) | Incremental multi-copy data synchronization method and system | |
| JP2006018796A (en) | Data processing method and device, and storage device and its processing program | |
| CN114942965B (en) | Method and system for accelerating synchronous operation of main database and standby database | |
| CN108536833A (en) | A kind of distributed, database and its construction method towards big data | |
| CN113934797A (en) | Banking industry super-large data synchronization method and system | |
| Graefe | Instant recovery for data center savings | |
| CN112612647B (en) | Log parallel replay method, device, equipment and storage medium | |
| Liu et al. | Incorporating transaction semantics to reduce reprocessing overhead in replicated mobile data applications | |
| CN112463447B (en) | Optimization method for realizing physical backup based on distributed database | |
| CN103685350B (en) | The synchronous method of storage system and relevant equipment | |
| Mittra | Database performance tuning and optimization: using Oracle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20120523 Termination date: 20171230 |