CN113656645B - Log consumption method and device - Google Patents
Log consumption method and device Download PDFInfo
- Publication number
- CN113656645B CN113656645B CN202010396856.0A CN202010396856A CN113656645B CN 113656645 B CN113656645 B CN 113656645B CN 202010396856 A CN202010396856 A CN 202010396856A CN 113656645 B CN113656645 B CN 113656645B
- Authority
- CN
- China
- Prior art keywords
- log
- target
- target log
- logs
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 230000004044 response Effects 0.000 claims abstract description 25
- 230000008859 change Effects 0.000 claims abstract description 18
- 238000003780 insertion Methods 0.000 claims description 45
- 230000037431 insertion Effects 0.000 claims description 45
- 238000004590 computer program Methods 0.000 claims description 9
- 230000000875 corresponding effect Effects 0.000 description 65
- 238000010586 diagram Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 10
- 238000012545 processing Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the disclosure discloses a log consumption method and a log consumption device. One embodiment of the method comprises the following steps: acquiring a log pointed by a queue head pointer of a target log queue, wherein the log in the target log queue is used for recording information for performing line change operation on a target database; in response to the acquired logs meeting preset merging conditions, merging a first number of logs in a target log list corresponding to a target log queue to obtain a second number of merged logs, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number; and (5) consuming the log after merging the second number of the bars. According to the embodiment, the number of logs required to be consumed is reduced, the consumption speed of the logs is improved, and the pressure of the electronic equipment for log consumption is reduced.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to a log consumption method and device.
Background
Typically, the database has a log for recording all transactions (events) and the modifications each transaction makes to the database.
For example, binlog is a log used to record database changes. In practice, consuming binlog logs through MQ (message queuing, message queues) is a relatively common means of production. Of these, binlog has three modes, raw, statement and mixed, respectively. Usually the raw mode is most commonly used. In the raw mode, it is possible to know which table and which row in the database changes the specific content. Many change events are often generated when a large number of update operations are performed on data in a database.
Disclosure of Invention
The disclosure provides a log consumption method and a log consumption device.
In a first aspect, embodiments of the present disclosure provide a log consumption method, the method comprising: acquiring a log pointed by a queue head pointer of a target log queue, wherein the log in the target log queue is used for recording information for performing line change operation on a target database; in response to the acquired logs meeting preset merging conditions, merging a first number of logs in a target log list corresponding to a target log queue to obtain a second number of merged logs, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number; and (5) consuming the log after merging the second number of the bars.
In some embodiments, the logs include operation information indicating operations performed on data in the target database, and table information indicating tables including the operated data, the respective logs included in the target log list indicating the same operations, and the respective logs included in the target log list indicating the same tables.
In some embodiments, the preset merge condition comprises: the target log list corresponding to the target log queue is not empty, and one of: the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is larger than or equal to a preset time interval; the time difference between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than a preset time interval; and, the operation information or table information included in the log pointed by the column tail pointer included in the target log list and the operation information or table information included in the acquired log indicate different operations or tables.
In some embodiments, merging a first number of logs in a target log list corresponding to a target log queue includes: and merging the logs included in the target log list into a group of logs.
In some embodiments, before merging the first number of logs in the target log list corresponding to the target log queue to obtain the second number of merged logs in response to the acquired logs meeting a preset merging condition, the method further includes: in response to the target log list corresponding to the target log queue being empty, inserting the acquired log into the target log list corresponding to the target log queue, and recording an insertion time of the acquired log.
In some embodiments, before merging the first number of logs in the target log list corresponding to the target log queue to obtain the second number of merged logs in response to the acquired logs meeting a preset merging condition, the method further includes: in response to a preset insertion condition being met, inserting the acquired log into a target log list corresponding to a target log queue; wherein, the preset insertion condition includes: the target log list corresponding to the target log queue is not empty; the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than a preset time interval; the operation information included in the log pointed by the tail pointer included in the target log list and the operation information included in the acquired log indicate the same operation; the list tail pointer included in the target log list points to the same table as the table information included in the acquired log.
In some embodiments, after obtaining the log pointed to by the head pointer of the target log queue, the method further comprises: the acquired log pointed to by the head pointer of the target log queue is deleted from the target log queue.
In some embodiments, after consuming the second number of consolidated logs, the method further comprises: and deleting the consumed logs included in the target log list from the target log list in response to completing the consumption of the second number of merged logs.
In a second aspect, embodiments of the present disclosure provide a log consuming device comprising: an acquisition unit configured to acquire a log pointed by a head pointer of a target log queue, wherein the log in the target log queue is information for recording a line change operation on a target database; the merging unit is configured to respond to the fact that the acquired logs meet preset merging conditions, merge a first number of logs in a target log list corresponding to a target log queue to obtain a second number of merged logs, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number; and the consumption unit is configured to consume the log after the second number of the merged stripes.
In some embodiments, the logs include operation information indicating operations performed on data in the target database, and table information indicating tables including the operated data, the respective logs included in the target log list indicating the same operations, and the respective logs included in the target log list indicating the same tables.
In some embodiments, the preset merge condition comprises: the target log list corresponding to the target log queue is not empty, and one of: the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is larger than or equal to a preset time interval; the time difference between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than a preset time interval; and, the operation information or table information included in the log pointed by the column tail pointer included in the target log list and the operation information or table information included in the acquired log indicate different operations or tables.
In some embodiments, the merging unit comprises: and a merging subunit configured to merge the respective logs included in the target log list into a set of logs.
In some embodiments, the apparatus further comprises: a first inserting unit configured to insert the acquired log into the target log list corresponding to the target log queue in response to the target log list corresponding to the target log queue being empty, and record an insertion time of the acquired log.
In some embodiments, the apparatus further comprises: a second inserting unit configured to insert the acquired log into a target log list corresponding to the target log queue in response to satisfaction of a preset insertion condition; wherein, the preset insertion condition includes: the target log list corresponding to the target log queue is not empty; the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than a preset time interval; the operation information included in the log pointed by the tail pointer included in the target log list and the operation information included in the acquired log indicate the same operation; the list tail pointer included in the target log list points to the same table as the table information included in the acquired log.
In some embodiments, the apparatus further comprises: a first deleting unit configured to delete, from a target log queue, an acquired log pointed to by a head pointer of the target log queue.
In some embodiments, the apparatus further comprises: and a second deleting unit configured to delete, from the target log list, the consumed log included in the target log list in response to completion of consumption of the second number of the merged logs.
In a third aspect, embodiments of the present disclosure provide an electronic device, comprising: one or more processors; and a storage device having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to implement a method as in any of the log consumption methods described above.
In a fourth aspect, embodiments of the present disclosure provide a computer readable medium having stored thereon a computer program which, when executed by a processor, implements a method as in any of the embodiments of the log consumption method described above.
According to the log consumption method and device, the logs pointed by the queue head pointer of the target log queue are obtained, the logs in the target log queue are information for recording line change operation on the target database, then the first number of the logs in the target log list corresponding to the target log queue are combined to obtain the second number of combined logs in response to the fact that the obtained logs meet preset combining conditions, wherein the first number is greater than or equal to 2, the second number is smaller than the first number, finally the second number of combined logs are consumed, the number of logs required to be consumed is reduced, the consumption speed of the logs is improved, and the pressure of log consumption of electronic equipment is reduced.
Drawings
Other features, objects and advantages of the present disclosure will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the following drawings:
FIG. 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;
FIG. 2 is a flow chart of one embodiment of a log consumption method according to the present disclosure;
FIG. 3 is a schematic diagram of one application scenario of a log consumption method according to the present disclosure;
FIG. 4 is a flow chart of yet another embodiment of a log consumption method according to the present disclosure;
FIG. 5 is a flow chart for one application scenario of FIG. 4;
FIG. 6 is a schematic structural diagram of one embodiment of a log consuming device according to the present disclosure;
Fig. 7 is a schematic diagram of a computer system suitable for use in implementing embodiments of the present disclosure.
Detailed Description
The present disclosure is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.
It should be noted that, without conflict, the embodiments of the present disclosure and features of the embodiments may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
FIG. 1 illustrates an exemplary system architecture 100 to which embodiments of a log consumption method or log consumption device of embodiments of the present disclosure may be applied.
As shown in fig. 1, a system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or transmit data or the like. Various client applications, such as shopping class application video playing software, news information class applications, image processing class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc., may be installed on the terminal devices 101, 102, 103.
The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices having data transmission functions, including but not limited to smart phones, tablet computers, laptop and desktop computers, and the like. When the terminal devices 101, 102, 103 are software, they can be installed in the above-listed electronic devices. Which may be implemented as multiple software or software modules (e.g., software or software modules for providing distributed services) or as a single software or software module. The present invention is not particularly limited herein.
The server 105 may be a server providing various services, such as a background server operating on a database (e.g., a database for storing user data) based on data sent by the terminal devices 101, 102, 103. The background server may perform operations such as updating, adding, deleting, querying, etc. on the database based on the data sent by the terminal devices 101, 102, 103. As an example, the server 105 may be a cloud server.
It should be noted that, the server may be hardware, or may be software. When the server is hardware, the server may be implemented as a distributed server cluster formed by a plurality of servers, or may be implemented as a single server. When the server is software, it may be implemented as a plurality of software or software modules (e.g., software or software modules for providing distributed services), or as a single software or software module. The present invention is not particularly limited herein.
It should also be noted that, the log consumption method provided by the embodiment of the present disclosure may be executed by a server, or may be executed by a terminal device, or may be executed by the server and the terminal device in cooperation with each other. Accordingly, each part (for example, each unit, subunit) included in the log consuming apparatus may be all disposed in the server, may be all disposed in the terminal device, or may be disposed in the server and the terminal device, respectively.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation. When the electronic device on which the log consuming method is running does not need to perform data transmission with other electronic devices, the system architecture may include only the electronic device (e.g., server or terminal device) on which the log consuming method is running.
With continued reference to fig. 2, a flow 200 of one embodiment of a log consumption method according to the present disclosure is shown. The log consumption method comprises the following steps:
step 201, a log pointed by a queue head pointer of a target log queue is obtained.
In this embodiment, the execution body of the log consuming method (for example, the server or the terminal device shown in fig. 1) may obtain, from the target log queue, the log pointed by the queue head pointer of the target log queue from other electronic devices or locally through a wired connection manner or a wireless connection manner. The log in the target log queue is used for recording information for performing line change operation on the target database.
The target log queue may be a log queue stored in an MQ (message queuing, message queue). Typically, one thread may be started for each MQ.
As an example, the log in the target log queue may be a binlog log. The binlog is a binary log of a MySQL (relational database management system) database, and is used for recording information of performing line change operation on a target database. Here, the user may perform a row change operation on the database through the SQL statement. In practice, binlog logs may be consumed by MQ. Of these, binlog has three modes, raw, statement and mixed, respectively. Usually the raw mode is most commonly used. In the raw mode, it is possible to know which table and which row in the database changes the specific content. Many change events are often generated when a large number of update operations are performed on data in a database.
In some alternative implementations of the present embodiment, the log includes operation information and table information. Wherein the operation information indicates an operation performed on data in the target database. The table information indicates a table including the operated data. The operation information included in each log in the target log list indicates the same operation. Each log in the target log list includes a table letter indicating the same table.
Step 202, merging a first number of logs in a target log list corresponding to a target log queue to obtain a second number of merged logs in response to the acquired logs meeting preset merging conditions.
In this embodiment, in the case where the log obtained in step 201 meets the preset merging condition, the execution body may merge the first number of logs in the target log list corresponding to the target log queue to obtain the second number of merged logs. Wherein the first number is greater than or equal to 2 and less than or equal to the number of logs included in the current (e.g., when performing this step 202) target log list. The second number is less than the first number and greater than or equal to 1. Typically, the first number of logs may be adjacent to each other, i.e. in the target log list, there are no logs to be merged between the first number of logs to be merged. Each target log queue may correspond to a target log list. The correspondence of the target log queue and the target log list may be established in advance.
It will be appreciated that consuming the second number of consolidated journals will produce the same results as consuming the first number of journals. As an example, the first number of logs may call 1000 times a method (function, also called function) for subtracting 1 from the current value. After back-end processing, a second number of strips of the consolidated log may be used to invoke 1 method for subtracting 1000 from the current value. Illustratively, the backend may generate 1 piece of processing information from the second number of consolidated logs. This process information may call 1 method for subtracting 1000 from the current value.
The above-described preset merge condition may be a predetermined condition for indicating the log in the merge target log list. As an example, the preset merge condition may include: the storage space occupied by the obtained log is larger than or equal to a preset storage space threshold value.
As an example, in the case where the obtained logs meet the preset merging condition, the execution body may merge all the logs in the target log list into 1 log, thereby obtaining 1 merged log. Optionally, in the case that the obtained logs meet the preset merging condition, the executing body may merge 10 logs with the insertion time farthest from the current time (for example, the time when the step 202 is executed) in the target log list into 9 logs, so as to obtain 9 merged logs.
And 203, consuming the log after the second number of the merged stripes.
In this embodiment, the execution body may consume the log after merging the second number of strips obtained in step 202.
In practice, the second number of consolidated logs may be consumed based on the binlog consumption component. Optionally, log consumption logic can be determined according to actual requirements, so that the log after the second number of the strips are combined is consumed.
Here, the log consumption method of the present disclosure may be applied in raw, statement or mixed modes. Since in the raw mode, many change events (i.e., journals) are often generated when a large number of update operations are performed on the data in the database. Therefore, when the log consumption method disclosed by the invention is applied to a raw mode, compared with other modes, the number of logs required to be consumed can be further reduced, the log consumption speed is improved, and the pressure of the electronic equipment for log consumption is reduced.
With continued reference to fig. 3, fig. 3 is a schematic diagram of an application scenario of the log consumption method according to the present embodiment. In the application scenario of fig. 3, the server 301 first acquires the log (for example, log 1) pointed by the queue head pointer of the target log queue 302, where the log in the target log queue is used to record information for performing a row change operation on the target database, and then, if the acquired log (i.e., log 1) meets a preset merge condition, the server 301 merges a first number of logs (for example, log a and log b, two logs) in the target log list 303 corresponding to the target log queue 302, and obtains a second number of merged logs 304 (in the illustrated example, log x,1 merged log). Finally, the server 301 consumes the second number of consolidated logs 304.
In the prior art, the binlog log consumption is usually performed piece by piece, and each time a binlog log ack (a post-consumption confirmation operation) is consumed, only after one binlog is consumed, the next binlog is consumed.
According to the method provided by the embodiment of the disclosure, the logs pointed by the queue head pointer of the target log queue are obtained, wherein the logs in the target log queue are used for recording information for performing line change operation on the target database, then, under the condition that the obtained logs meet preset merging conditions, the first number of the logs in the target log list corresponding to the target log queue are merged to obtain the second number of the merged logs, wherein the first number is greater than or equal to 2, the second number is smaller than the first number, and finally, the logs after the second number of the merged logs are consumed, so that the number of the logs required to be consumed is reduced, the consumption speed of the logs is improved, and the pressure of log consumption of electronic equipment is reduced. And the back end can directly consume the combined logs, thereby reducing a great amount of network request time delay.
In some optional implementations of this embodiment, before executing step 202, in a case where the target log list corresponding to the target log queue is empty, the execution body may further insert the obtained log into the target log list corresponding to the target log queue, and record an insertion time of the obtained log.
In some optional implementations of this embodiment, the preset combining conditions include: the target log list corresponding to the target log queue is not empty, and any of:
The first item, the time interval between the acquisition time of the acquired log and the insertion time of the first log inserted into the target log list, which is included in the target log list, is greater than or equal to a preset time interval.
The preset time interval may be a fixed value that is preset, or may be a value that is preset and changeable.
It will be appreciated that when the preset merge condition includes the first item described above, the throughput of log consumption may be controlled by the setting of the preset time interval.
The second item, the time difference between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than a preset time interval; and, the operation information or table information included in the log pointed by the column tail pointer included in the target log list and the operation information or table information included in the acquired log indicate different operations or tables.
The "operation information or table information included in the log pointed by the tail pointer included in the target log list" and the obtained operation information or table information included in the log indicate different operations or tables ", that is: the operation information included in the log pointed by the tail pointer included in the target log list and the operation information included in the acquired log indicate different operations; or the list tail pointer included in the target log list points to the table information included in the log and the acquired table information included in the log, and indicates different tables.
It is understood that when the preset merge condition includes the above-described second item, it may be ensured that the included operation information of each of the non-consumed logs in the target log list corresponding to the target log queue indicates the same operation, and the included table information of each of the non-consumed logs in the target log list corresponding to the target log queue indicates the same table, so that log merge and log consumption proceed in order.
In some optional implementations of this embodiment, before executing step 202, the executing body may further insert the obtained log into a target log list corresponding to the target log queue if a preset insertion condition is satisfied.
Wherein, the preset insertion conditions include:
the target log list corresponding to the target log queue is not empty;
the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than a preset time interval;
The operation information included in the log pointed by the tail pointer included in the target log list and the operation information included in the acquired log indicate the same operation; and
The list tail pointer included in the target log list points to the same table as the table information included in the acquired log.
It will be appreciated that in the above alternative implementation, it may be ensured that the included operation information of each of the non-consumed logs in the target log list corresponding to the target log queue indicates the same operation, and that the included table information of each of the non-consumed logs in the target log list corresponding to the target log queue indicates the same table for the purpose of log merging and ordered progress of log consumption.
In some optional implementations of this embodiment, after performing step 201, the executing entity may further delete the acquired log pointed to by the head pointer of the target log queue from the target log queue.
It will be appreciated that after the above step 201 is performed, the log obtained in step 201 may be processed, for example, the log obtained in step 201 is inserted into a target log list. Therefore, the execution body can delete the acquired logs pointed by the queue head pointer of the target log queue from the target log queue, so that the queue head pointer of the target log queue points to other logs, the other logs pointed by the queue head pointer can be acquired later, and the other logs are processed, thereby avoiding repeated processing of the logs.
In some optional implementations of this embodiment, after performing step 203, the execution body may further delete the consumed log included in the target log list from the target log list when the consumption of the second number of combined logs is completed.
It will be appreciated that after performing step 203 described above, consumption of the second number of consolidated logs resulting from step 203 may generally be completed. Therefore, the execution body can delete the consumed logs which are included in the target log list and are deleted from the target log list, so that the combined logs which are not consumed in the target log list are consumed later, repeated consumption of the logs is avoided, and the consumed logs which are included in the target log list are deleted from the target log list only when the consumption of the second number of combined logs is completed, so that the logs can be prevented from being lost.
With further reference to fig. 4, a flow 400 of yet another embodiment of a log consumption method is shown. The process 400 of the log consumption method includes the steps of:
step 401, obtaining a log pointed by a queue head pointer of a target log queue.
In this embodiment, step 401 is substantially identical to step 201 in the corresponding embodiment of fig. 2, and will not be described herein.
Step 402, in response to the target log list corresponding to the target log queue being empty, inserting the acquired log into the target log list corresponding to the target log queue, and recording an insertion time of the acquired log.
In the present embodiment, in the case where the target log list corresponding to the target log queue is empty, the execution subject of the log consuming method (e.g., the server or the terminal device shown in fig. 1) may insert the acquired log into the target log list corresponding to the target log queue and record the insertion time of the acquired log.
Step 403, merging the first number of logs in the target log list corresponding to the target log queue to obtain a second number of merged logs in response to the acquired logs meeting the preset merging condition.
In this embodiment, step 403 is substantially identical to step 202 in the corresponding embodiment of fig. 2, and will not be described herein.
And step 404, consuming the log after the second number of the merged stripes.
In this embodiment, step 404 is substantially identical to step 203 in the corresponding embodiment of fig. 2, and will not be described herein.
It should be noted that, in addition to the above, the embodiments of the present disclosure may further include the same or similar features and effects as those of the embodiment corresponding to fig. 2, which are not described herein.
As can be seen from fig. 4, in the case that the target log list corresponding to the target log queue is empty, the flow 400 of the log consuming method in this embodiment may insert the obtained log into the target log list corresponding to the target log queue, and record the insertion time of the obtained log, so as to obtain the time when the target log list is inserted into the first log, which is helpful for realizing more timely log consumption according to the insertion time.
As an example, please refer to fig. 5, fig. 5 is a flowchart for one application scenario of fig. 4. The application scenario process 500 includes the following steps:
step 501, a log pointed by a queue head pointer of a target log queue is obtained. Thereafter, step 502 is performed.
In this embodiment, in a case where a target log list corresponding to a target log queue is empty, an execution subject of the log consuming method (for example, a server or a terminal device shown in fig. 1) may acquire a log pointed to by a head pointer of the target log queue.
Step 502, determining whether a target log list corresponding to the target log queue is empty. Then, if the target log list corresponding to the target log queue is empty, executing step 504; if the target log list corresponding to the target log queue is not empty, step 503 is performed.
In this embodiment, the execution body may determine whether a target log list corresponding to the target log queue is empty.
Step 503, determining whether a time interval between an acquisition time of the acquired log and an insertion time of a first inserted log included in the target log list is greater than or equal to a preset time interval. Then, if the time interval between the acquired time of the acquired log and the insertion time of the first inserted log included in the target log list is greater than or equal to the preset time interval, executing step 507; if the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval, step 505 is executed.
In this embodiment, in the case where it is determined in step 502 that the target log list corresponding to the target log queue is not empty, the execution body may determine whether a time interval between an acquisition time of the acquired log and an insertion time of a first inserted log included in the target log list is greater than or equal to a preset time interval.
Step 504, the obtained log is inserted into a target log list corresponding to the target log queue, and the insertion time of the obtained log is recorded.
In this embodiment, in the case where it is determined in step 502 that the target log list corresponding to the target log queue is empty, the execution body may insert the acquired log into the target log list corresponding to the target log queue, and record the insertion time of the acquired log.
In step 505, it is determined whether the operation information and table information included in the log pointed to by the tail pointer included in the target log list and the operation information and table information included in the acquired log indicate the same operation and table. Then, if the operation information and table information included in the log pointed by the tail pointer included in the target log list are the same as the operation information and table information included in the acquired log, executing step 506; if the operation information and table information included in the log pointed to by the tail pointer included in the target log list are not indicative of the same operation and table as the operation information and table information included in the acquired log, step 507 is performed.
In this embodiment, in the case where it is determined in step 502 that the target log list corresponding to the target log queue is not empty, and where the time interval between the acquisition time of the log acquired in step 503 and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval, the execution body may determine whether the operation information and the table information included in the log pointed to by the tail pointer included in the target log list and the operation information and the table information included in the acquired log indicate the same operation and table.
Step 506, the obtained log is inserted into a target log list corresponding to the target log queue. Thereafter, step 507 is performed.
In this embodiment, it is determined in step 502 that the target log list corresponding to the target log queue is not empty; the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list in step 503 is smaller than a preset time interval, and the execution body may insert the acquired log into the target log list corresponding to the target log queue in a case where the operation information and the table information included in the log pointed by the column tail pointer included in the target log list in step 505 indicate the same operation and table as the operation information and the table information included in the acquired log, respectively.
And step 507, merging all the logs included in the target log list into a group of logs to obtain a merged log. Thereafter, step 508 is performed.
In this embodiment, the execution body may merge each log included in the target log list into a group of logs, to obtain a merged log. Wherein, a group of logs obtained after the combination is consumed will produce the same result as the first number of logs are consumed. As an example, the first number of logs may call 1000 times a method (function, also called function) for subtracting 1 from the current value. After back-end processing, the combined set of logs may be used to invoke 1 method for subtracting 1000 from the current value. Illustratively, the backend may generate 1 piece of processing information from a set of logs obtained after the merging. This process information may call 1 method for subtracting 1000 from the current value.
And 508, consuming the combined logs, and deleting each log included in the target log list from the target log list.
In this embodiment, the execution body may consume the combined log, and delete each log included in the target log list from the target log list.
In some optional implementations of this embodiment, after performing step 505, the execution body may further perform steps 507, 508, and 504 at a time.
It should be noted that, in addition to the above, the embodiments of the present disclosure may further include the same or similar features and effects as those of the embodiment corresponding to fig. 2, which are not described herein.
It is understood that the flow 500 of the log consumption method in this embodiment may accelerate log consumption by merging logs. And moreover, ordered log consumption and no loss can be ensured. And controlling the throughput of log consumption by adjusting a preset time interval.
With further reference to fig. 6, as an implementation of the method shown in the foregoing figures, the present disclosure provides an embodiment of a log consuming device, which corresponds to the method embodiment shown in fig. 2, and which may include the same or corresponding features as the method embodiment shown in fig. 2, and produces the same or corresponding effects as the method embodiment shown in fig. 2, in addition to the features described below. The device can be applied to various electronic equipment.
As shown in fig. 6, the log consuming apparatus 600 of the present embodiment includes: an obtaining unit 601 configured to obtain a log pointed by a head pointer of a target log queue, where the log in the target log queue is information for recording a line change operation on a target database; a merging unit 602, configured to, in response to the acquired logs meeting a preset merging condition, merge a first number of logs in a target log list corresponding to a target log queue to obtain a second number of merged logs, where the first number is greater than or equal to 2, and the second number is less than the first number; the consumption unit 603 is configured to consume the second number of combined logs.
In the present embodiment, the acquisition unit 601 of the log consuming apparatus 600 may acquire the log pointed to by the head pointer of the target log queue. The log in the target log queue is information for recording line change operation on the target database.
In this embodiment, in the case where the log acquired by the acquiring unit 601 meets the preset merging condition, the merging unit 602 may merge the first number of logs in the target log list corresponding to the target log queue to obtain the second number of merged logs. Wherein the first number is greater than or equal to 2 and the second number is less than the first number.
In this embodiment, the consumption unit 603 may consume the log obtained by the merging unit 602 after merging the second number of strips.
In some optional implementations of this embodiment, the log includes operation information indicating an operation performed on the data in the target database, and table information indicating a table including the operated data, the operation information included in each log included in the target log list indicating the same operation, and the table information included in each log included in the target log list indicating the same table.
In some optional implementations of this embodiment, the preset combining conditions include: the target log list corresponding to the target log queue is not empty, and one of: the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is larger than or equal to a preset time interval; the time difference between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than a preset time interval; and, the operation information or table information included in the log pointed by the column tail pointer included in the target log list and the operation information or table information included in the acquired log indicate different operations or tables.
In some optional implementations of the present embodiment, the merging unit 602 includes: a merging subunit (not shown in the figure) configured to merge the respective logs included in the target log list into a set of logs.
In some optional implementations of this embodiment, the apparatus 600 further includes: a first inserting unit (not shown in the figure) configured to insert the acquired log into the target log list corresponding to the target log queue in response to the target log list corresponding to the target log queue being empty, and record an insertion time of the acquired log.
In some optional implementations of this embodiment, the apparatus 600 further includes: a second inserting unit (not shown in the figure) configured to insert the acquired log into a target log list corresponding to the target log queue in response to satisfaction of a preset insertion condition. Wherein, the preset insertion condition includes: the target log list corresponding to the target log queue is not empty; the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than a preset time interval; the operation information included in the log pointed by the tail pointer included in the target log list and the operation information included in the acquired log indicate the same operation; the list tail pointer included in the target log list points to the same table as the table information included in the acquired log.
In some optional implementations of this embodiment, the apparatus 600 further includes: a first deleting unit (not shown in the figure) configured to delete, from the target log queue, the acquired log pointed to by the head pointer of the target log queue.
In some optional implementations of this embodiment, the apparatus 600 further includes: a second deleting unit (not shown in the figure) configured to delete, from the target log list, the consumed log included in the target log list in response to completion of consumption of the second number of merged logs.
According to the device provided by the embodiment of the present disclosure, the obtaining unit 601 obtains the log pointed by the queue head pointer of the target log queue, where the log in the target log queue is used to record the information for performing the line change operation on the target database, then, if the obtained log meets the preset merging condition, the merging unit 602 merges the first number of logs in the target log list corresponding to the target log queue to obtain the second number of merged logs, where the first number is greater than or equal to 2, the second number is less than the first number, and finally, the consumption unit 603 consumes the second number of merged logs, thereby reducing the number of logs required to be consumed, improving the consumption speed of the logs, and reducing the pressure of the electronic device for log consumption. And the back end can directly consume the combined logs, thereby reducing a great amount of network request time delay.
Referring now to fig. 7, a schematic diagram of an electronic device (e.g., server or terminal device of fig. 1) 700 suitable for use in implementing embodiments of the present disclosure is shown. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), car terminals (e.g., car navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The terminal device/server illustrated in fig. 7 is merely an example, and should not impose any limitation on the functionality and scope of use of embodiments of the present disclosure.
As shown in fig. 7, the electronic device 700 may include a processing means (e.g., a central processor, a graphics processor, etc.) 701, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage means 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
In general, the following devices may be connected to the I/O interface 705: input devices 706 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 707 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 708 including, for example, magnetic tape, hard disk, etc.; and a communication device 709. The communication means 709 may allow the electronic device 700 to communicate wirelessly or by wire with other devices to exchange data. While fig. 7 shows an electronic device 700 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead. Each block shown in fig. 7 may represent one device or a plurality of devices as needed.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via communication device 709, or installed from storage 708, or installed from ROM 702. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing device 701.
It should be noted that, the computer readable medium according to the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In an embodiment of the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. Whereas in embodiments of the present disclosure, the computer-readable signal medium may comprise a data signal propagated in baseband or as part of a carrier wave, with computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring a log pointed by a queue head pointer of a target log queue, wherein the log in the target log queue is used for recording information for performing line change operation on a target database; in response to the acquired logs meeting preset merging conditions, merging a first number of logs in a target log list corresponding to a target log queue to obtain a second number of merged logs, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number; and (5) consuming the log after merging the second number of the bars.
Computer program code for carrying out operations of embodiments of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments described in the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The described units may also be provided in a processor, for example, described as: a processor includes an acquisition unit, a merging unit, and a consumption unit. Where the names of these units do not constitute a limitation on the unit itself in some cases, for example, the acquisition unit may also be described as "a unit that acquires a log pointed to by the head pointer of the target log queue".
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above technical features, but encompasses other technical features formed by any combination of the above technical features or their equivalents without departing from the spirit of the invention. Such as the above-described features, are mutually substituted with (but not limited to) the features having similar functions disclosed in the embodiments of the present disclosure.
Claims (9)
1. A log consumption method, comprising:
Acquiring a log pointed by a queue head pointer of a target log queue, wherein the log in the target log queue is used for recording information for performing line change operation on a target database;
In response to the acquired logs meeting preset merging conditions, merging a first number of logs in a target log list corresponding to the target log queue to obtain a second number of merged logs, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number;
Consuming the log after the second number of the strips are combined;
The logs comprise operation information and table information, the operation information indicates operation on data in the target database, the table information indicates a table comprising the operated data, the operation information included in each log included in the target log list indicates the same operation, and the table information included in each log included in the target log list indicates the same table;
The preset merging conditions comprise: the target log list corresponding to the target log queue is not empty, and one of:
The time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is larger than or equal to a preset time interval;
The time difference between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval; and, the operation information or table information included in the log pointed by the column tail pointer included in the target log list and the operation information or table information included in the acquired log indicate different operations or tables.
2. The method of claim 1, wherein the merging the first number of logs in the target log list corresponding to the target log queue comprises:
and merging the logs included in the target log list into a group of logs.
3. The method of claim 1 or 2, wherein, before merging a first number of logs in a target log list corresponding to the target log queue in response to the acquired log meeting a preset merge condition, to obtain a second number of merged logs, the method further comprises:
In response to the target log list corresponding to the target log queue being empty, inserting the acquired log into the target log list corresponding to the target log queue, and recording an insertion time of the acquired log.
4. The method of claim 1 or 2, wherein, before merging a first number of logs in a target log list corresponding to the target log queue in response to the acquired log meeting a preset merge condition, to obtain a second number of merged logs, the method further comprises:
in response to a preset insertion condition being met, inserting the acquired log into a target log list corresponding to the target log queue;
wherein the preset insertion conditions include:
The target log list corresponding to the target log queue is not empty;
the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than a preset time interval;
the operation information included in the log pointed by the tail pointer included in the target log list and the operation information included in the acquired log indicate the same operation;
The list tail pointer included in the target log list points to the same table as the table information included in the acquired log.
5. The method of claim 1 or 2, wherein after obtaining the log pointed to by the head pointer of the target log queue, the method further comprises:
And deleting the acquired log pointed by the queue head pointer of the target log queue from the target log queue.
6. The method of claim 1 or 2, wherein after said consuming the second number of consolidated logs, the method further comprises:
And deleting the consumed logs included in the target log list from the target log list in response to completing the consumption of the second number of combined logs.
7. A log consuming device, comprising:
An acquisition unit configured to acquire a log pointed by a head pointer of a target log queue, wherein the log in the target log queue is information for recording line change operation on a target database;
The merging unit is configured to respond to the fact that the acquired logs meet preset merging conditions, merge a first number of logs in a target log list corresponding to the target log queue to obtain a second number of merged logs, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number;
the consumption unit is configured to consume the log after the second number of the strips are combined;
The method comprises the steps that a log comprises operation information and table information, the operation information indicates operation on data in a target database, the table information indicates a table comprising the operated data, the operation information included in each log included in a target log list indicates the same operation, and the table information included in each log included in the target log list indicates the same table;
The preset merging conditions comprise: the target log list corresponding to the target log queue is not empty, and one of:
The time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is larger than or equal to a preset time interval;
The time difference between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval; and, the operation information or table information included in the log pointed by the column tail pointer included in the target log list and the operation information or table information included in the acquired log indicate different operations or tables.
8. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
When executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-6.
9. A computer readable medium having stored thereon a computer program, wherein the program when executed by a processor implements the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010396856.0A CN113656645B (en) | 2020-05-12 | 2020-05-12 | Log consumption method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010396856.0A CN113656645B (en) | 2020-05-12 | 2020-05-12 | Log consumption method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113656645A CN113656645A (en) | 2021-11-16 |
| CN113656645B true CN113656645B (en) | 2024-08-20 |
Family
ID=78476861
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010396856.0A Active CN113656645B (en) | 2020-05-12 | 2020-05-12 | Log consumption method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113656645B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101605028A (en) * | 2009-02-17 | 2009-12-16 | 北京安天电子设备有限公司 | A kind of combining log records method and system |
| CN101625703A (en) * | 2009-08-21 | 2010-01-13 | 华中科技大学 | Method and system for merging logs of memory database |
| CN108984686A (en) * | 2018-07-02 | 2018-12-11 | 中国电子科技集团公司第五十二研究所 | A kind of distributed file system indexing means and device merged based on log |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101557291B (en) * | 2009-05-25 | 2011-09-28 | 杭州华三通信技术有限公司 | Method for log aggregation and device thereof |
| CN109992469B (en) * | 2017-12-29 | 2023-08-18 | 三六零科技集团有限公司 | Method and device for merging logs |
-
2020
- 2020-05-12 CN CN202010396856.0A patent/CN113656645B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101605028A (en) * | 2009-02-17 | 2009-12-16 | 北京安天电子设备有限公司 | A kind of combining log records method and system |
| CN101625703A (en) * | 2009-08-21 | 2010-01-13 | 华中科技大学 | Method and system for merging logs of memory database |
| CN108984686A (en) * | 2018-07-02 | 2018-12-11 | 中国电子科技集团公司第五十二研究所 | A kind of distributed file system indexing means and device merged based on log |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113656645A (en) | 2021-11-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110781373B (en) | List updating method and device, readable medium and electronic equipment | |
| CN111427706A (en) | Data processing method, multi-server system, database, electronic device, and storage medium | |
| CN111240564B (en) | Material display method and device, electronic equipment and storage medium | |
| CN110673959A (en) | System, method and apparatus for processing tasks | |
| CN111163336A (en) | Video resource pushing method and device, electronic equipment and computer readable medium | |
| CN112379982B (en) | Task processing method, device, electronic equipment and computer readable storage medium | |
| CN114116247A (en) | Redis-based message processing method, device, system, server and medium | |
| CN111225255B (en) | Target video push playing method and device, electronic equipment and storage medium | |
| CN112148744A (en) | Page display method and device, electronic equipment and computer readable medium | |
| CN113656645B (en) | Log consumption method and device | |
| CN112685075A (en) | Gray scale distribution method and device, electronic equipment and computer readable medium | |
| CN112291325A (en) | Message processing method and device and computer system | |
| CN111596992A (en) | Navigation bar display method and device and electronic equipment | |
| CN110619079B (en) | Method and device for pushing information | |
| CN115454666A (en) | Data synchronization method and device among message queue clusters | |
| CN111367592B (en) | Information processing method and device | |
| CN112163176A (en) | Data storage method and device, electronic equipment and computer readable medium | |
| CN112115154A (en) | Data processing and data query method, device, equipment and computer readable medium | |
| CN111291254A (en) | Information processing method and device | |
| CN112163399A (en) | Online document pushing method and device, electronic equipment and computer readable medium | |
| CN112799863A (en) | Method and apparatus for outputting information | |
| CN111294321B (en) | Information processing method and device | |
| CN117692672B (en) | Snapshot-based video information sending method and device, electronic equipment and medium | |
| CN113779082B (en) | Method and device for updating data | |
| CN113778387B (en) | Method and device for generating code |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |