CN118193547A - Log management method, log management system and storage medium - Google Patents
Log management method, log management system and storage medium Download PDFInfo
- Publication number
- CN118193547A CN118193547A CN202410182836.1A CN202410182836A CN118193547A CN 118193547 A CN118193547 A CN 118193547A CN 202410182836 A CN202410182836 A CN 202410182836A CN 118193547 A CN118193547 A CN 118193547A
- Authority
- CN
- China
- Prior art keywords
- log
- target service
- service
- server cluster
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 75
- 238000003860 storage Methods 0.000 title claims abstract description 23
- 238000012545 processing Methods 0.000 claims description 82
- 238000013500 data storage Methods 0.000 claims description 41
- 238000000034 method Methods 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 7
- 238000012423 maintenance Methods 0.000 abstract description 5
- 230000005540 biological transmission Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000002159 abnormal effect Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000012552 review Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
- G06F16/2322—Optimistic concurrency control using timestamps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Security & Cryptography (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to a log management method, a log management system and a storage medium, which are used for acquiring a target service log of a server cluster sent by log acquisition equipment; the target service log is acquired by log acquisition equipment based on time variables corresponding to all service servers in the server cluster; determining whether the target service log is a duplicate log; if not, storing the target service log, and sending the log printing time of the target service log to the log acquisition equipment, so that the log acquisition equipment updates the time variable corresponding to each service server in the server cluster according to the log printing time. The consistency and the integrity of the service logs of the acquired server cluster are ensured, repeated collection and missing collection of the service logs by the log collection equipment are avoided, the log management efficiency is improved under the condition of not intervening in a third-party system component, the operation and maintenance burden is reduced, repeated storage of log files can be avoided, and log management resources are saved.
Description
Technical Field
The present application relates to the field of log management, and in particular, to a log management method, a log management system, and a storage medium.
Background
The log management is one of important works of enterprises or organization departments, the log is an important reference basis for problems in a system or application, and the log management can analyze, store and backup business records generated in the working process of the enterprises or departments, assist the enterprises to find potential safety hazards, timely take countermeasures for the potential safety hazards and guarantee the safety of the enterprises.
At present, an enterprise-level log system architecture ELK (Elasticsearch Logstash Kibana) is often adopted to realize log management, and ELK can be used for centralized storage, processing and visualization of log data, but ELK mainly focuses on collection, indexing and display of logs, and no built-in mechanism is provided for guaranteeing consistency of collected logs and stored logs, and data delay problem exists, so that integrity of stored logs cannot be guaranteed. The adoption of ELK for log management requires regular maintenance of the log management system, including monitoring the running state of each component, optimizing query performance, backing up and recovering data, etc., and for smaller teams and environments with limited resources, additional time and labor may be required to manage and maintain the whole log scheme, the log management cost is high, and the ELK requires a large amount of computing resources and storage space to support high-performance data retrieval and query, and has the disadvantages of high resource consumption and high running cost. Therefore, how to ensure the consistency and the integrity of the log files stored in the log management process, save the log management cost and improve the log management efficiency is a problem to be solved.
Disclosure of Invention
Based on the above, it is necessary to provide a log management method, a log management system and a storage medium capable of ensuring consistency and integrity of log files stored in a log management process, saving log management cost and improving log management efficiency.
In a first aspect, the present application provides a log management method, performed by a log processing apparatus, the method comprising:
Acquiring a target service log of a server cluster sent by log acquisition equipment; the target service log is acquired by the log acquisition equipment based on time variables corresponding to all service servers in the server cluster;
Determining whether the target service log is a duplicate log;
If not, storing the target service log, and sending the log printing time of the target service log to the log acquisition equipment, so that the log acquisition equipment updates the time variable corresponding to each service server in the server cluster according to the log printing time.
In one embodiment, determining whether the target service log is a duplicate log comprises:
determining a target hash value of the target service log;
and determining whether the target service log is a repeated log according to the target hash value and the historical hash value of the historical service log stored in the buffer data storage file.
In one embodiment, storing the target service log includes:
And storing the target service log into a log data storage file, and storing the target hash value into a cache data storage file.
In one embodiment, after determining whether the target service log is a repeated log, the method further includes:
if yes, the target service log is ignored.
In a second aspect, the present application further provides a log management method, which is executed by a log collection device, the method including:
Acquiring a target service log of a server cluster based on time variables corresponding to each service server in the server cluster, and sending the target service log to log processing equipment so that the log processing equipment stores the target service log;
and acquiring the log printing time of the target service log fed back by the log processing equipment, and updating time variables corresponding to each service server in the server cluster according to the log printing time.
In one embodiment, collecting a target service log of a server cluster according to time variables corresponding to service servers in the server cluster, and sending the target service log to a log processing device, so that the log processing device stores the target service log, and then further includes:
determining the sending time length of the target service log according to the sending time stamp of the target service log;
If the sending time length is greater than a time length threshold value and the log printing time fed back by the log processing equipment is not received, the connection with the log processing equipment is reestablished, and the target service log is sent to the log processing equipment again.
In a third aspect, the present application further provides a log management system, including a log processing device and a log collecting device, where the log processing device is connected to the log collecting device, where the log collecting device is configured to collect a target service log of a server cluster according to time variables corresponding to each service server in the server cluster, and send the target service log to the log processing device, and the log processing device is configured to obtain the target service log of the server cluster sent by the log collecting device; determining whether the target service log is a duplicate log; if not, storing the target service log, and sending the log printing time of the target service log to the log acquisition equipment.
In one embodiment, the log collection device in the log management system is further configured to collect a target service log of the server cluster based on time variables corresponding to each service server in the server cluster, and send the target service log to the log processing device; and acquiring the log printing time of the target service log fed back by the log processing equipment, and updating time variables corresponding to each service server in the server cluster according to the log printing time.
In one embodiment, the log collection device in the log management system further includes a log data storage module and a cache data storage module, where the log data storage module is configured to store the target service log in a log data storage file, and the cache data storage module is configured to store the target hash value of the target service log in the cache data storage file.
In a fourth aspect, the present application also provides a computer readable storage medium, where a computer program is stored, where the computer program is executed by a processor to implement the log management method according to any one of the embodiments of the present application.
The log management method, the log management system and the storage medium acquire the target service log of the server cluster sent by the log acquisition equipment; determining whether the target service log is a duplicate log; if not, storing the target service log, and sending the log printing time of the target service log to the log acquisition equipment, so that the log acquisition equipment updates the time variable corresponding to each service server in the server cluster according to the log printing time. The method solves the problems that when log management is carried out through a log system architecture, the log management cost is high, the resource consumption is high, and the integrity and consistency of the stored log cannot be ensured. According to the scheme, after the target service log sent by the log acquisition equipment is acquired, the target service log is subjected to repeatability test, the target service log is stored when the target service log is determined not to be the repeated log, and after the target service log is successfully stored, the log printing time of the target service log is returned to the log acquisition equipment, so that the target service log can execute the task of acquiring the service log of the server cluster in the next round based on the log printing time, the consistency and the integrity of the acquired service log of the server cluster are ensured, the repeated acquisition and the missed acquisition of the service log by the log acquisition equipment are avoided, the log management efficiency is improved under the condition that a third-party system component is not involved, the operation and maintenance burden is reduced, the repeated storage of log files is avoided, and the log management resources are saved.
Drawings
FIG. 1 is a diagram of an application environment for a method of log management in one embodiment;
FIG. 2 is a flow chart of a log management method according to an embodiment;
FIG. 3 is a flowchart illustrating a log management method according to another embodiment;
FIG. 4 is a flowchart of a log management method according to another embodiment;
FIG. 5 is a signaling diagram of a log management method according to one embodiment;
FIG. 6 is a flowchart of a log management method according to another embodiment;
FIG. 7 is a diagram showing an example of the structure of a log management system according to one embodiment;
FIG. 8 is a diagram showing an example of the structure of a log management system according to another embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
The log management method provided by the embodiment of the application can be applied to an application environment shown in figure 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on a cloud or other network server. The server 104 acquires a target service log of a server cluster sent by log acquisition equipment; the target service log is acquired by log acquisition equipment based on time variables corresponding to all service servers in the server cluster; determining whether the target service log is a duplicate log; if not, storing the target service log, and sending the log printing time of the target service log to the log collecting device, so that the log collecting device updates the time variable corresponding to each service server in the server cluster according to the log printing time, and can send the target service log to the terminal 102 through the communication network. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, where the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server 104 may be implemented as a stand-alone server or as a server cluster of multiple servers.
In one embodiment, as shown in fig. 2, a log management method is provided, which is performed by a log processing device. In this embodiment, the method includes the steps of:
S210, acquiring a target service log of a server cluster sent by log acquisition equipment.
The target service log is acquired by log acquisition equipment based on time variables corresponding to all service servers in the server cluster.
The server cluster comprises at least two service servers, and the service servers can generate service logs according to the transacted services. The log collection device refers to a device for collecting service logs generated by service servers in a server cluster. The time variable refers to the time when the service log corresponding to the service server was last stored successfully. It should be noted that, the log collection device stores a time variable SINCETIME corresponding to each service server.
Specifically, the log collection device establishes connection with service servers in the server cluster, determines offset cursors of each service server according to time variables corresponding to each service server through the log collection device, collects service logs corresponding to each service server in the server cluster according to the offset cursors of each service server, takes the collected service logs corresponding to the service servers as target service logs of the server cluster, and the log processing device obtains the target service logs of the server cluster sent by the log collection device.
S220, determining whether the target service log is a repeated log.
It should be noted that, when the time variable stored in the log collecting device is not updated timely, or the log processing device feeds back the error log printing time to the log collecting device, the target service log collected by the log collecting device may include a repeated log, so that the received target service log needs to be subjected to deduplication.
Specifically, the log content of the target service log may be compared with the log content of the history service log stored in the log storage device, if the content similarity of the target service log and the history service log is higher than a preset similarity threshold, the target service log is determined to be a repeated log, otherwise, the target service log is determined not to be a repeated log.
And S230, if not, storing the target service log, and sending the log printing time of the target service log to the log acquisition equipment so that the log acquisition equipment updates the time variable corresponding to each service server in the server cluster according to the log printing time.
The log printing time of the target service log refers to the generation time of the target service log. It should be noted that, the log processing device and the log collecting device are connected by adopting a gRPC bidirectional flow mode connection mode, so that the log printing time and the data integrity of the target service log in the transmission process can be ensured, and the data loss is avoided.
Specifically, if the target service log is not the duplicate log, storing the target service log in the log storage device, and determining the log printing time of the target service log. Based on a preset TCP (Transmission Control Protocol, transmission control) protocol for carrying out log transmission with log processing equipment, the log printing time of the target service log is sent to log acquisition equipment, so that the log acquisition equipment updates time variables corresponding to all service servers in the server cluster based on the log printing time, and when the log acquisition equipment acquires the service log of the server cluster in the next round, the log acquisition equipment can acquire the service log of the server cluster based on the updated time variables corresponding to all service servers.
In the log management method, a target service log of a server cluster sent by log acquisition equipment is obtained; determining whether the target service log is a duplicate log; if not, storing the target service log, and sending the log printing time of the target service log to the log acquisition equipment, so that the log acquisition equipment updates the time variable corresponding to each service server in the server cluster according to the log printing time. The method solves the problems that when log management is carried out through a log system architecture, the log management cost is high, the resource consumption is high, and the integrity and consistency of the stored log cannot be ensured. According to the scheme, after the target service log sent by the log acquisition equipment is acquired, the target service log is subjected to repeatability test, the target service log is stored when the target service log is determined not to be the repeated log, and after the target service log is successfully stored, the log printing time of the target service log is returned to the log acquisition equipment, so that the target service log can execute the task of acquiring the service log of the server cluster in the next round based on the log printing time, the consistency and the integrity of the acquired service log of the server cluster are ensured, the repeated acquisition and the missed acquisition of the service log by the log acquisition equipment are avoided, the log management efficiency is improved under the condition that a third-party system component is not involved, the operation and maintenance burden is reduced, the repeated storage of log files is avoided, and the log management resources are saved.
Illustratively, after determining whether the target service log is a duplicate log, further comprising:
if yes, the target service log is ignored.
If the target service log is a repeated log, the target service log is already stored in the log storage device, and the target service log needs to be ignored at the moment and the next service log sent by the log acquisition device is continuously received. The method and the system avoid repeated storage of the target service log, avoid the problem of system resource waste and improve log management efficiency.
In one embodiment, determining whether the target service log is a duplicate log comprises:
Determining a target hash value of a target service log; and determining whether the target service log is a repeated log according to the target hash value and the historical hash value of the historical service log stored in the buffer data storage file.
The cache data storage file refers to a storage device for storing a history hash value of a history service log. The target hash value of the target service log may be an MD5 value of the target service log, and the historical hash value of the historical service log is the MD5 value of the historical service log.
Specifically, the MD5 value of the target service log is calculated according to the log content of the target service log. And comparing the MD5 value of the target service log with the MD5 value of the history service log stored in the cache data storage file in a consistency mode, and determining whether the MD5 value of the target service log is consistent with the MD5 value of the history service log. If yes, determining that the target service log is a repeated log, and if not, determining that the target service log is not the repeated log.
According to the consistency comparison result of the target hash value of the target service log and the historical hash value of the historical service log stored in the buffer data storage file, whether the target service log is a repeated log is determined, so that the identification efficiency of whether the target service log is the repeated log can be improved, and the accuracy of repeated detection of the target service log can be improved.
Illustratively, storing the target service log includes:
And storing the target service log into a log data storage file, and storing the target hash value into a cache data storage file.
Specifically, if it is determined that the target service log is not a duplicate log according to the comparison result of the target hash value of the target service log and the historical hash value of the historical service log stored in the cache data storage file, the target service log is stored in the log data storage file, so that a subsequent worker can call the target service log from the log data storage file to review, and meanwhile, the target hash value of the target service log is stored in the cache data storage file as a new historical hash value.
According to the scheme, when the target service log is determined not to be the repeated log, the target hash value of the target service log is stored in the cache data storage file, so that the cache data can be enriched, and the accuracy of subsequent repeated detection of the acquired service log can be improved.
In one embodiment, as shown in fig. 3, a log management method is provided, which is performed by a log collection device. In this embodiment, the log management method includes the following steps:
s310, collecting target service logs of the server cluster based on time variables corresponding to all service servers in the server cluster, and sending the target service logs to log processing equipment so that the log processing equipment stores the target service logs.
Specifically, connection is established with each service server in the server cluster, service logs of each service server are respectively collected based on time variables corresponding to each service server in the stored server cluster, and the collected service logs of each service server are used as target service logs of the server cluster. And transmitting the target service log of the server cluster to the log processing equipment based on a preset TCP protocol for log transmission with the log processing equipment, so that the log processing equipment analyzes and stores the target service log. The method for analyzing and storing the target service log by the log processing device is consistent with the log management method executed by the log processing device, and will not be described herein.
S320, acquiring the log printing time of the target service log fed back by the log processing equipment, and updating the time variable corresponding to each service server in the server cluster according to the log printing time.
Specifically, after the log processing device completes storage of the target service log, determining the log printing time of the target service log from the target service log, and after the log collecting device obtains the log printing time fed back by the log processing device, updating the time variable corresponding to each service server in the server cluster according to the log printing time corresponding to each service server, so as to determine an offset cursor for collecting the service log of each service server based on the updated time variable, thereby ensuring the integrity of the collected service log of each service server.
According to the log management method, the target service logs of the server cluster are collected based on the time variable corresponding to each service server in the server cluster, and the target service logs are sent to the log processing equipment, so that the log processing equipment stores the target service logs; and acquiring the log printing time of the target service log fed back by the log processing equipment, and updating the time variable corresponding to each service server in the server cluster according to the log printing time. The method and the system can collect service logs of each service server according to the time variable corresponding to each service server, update the stored time variable corresponding to each service server according to the log printing time fed back by the log processing equipment, avoid repeated collection of the service logs and missed collection of the service logs, ensure consistency and integrity of the stored log files and improve log management efficiency.
In one embodiment, as shown in fig. 4, the method includes collecting a target service log of a server cluster according to time variables corresponding to each service server in the server cluster, and sending the target service log to a log processing device, so that after the log processing device stores the target service log, further includes:
s410, determining the sending duration of the target service log according to the sending time stamp of the target service log.
Specifically, when the target service log is transmitted to the log processing apparatus, a transmission time stamp of the target service log is recorded, and a transmission time period for transmitting the target service log to the log processing apparatus is determined based on the transmission time stamp.
And S420, if the sending time length is greater than the time length threshold value and the log printing time fed back by the log processing equipment is not received, reestablishing the connection with the log processing equipment and resending the target service log to the log processing equipment.
The time length threshold can be set according to actual needs.
Specifically, if the sending time of the target service log is longer than the preset time threshold, the log printing time fed back by the log processing device cannot be received, so that the log collecting device can be considered to be abnormal, the target service log cannot be successfully sent to the log processing device, at the moment, the log collecting device firstly disconnects the log processing device, then reestablishes the connection with the log processing device, and after reestablishing the connection, the target service log is sent to the log processing device again.
According to the scheme, under the condition that the log acquisition equipment is abnormal, the abnormality can be found in time according to the sending time of the target service log, connection is established with the log processing equipment again, the target service log is sent to the log processing equipment again, the problem of log missing is avoided, and the integrity of the service log is further guaranteed.
As shown in fig. 5, the log management method includes, based on the above embodiment:
The method comprises the steps that a log collection device is connected with each service server in a server cluster, an offset cursor for collecting service logs of each service server in the last round is determined based on a time variable corresponding to each service server in the stored server cluster, the service logs of each service server are collected based on the offset cursor for collecting the service logs of each service server in the last round, and the collected service logs of each service server are used as target service logs of the server cluster. After the log acquisition device acquires the target service log, the target service log of the server cluster is sent to the log processing device based on a preset TCP protocol for log transmission with the log processing device. If the log collection device does not search the time variable corresponding to a certain service server, initializing the time variable of the service server which does not search the time variable, for example, the time variable of the service server which does not search the time variable can be initialized to 1970-01-00:00:00.
When the target service log is transmitted to the log processing device, a transmission time stamp of the target service log is recorded, and a transmission time period for transmitting the target service log to the log processing device is determined based on the transmission time stamp. If the sending time length of the target service log is longer than the preset time length threshold, the log printing time fed back by the log processing equipment cannot be received, and the log acquisition equipment can be considered to be abnormal, so that the target service log cannot be successfully sent to the log processing equipment.
After the log processing device obtains the target service log of the server cluster sent by the log collecting device, whether the target service log is a repeated log needs to be determined according to the target hash value of the target service log and the history hash value of the history service log. The process of detecting the repeatability of the target service log is shown in fig. 6, specifically, the MD5 value of the target service log is calculated according to the log content of the target service log, the MD5 value of the target service log and the MD5 value of the history service log stored in the cache data storage file are compared in consistency, and whether the MD5 value of the target service log is consistent with the MD5 value of the history service log is determined. If yes, determining that the target service log is a repeated log, and if not, determining that the target service log is not the repeated log.
If the target service log is a repeated log, the log processing device ignores the target service log and continuously receives the service log of the server cluster sent by the log acquisition device. If the target service log is not the repeated log, storing the target service log into a log data storage file so that a subsequent worker can conveniently call the target service log from the log data storage file to review, storing the target hash value of the target service log into a cache data storage file as a new historical hash value, extracting log printing time from the target service log, and sending the log printing time of the target service log to log acquisition equipment.
After the log acquisition equipment acquires the log printing time fed back by the log processing equipment, the time variable corresponding to each service server in the server cluster is updated according to the log printing time corresponding to each service server, so that an offset cursor for acquiring the service log of each service server is determined based on the updated time variable, and the integrity of the acquired service log of each service server is ensured.
According to the log management method, after the target service log sent by the log acquisition equipment is acquired, the target service log is subjected to repeatability test, the target service log is stored when the target service log is not the repeated log, and after the target service log is successfully stored, the log printing time of the target service log is returned to the log acquisition equipment, so that the target service log can execute the task of acquiring the service log of the server cluster in the next round based on the log printing time, the consistency and the integrity of the acquired service log of the server cluster are ensured, the repeated acquisition and the missed acquisition of the service log by the log acquisition equipment are avoided, the log management efficiency is improved under the condition that a third-party system component is not involved, the operation and maintenance burden is reduced, the repeated storage of log files is avoided, and the log management resources are saved.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
In an embodiment of the present application, there is further provided a log management system, fig. 7 is a schematic structural diagram of the log management system in this embodiment, and as shown in fig. 7, the log management system includes: the system comprises log processing equipment and log acquisition equipment, wherein the log processing equipment is connected with the log acquisition equipment, the log acquisition equipment is used for acquiring target service logs of a server cluster according to time variables corresponding to service servers in the server cluster and sending the target service logs to the log processing equipment, and the log processing equipment is used for acquiring the target service logs of the server cluster sent by the log acquisition equipment; determining whether the target service log is a duplicate log; if not, storing the target service log, and sending the log printing time of the target service log to the log acquisition equipment.
In one embodiment, the log collection device is further configured to collect a target service log of the server cluster based on time variables corresponding to each service server in the server cluster, and send the target service log to the log processing device; and acquiring the log printing time of the target service log fed back by the log processing equipment, and updating the time variable corresponding to each service server in the server cluster according to the log printing time. .
In one embodiment, as shown in fig. 8, the log collection device further includes a log data storage module and a cache data storage module, where the log data storage module is configured to store the target service log in a log data storage file, and the cache data storage module is configured to store the target hash value of the target service log in the cache data storage file.
It should be noted that, specific examples in this embodiment may refer to examples described in the foregoing embodiments and alternative implementations, and are not described in detail in this embodiment.
In addition, in combination with the log management method provided in the above embodiment, a storage medium may be provided in this embodiment. The storage medium has a computer program stored thereon; the computer program, when executed by a processor, implements any of the log management methods of the above embodiments.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magneto-resistive random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (PHASE CHANGE Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (10)
1. A log management method performed by a log processing apparatus, the log management method comprising:
Acquiring a target service log of a server cluster sent by log acquisition equipment; the target service log is acquired by the log acquisition equipment based on time variables corresponding to all service servers in the server cluster;
Determining whether the target service log is a duplicate log;
If not, storing the target service log, and sending the log printing time of the target service log to the log acquisition equipment, so that the log acquisition equipment updates the time variable corresponding to each service server in the server cluster according to the log printing time.
2. The method of claim 1, wherein the determining whether the target service log is a duplicate log comprises:
determining a target hash value of the target service log;
and determining whether the target service log is a repeated log according to the target hash value and the historical hash value of the historical service log stored in the buffer data storage file.
3. The method of claim 2, wherein the storing the target service log comprises:
And storing the target service log into a log data storage file, and storing the target hash value into a cache data storage file.
4. The method of claim 1, wherein the determining whether the target service log is a duplicate log further comprises:
if yes, the target service log is ignored.
5. A log management method performed by a log acquisition apparatus, the log management method comprising:
Acquiring a target service log of a server cluster based on time variables corresponding to each service server in the server cluster, and sending the target service log to log processing equipment so that the log processing equipment stores the target service log;
and acquiring the log printing time of the target service log fed back by the log processing equipment, and updating time variables corresponding to each service server in the server cluster according to the log printing time.
6. The method according to claim 5, wherein the collecting the target service log of the server cluster according to the time variable corresponding to each service server in the server cluster, and sending the target service log to the log processing device, so that after the log processing device stores the target service log, further includes:
determining the sending time length of the target service log according to the sending time stamp of the target service log;
If the sending time length is greater than a time length threshold value and the log printing time fed back by the log processing equipment is not received, the connection with the log processing equipment is reestablished, and the target service log is sent to the log processing equipment again.
7. The log management system is characterized by comprising log processing equipment and log acquisition equipment, wherein the log processing equipment is connected with the log acquisition equipment, the log acquisition equipment is used for acquiring target service logs of a server cluster according to time variables corresponding to all service servers in the server cluster and sending the target service logs to the log processing equipment, and the log processing equipment is used for acquiring the target service logs of the server cluster sent by the log acquisition equipment; determining whether the target service log is a duplicate log; if not, storing the target service log, and sending the log printing time of the target service log to the log acquisition equipment.
8. The log management system of claim 7, wherein the log collection device is further configured to collect a target service log of the server cluster based on time variables corresponding to each service server in the server cluster, and send the target service log to the log processing device; and acquiring the log printing time of the target service log fed back by the log processing equipment, and updating time variables corresponding to each service server in the server cluster according to the log printing time.
9. The log management system of claim 7, wherein the log collection device further comprises a log data storage module and a cache data storage module, the log data storage module configured to store the target service log in a log data storage file, the cache data storage module configured to store the target hash value of the target service log in a cache data storage file.
10. A computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the log management method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410182836.1A CN118193547A (en) | 2024-02-19 | 2024-02-19 | Log management method, log management system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410182836.1A CN118193547A (en) | 2024-02-19 | 2024-02-19 | Log management method, log management system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118193547A true CN118193547A (en) | 2024-06-14 |
Family
ID=91407512
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410182836.1A Pending CN118193547A (en) | 2024-02-19 | 2024-02-19 | Log management method, log management system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118193547A (en) |
-
2024
- 2024-02-19 CN CN202410182836.1A patent/CN118193547A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11989707B1 (en) | Assigning raw data size of source data to storage consumption of an account | |
| US12079255B1 (en) | Systems and methods for updating a status indication in a system providing dynamic indexer discovery | |
| US11604782B2 (en) | Systems and methods for scheduling concurrent summarization of indexed data | |
| US10262032B2 (en) | Cache based efficient access scheduling for super scaled stream processing systems | |
| US11615082B1 (en) | Using a data store and message queue to ingest data for a data intake and query system | |
| CN111309720B (en) | Time sequence data storage and reading method and device, electronic equipment and storage medium | |
| US10417265B2 (en) | High performance parallel indexing for forensics and electronic discovery | |
| US11966797B2 (en) | Indexing data at a data intake and query system based on a node capacity threshold | |
| US12019634B1 (en) | Reassigning a processing node from downloading to searching a data group | |
| CN110928851B (en) | Method, device and equipment for processing log information and storage medium | |
| US11093340B2 (en) | Summary file change log for faster forever incremental backup | |
| US11892976B2 (en) | Enhanced search performance using data model summaries stored in a remote data store | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| CN108228432A (en) | A kind of distributed link tracking, analysis method and server, global scheduler | |
| CN113918437B (en) | User behavior data analysis method, device, computer equipment and storage medium | |
| US11841827B2 (en) | Facilitating generation of data model summaries | |
| CN118193547A (en) | Log management method, log management system and storage medium | |
| CN115809311A (en) | Data processing method and device of knowledge graph and computer equipment | |
| CN115858471A (en) | Service data change recording method, device, computer equipment and medium | |
| CN115269519A (en) | Log detection method and device and electronic equipment | |
| CN114925283A (en) | Management method and system of push task, electronic device and medium | |
| CN116932779B (en) | Knowledge graph data processing method and device | |
| US20220358095A1 (en) | Managing data requests to a data shard | |
| CN117950942A (en) | Log management method, device, equipment, storage medium and program product | |
| CN115827647A (en) | Data storage method, data storage device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |