CN109062774A - Log processing method, device and storage medium, server - Google Patents
Log processing method, device and storage medium, server Download PDFInfo
- Publication number
- CN109062774A CN109062774A CN201810642922.0A CN201810642922A CN109062774A CN 109062774 A CN109062774 A CN 109062774A CN 201810642922 A CN201810642922 A CN 201810642922A CN 109062774 A CN109062774 A CN 109062774A
- Authority
- CN
- China
- Prior art keywords
- log
- target traffic
- compression
- server
- analysis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention relates to traffic, computer networks to calculate, data analysis, retrieval, storage, redundancy technique field, more particularly to a kind of log processing method provided in an embodiment of the present invention, it include: by target traffic Log backup into corresponding network memory, network memory is assigned internet protocol address；The server for being used to analyze target traffic log is linked to internet protocol address, so that the server analyzes target traffic log；Target traffic log compression after analysis is obtained into compression log, the filing path specified according to network memory is stored log is compressed.Log is replicated away by proprietary network and is analyzed again, it ensure that timeliness also reduces the excessive occupancy analysis of log analysis and compression all occupies the resources such as cpu, by the way that log compression is backuped to network memory, and the server for analyzing log is hung to the path of the network memory of backup log, it realizes distributed analysis and compression, solves the performance bottleneck of analysis.
Description
Technical field
The present invention relates to traffic, computer networks to calculate, data analysis, retrieval, storage, redundancy technique field, specifically relates to
And a kind of log processing method, device and computer readable storage medium, server.
Background technique
With the fast development of internet industry, the especially sudden emergence of mobile terminal application, the data explosion of a new round
The gate in epoch oneself through opening, the mutual competition mouth of all kinds of application on site becomes from thermalization.Wherein, partial log can not only be anti-
The operation conditions of server is answered, another part log is even more to provide the important indicator of development plan for business.Particularly, it is talking about
In business system, due to needing to analyze abnormal speech channel, or inquiry of the inquiry user for traffic, it can only pass through tserver days
Will is analyzed, complete different due to log-structured and web, and is divided into active and standby log, and active and standby can all be exported
Log, need to distinguish it is active and standby, analysis become extremely difficult, in addition further relate to cross-platform telephone traffic transfer, call-data analysis caused to become
Must be difficult, it can be cleaned additionally, due to log, program itself does not back up.In addition the log amount generated in a full platform is non-
Chang great, the tserver log of single platform log amount per second all occupy cpu higher in 5M or so, analysis and compression, and separate unit can not expire
Foot becomes the bottleneck of traffic log analysis and positioning so as to cause the tserver log analysis of single platform.
Summary of the invention
To overcome the above technical problem, the tserver log analysis of especially single platform becomes traffic log analysis and determines
The problem of bottleneck of position, spy propose following technical scheme:
A kind of log processing method provided in an embodiment of the present invention, comprising:
By target traffic Log backup into corresponding network memory, the network memory is assigned network protocol
Address；
The server for being used to analyze the target traffic log is linked to the internet protocol address, so that the clothes
Business device analyzes the target traffic log；
Target traffic log compression after analysis is obtained into compression log, the filing road specified according to the network memory
Diameter stores the compression log.
Further, described by the server for being used to analyze the target traffic log with being linked to the network protocol
Location, so that after the server analyzes the target traffic log, further includes:
The critical field in the target traffic log after extraction and analysis, by critical field storage to database
In.
Preferably, the critical field is stored in the database with indexed mode.
Further, the compression log is stored in the filing path specified according to the network memory
Later, further includes:
The target traffic log after deleting the analysis in the network memory.
Further, the compression log is stored in the filing path specified according to the network memory
In after, further includes:
Construct log corresponding with web-query interface and that the compression log can be got from the network memory
Query interface.
Further, described by target traffic Log backup into corresponding network memory before, further includes:
Determine whether current log is the target for needing to be backed up according to default key message and preset matching rule
Traffic log；
When determining the current log is the target traffic log for needing to be backed up, the target traffic will be backed up
The server of log is switched to server corresponding with the target traffic log.
Preferably, the preset matching rule is canonical matching rule.
Further, it is obtained among compression log in the target traffic log compression by after analysis, further includes:
The log of target traffic is temporally divided into the target traffic log of unit interval；Two neighboring unit interval
Target traffic log in, the target traffic log of previous unit interval, comprising the latter unit interval target words
The target traffic log content of front preset time period in business log, and mark the target traffic log content coherent as the time
Property verification content；The target traffic log of the unit interval is compressed respectively.
A kind of log processing device is additionally provided in the embodiment of the present invention, comprising:
Backup module is used for by target traffic Log backup into corresponding network memory, the network memory quilt
It is assigned internet protocol address；
Link module, for by the server for being used to analyze the target traffic log with being linked to the network protocol
Location, so that the server analyzes the target traffic log；
Compression module obtains compression log for the target traffic log compression after analyzing, according to the network storage
The specified filing path of device stores the compression log.
A kind of computer readable storage medium is additionally provided in the embodiment of the present invention, on the computer readable storage medium
It is stored with computer program, which realizes log processing method described in any technical solution when being executed by processor.
A kind of server is additionally provided in the embodiment of the present invention includes:
One or more processors；
Memory；
One or more application program, wherein one or more of application programs are stored in the memory and quilt
It is configured to be executed by one or more of processors, one or more of application programs are configured to carry out according to any skill
Log processing method described in art scheme.
Compared with the prior art, the present invention has the following beneficial effects:
1, a kind of log processing method provided in an embodiment of the present invention, comprising: by target traffic Log backup to corresponding
In network memory, the network memory is assigned internet protocol address；The target traffic log will be used to analyze
Server is linked to the internet protocol address, so that the server analyzes the target traffic log；It will divide
Target traffic log compression after analysis obtains compression log, and the filing path specified according to the network memory is by the compression
Log is stored.Log is replicated away by proprietary network and is analyzed again, and it is excessive to ensure that timeliness also reduces log analysis
Occupancy analysis and compression all occupy the resources such as cpu, by the way that log compression is backuped to network memory, and log will be analyzed
Server hangs the path of the network memory of backup log, enables and stores log memory and analyze the server one of log
One is corresponding, when needing to analyze log, is just able to achieve the analysis to log by the network memory path of extension, realizes distributed
Analysis and compression, solve the performance bottleneck of analysis.
2, a kind of log processing method provided in an embodiment of the present invention will be used to analyze the target traffic log described
Server be linked to the internet protocol address so that the server carries out analyzing it to the target traffic log
Afterwards, further includes: the critical field in the target traffic log after extraction and analysis deposits the critical field with indexed mode
It stores up in database.By extracting the critical field in the log after analysis, when needing to position traffic, just
First it can determine that log corresponds to the time of traffic generation, event, node particularly can pass through table according to critical field
Form directly shows the practical table of two dimension relevant to corresponding log, and the call will be shown on time by such as clicking specific call
Between and object two-dimensional practical table, clicking will call after specific event long-range rpc interface to pass through file name and file
Pointer and byte obtain log content；And then the function of quickly positioning is realized, but also user can be determined by critical field
Key message in log.
3, a kind of log processing method provided in an embodiment of the present invention is returned described according to what the network memory was specified
Shelves path by the compression log stored in after, further includes: construct corresponding with web-query interface and can be from the net
The log query interface of the compression log is got in network memory.Due to log-structured and web complete different, and
Be divided into active and standby log, and it is active and standby all can output journal, by developing web-query interface, so that can not only pass through
Tserver log is analyzed, can be with query and search network by the web interface after the web-query page inputs critical field
Log in memory inquires the corresponding logic of traffic, reaches quick analyzing and positioning, and by log interface, obtains compression
The detailed data of corresponding log inside log.
The additional aspect of the present invention and advantage will be set forth in part in the description, these will become from the following description
Obviously, or practice through the invention is recognized.
Detailed description of the invention
Above-mentioned and/or additional aspect and advantage of the invention will become from the following description of the accompanying drawings of embodiments
Obviously and it is readily appreciated that, in which:
Fig. 1 is a kind of flow diagram of embodiment in the exemplary embodiments of log processing method of the present invention；
Fig. 2 is the flow diagram of another embodiment in the exemplary embodiments of log processing method of the present invention；
Fig. 3 is log recording example in the exemplary embodiments of log processing method of the present invention；
Fig. 4 is the flow diagram of another embodiment in the exemplary embodiments of log processing method of the present invention；
Fig. 5 is the database table structure of key message in traffic log in the embodiment of the present invention；
Fig. 6 is the flow diagram of another embodiment in the exemplary embodiments of log processing method of the present invention；
Fig. 7 is the structural schematic diagram of the exemplary embodiments of log processing device of the present invention；
Fig. 8 is an example structure schematic diagram of server of the present invention.
Specific embodiment
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, and for explaining only the invention, and is not construed as limiting the claims.
Those skilled in the art of the present technique are appreciated that unless expressly stated, singular " one " used herein, " one
It is a ", " described " and "the" may also comprise plural form.It is to be further understood that being arranged used in specification of the invention
Diction " comprising " refers to that there are the feature, integer, step, operations, but it is not excluded that in the presence of or addition it is one or more other
Feature, integer, step, operation.
Those skilled in the art of the present technique are appreciated that unless otherwise defined, all terms used herein (including technology art
Language and scientific term), there is meaning identical with the general understanding of those of ordinary skill in fields of the present invention.Should also
Understand, those terms such as defined in the general dictionary, it should be understood that have in the context of the prior art
The consistent meaning of meaning, and unless idealization or meaning too formal otherwise will not be used by specific definitions as here
To explain.
It will be appreciated by those skilled in the art that so-called " application ", " application program ", " application software " and class of the invention
It is same concept well known to those skilled in the art like the concept of statement, refers to and instructed by series of computation machine and related data
The computer software for being suitable for electronics operation of the organic construction of resource.Unless specified, this name itself is not by programming language
Type, rank, the operating system of operation of also not rely by it or platform are limited.In the nature of things, this genus also not by appoint
The terminal of what form is limited.
A kind of log processing method provided in the embodiment of the present invention, is mainly used in genesys environment, by this hair
The log processing method provided in bright embodiment, by being pre-processed to log, so that analyzing abnormal traffic or inquiry words
Business logic becomes more simple, reduces the resource occupied during log analysis, accelerates the speed of call-data analysis.
The embodiment of the present invention provides a kind of log processing method, in a kind of wherein embodiment, as shown in Figure 1, including
S100 to S300.
S100: by target traffic Log backup into corresponding network memory, the network memory is assigned net
Network protocol address.
The analysis for the logical AND log content that the embodiment of the present invention is generated mainly for the log of identical product.In log
In structure, it is divided into master log and standby log, and both output journals, in order to reduce the difficulty of analysis log, in advance by Lord's Day
Will is copied in network memory (NAS) by reproducer, therefore at least one corresponding NAS of each product can pass through
It is arranged and partial log is stored in specified NAS.NAS (Network Attached Storage) network storage is based on standard
Network protocol realizes data transmission, is the computer of the various different operating systems such as the Windows/Linux/Mac OS in network
File-sharing and data backup are provided.Itself or a kind of special installation that use is directly connected with network medium realize data storage
Mechanism, the network memory is assigned internet protocol address, i.e., the equipment is assigned IP address, so client computer is by filling
When the server of data gateway can carry out access visit to it, or even in some cases, any intermediate medium visitor is not needed
Family machine can also directly access these equipment.Specifically, further including other steps before by target traffic Log backup into NAS
Suddenly, Details as Follows:
Further, in a kind of wherein embodiment, as shown in Fig. 2, it is described by target traffic Log backup to pair
Before in the network memory answered, further includes:
S101: according to preset key message and preset matching rule determine current log whether be need to be backed up it is described
Target traffic log；
S102: when determining the current log is the target traffic log for needing to be backed up, the mesh will be backed up
The server of mark traffic log is switched to server corresponding with the target traffic log.
The log of target traffic is to need analyzed log, since log includes master log and standby log, in determination
When target traffic log, by presetting key message and determining whether current log is to need to be analyzed with preset matching rule
Target traffic log, by analyzing current log, can determine if log is current such as when target traffic log is master log
Based on, deposited inside tserver or sipserver log ' HA Role:Primary' keyword, that is to say, that ' HA Role:
Primary' is key message.It just can determine that the log is master log by the key message, due to the service of backup log
Device is corresponding with master log and backup log respectively, is determining that current log is the target traffic log for needing analyzed backup
When (master log), Role changed to (primary/backup) is being generated, if the server of current backup is not and Lord's Day
When the corresponding primary server of will, determines and the server of current backup is switched to primary server corresponding with master log；If
When the server of current backup is primary server corresponding with master log, then without switching.When target traffic log is standby
When log, processing is identical, and this will not be repeated here.It in addition is not in that above-mentioned switching is crucial inside route service log
Word, but according to canonical match request to d+ and do not include that the determination of RequestRegisterAddress keyword is
No is master log.
Preferably, the preset matching rule is canonical matching rule.
Canonical matching rule mainly passes through regular expression and is matched, and regular expression is a kind of for operation and checkout
The matching tool of string data can carry out text the operation such as matching for a string of special characters, regular expression
Matching grammer can be referring specifically to network address: http://www.regular-expressions.infoj.For example, for Fig. 3's
Log recording can construct following regular expression and be matched.
Pl=[" Processing s+ (w+) # (w+) s (for s+ ((d+) { 3 } d+) s+at s+ (d+-
D+- d+ s d+: d+: d+)) and s+ [(w+)] n+ (Parameters: .+) ", controller, method,
client_P,timestamp,http_method,content]。
The data gone out by regular expression matching are by the data as field location corresponding in log profile.Above
In regular expression example, the value of controller field correspond to " w+ ", the value of method field corresponds to " w+ ",
The value of client_IP field correspond to " (d+) { 3 } d+ ", the value of timestamp field correspond to " d+- d+- d+ s
D+: d+: d+ ", the value of http_method field corresponds to " w+ ", and the value of content field corresponds to "
Parameters:\.+".It by the matching rule of regular expression, can be matched with log recording, so as to parse simultaneously
Extract the daily record data in log recording.Therefore, regular expression can be parsed and be extracted dynamic in log content
State data.Preferably, in other implementations, when log is structural data, the preset matching rule can be with
It is realized by character string template, character string template is a kind of template matching engine, supports java, the language such as C#, Python, character
The matching grammer of string template may refer to network address: http://www.stringtemplate.org/.The matching of character string template
It is different with regular expression, however the two can achieve the effect that counterpart will record is extracted, i.e., in a large amount of log
In determine whether for target traffic log.
S200: being linked to the internet protocol address for the server for being used to analyze the target traffic log, so that
The server analyzes the target traffic log.
After determining the log of target traffic and server corresponding with target traffic log, by the server by mesh
Traffic Log backup (duplication) is marked into network memory, in order to enable the server of analysis target traffic log can be correct
Target traffic log is found, the server of target traffic log described in link analysis to the internet protocol address, i.e. analysis institute
The server for stating target traffic log hangs network memory path identical with storage target traffic log network memory, is needing
When analyzing log, analyzing the server of log by the path just can determine position where target traffic log, and according to
The log inside network memory is analyzed according to the path.Specifically the server for analyzing log and the path NAS are linked
Get up, when carrying out log analysis, analyzes the server just invocation target traffic log directly from NAS of log, or analysis
The server of log analyzes target traffic log directly in NAS, and then convenient for target traffic log analysis.In conjunction with above-mentioned
Illustrate, after the log of target traffic is copied in network memory by its corresponding proprietary network, then to target traffic log
It is analyzed, ensure that the timeliness of log analysis, while also avoided log analysis and occupying the resources such as production system CPU.
S300: the target traffic log compression after analysis is obtained into compression log, is specified according to the network memory
Filing path stores the compression log.
Further, in a kind of wherein embodiment, in the filing path specified according to the network memory
After the compression log is stored, further includes:
The target traffic log after deleting the analysis in the network memory.
After the completion of log analysis, by the log after analysis by the modes compress backup such as gzip, NAS then will be backuped to
In and the log that has been analyzed delete, can be by the Central Plains NAS due to being compressed the log after analysis
The log of target traffic is deleted after some analyses, and compression log can reduce the storage sky that log occupies database or index structure
Between, the log compressed after the corresponding original of log is analyzed is deleted, the memory space that log occupies database is further decreased, keeps away
Exempting from insufficient memory reduces the speed of subsequent query log.Preferably, can rapidly be determined by interface for the ease of subsequent
Position inquires compression log, and therefore, compression log is stored according to the filing path that network memory is specified.
Further, in a kind of wherein embodiment, as shown in figure 4, will be used to analyze the target traffic described
The server of log is linked to the internet protocol address, so that the server analyzes the target traffic log
Later, further includes:
S210: the critical field in the target traffic log after extraction and analysis, by the critical field with the side of index
Formula is stored into database.
When analyzing the log in NAS, an index structure (index data base), keyword in extraction and analysis log are constructed
Section deposit database (index structure) after the critical field structuring in log, and establishes incidence relation.By the pass in log
Key field is for constructing index structure (details such as Fig. 5).The data of each layer substantially need to be configured in index structure with user,
It such as can be with the data on entitled index structure top, using number and ID as lower layer of top data arranged side by side；It can certainly
Name is referred to as the data on lower layer of top by the data of number and/or ID as top index structure top.It has read in log
Key message, such as title of the event of call, number, id and related information etc., and have recorded the finger of the information hereof
Pin position and content byte number.The data type at the end of different log top layers is the same.Index data base can be with form exhibition
Show, show critical field in the table, each critical field is associated with other critical fielies such as: object, time, event.
In Fig. 5, using tbl_files_log as the data form on the entire top relevant to log of index structure, respectively at
The data on the peripheries such as tbl_app, tbl_logserver, tbl_ts_log, tbl_rels_log are associated, in each data
Data corresponding thereto are all respectively provided in table, the gauge outfit of each table can be used as the top in index structure, pass through
Tbl_files_log inquires corresponding gauge outfit, can obtain data corresponding with the gauge outfit according to the gauge outfit.Wherein, in Fig. 5
In, the main and standby relation of tbl_app record application and the corresponding relationship of the application and the server of tbl_logserver.In this way may be used
It active and standby be applied to bottom not have to distinguish which is currently main.It can all inquire.Tbl_files_log record log file name and
The real time point that the log generates is with log application type and is associated with the application of tbl_app.Tbl_ts_log passes through scanning
Tserver log generates traffic time, event title, the index of the key messages such as node, and is associated with tbl_files_log's
File id.The on-hook index that tbl_rels_log is generated by scanning tserver log, and it is associated with tbl_files_log's
File id.Tbl_uuid_log is associated with tbl_files_ by scanning the uuid and connid relationship that tserver log generates
The file id of log.The switching front and back connid relationship that tbl_connid_log is generated by scanning tserver log, and be associated with
The file id of tbl_files_log.The time for the sip that tbl_sip_log is generated by scanning sipserver log, event name
Claim, ip, and is associated with the file id of tbl_files_log.Tbl_callid_log is generated by scanning sipserver log
Callid and uuid relationship, and it is associated with the file id of tbl_files_log.Tbl_urs_log is generated by scanning urs log
Time, event title, node, and it is associated with the file id of tbl_files_log.Tbl_mcp_log is generated by scanning mcp log
Time, content, and be associated with tbl_files_log file id.Tbl_mcpsession_log is generated by scanning mcp log
Uuid and sessionid relationship, and be associated with tbl_files_log file id.Preferably, the critical field is with the side of index
Formula stores in the database.
On the basis of above-mentioned, by the critical field recorded in index structure relationship and index structure, looked into the page
The content (user inquires a period of time in traffic query page) for asking a period of time generation, will be by the pass in above-mentioned index structure
Key field shows relevant to tbl_files_log file table, click specific traffic will show the traffic temporally with
The two-dimensional practical table of object will call long-range rpc interface to pass through file name and file pointer after clicking specific event
And byte, log content is obtained, actual interface is read from nas by 3 parameters (source_dir, dist_path, fn2)
Simultaneously returned content comes out for compression log.Therefore 90% space can be saved by compression (100M is compressed to 10M), additionally it is possible to real
Now quickly inquiry, subsequent query just directly extract compressed log.It is of course also possible in the search of query page
Engine input critical field is inquired.Simultaneously because the log in original telephone traffic system under genesys environment can not be protected for a long time
It deposits, compress backup inside network memory will be aimed at day by the application realizes log keeping for a long time；In addition, in telephone traffic system
Problem escalation has time delay, provides foundation by compress backup to analyze passing problem.It should be noted that in the present invention
In the embodiment of offer, be only recorded in index structure the key content in target traffic log is a small amount of, it is practical also
Many contents are not recorded in index structure, wherein greatly also in target traffic journal file.Therefore, subsequent
Query steps in, can be matched by the critical field in the key message and index structure of input, successful match
When, corresponding traffic logic just can be found, or determine the corresponding position of traffic logic, the time such as occurred, event
Title, node etc..
Further, in a kind of wherein embodiment, as shown in fig. 6, described specified according to the network memory
Filing path by the compression log stored in after, further includes:
S310: building is corresponding with web-query interface and the compression log can be got from the network memory
Log query interface.
Exploitation constructs log query interface corresponding with web-query interface, in order to which user can be looked by web interface
Ask corresponding traffic logic and analyze it positioning etc., so that can not only be analyzed by tserver log, it can also
To compress the detailed data in log by web-query log, acquisition, so that cross-platform switching is easy.It inputs and closes in web-query
Key field, since it is corresponding with log query interface, the content that web-query interface is inputted passes through the log query
Interface gets the detailed data of the corresponding log inside corresponding compression log, that is, inquires the corresponding logic of traffic, real
Now rapidly reach quick analyzing and positioning.Specifically, for example, one section of program is arranged in interface, input is crucial in web-query
When field (critical field is one or more), that is, starts the section or call this section of program, obtained by corresponding path
Compressed package.The file of compression has specific Naming conventions, wherein can receive in log with Apply Names times, web interface
To the filename and file pointer and byte-sized of log, interface, which passes through filename specification after receiving, can navigate to this document.
Wherein, critical field, the log file name including tserver component, file corresponding application name, corresponding event, event
If the time, client's number, the connid of call, queuing message, is related to switching and meeting and cross-platform switching at object
Connid, the calluuid for being associated with sip log, client request id, file pointer and byte Sipserver component are except above-mentioned
In addition there are also sip signaling key message include the time of signaling, event title, the direction interacted with sip, callid, to and
The address from, ip, the log file name of file pointer and byte Urs component, the corresponding application name of file, corresponding event, thing
Time of part, object, client's number, call connid client request id, file pointer and byte.Provided by the invention
In embodiment, define 2 kinds of functions according to function and realize that one is the detailed content interface for obtaining corresponding event is as follows: interface is passed
Enter is one or more of filename, character string, file pointer, numeric type, byte, numeric type.It spreads out of and is
One or more of log content, character string type.A kind of interface is the compressed file of acquisition original log, and interface is such as
Under: interface is passed to as filename, and character string outflow is file stream, can remain the form of file.Specifically, in conjunction with above
Illustrate, in embodiments herein, Details as Follows as follows using code is realized for interface:
Further, it in a kind of wherein embodiment, is pressed in the target traffic log compression by after analysis
Among contracting log, further includes:
The log of target traffic is temporally divided into the target traffic log of unit interval；Two neighboring unit interval
Target traffic log in, the target traffic log of previous unit interval, comprising the latter unit interval target words
The target traffic log content of front preset time period in business log, and mark the target traffic log content coherent as the time
Property verification content；The target traffic log of the unit interval is compressed respectively.
For multithreading compressed package situation simultaneously, for example, 3 packets of acquisition segmentation, are followed successively by 9 points packets, 10 points 1
Packet, 11 points packets, three threads compress simultaneously, and generally prior time has first compressed, it is likely that 10 points of packet is very big, leads
11 points of packet is caused first to compress；In decompression, may cause before 11 points of packet comes at 10 points.To solve this problem, each
The content of packet, including one section before packet content below, for example, 10 point packets, backmost include 11 0-5 minutes contents of point.Then
In decompression, 10 point packets with 11 points of packet can proofread, so that it is determined that 10 points of packet including 11 0-5 minutes contents of point
Before 11 points, then again according to time sequence.
A kind of log processing device is also provided in the embodiment of the present invention, in a kind of wherein embodiment, such as Fig. 7 institute
Show, comprising:
Backup module 100 is used for by target traffic Log backup into corresponding network memory, the network memory
It is assigned internet protocol address；
Link module 200, for the server for being used to analyze the target traffic log to be linked to the network protocol
Address, so that the server analyzes the target traffic log；
Compression module 300 obtains compression log for the target traffic log compression after analyzing, deposits according to the network
The specified filing path of reservoir stores the compression log.
Further, as shown in fig. 7, a kind of log processing device provided in the embodiment of the present invention, further includes: extract mould
Block 210 is deposited the critical field with indexed mode for the critical field in the target traffic log after extraction and analysis
It stores up in database.Removing module 310, for deleting the target traffic log after the analysis in the network memory.
Module 320 is constructed, for constructing corresponding with web-query interface and the compression day can be got from the network memory
The log query interface of will.Judgment module 101, for determining that current log is according to default key message and preset matching rule
The no target traffic log being backed up for needs；Switching module 102, for standby to need when determining the current log
When the target traffic log of part, the server for backing up the target traffic log is switched to and the target traffic log
Corresponding server.Preferably, the preset matching rule is canonical matching rule；Cutting unit 301, by target traffic day
Will is temporally divided into the target traffic log of unit interval；It is preceding in the target traffic log of two neighboring unit interval
The target traffic log of one unit interval, front preset time period in the target traffic log comprising the latter unit interval
Target traffic log content, and mark the target traffic log content as time continuity check content；It compresses respectively
The target traffic log of the unit interval.
The embodiment of above-mentioned log processing method may be implemented in a kind of log processing device provided in an embodiment of the present invention, tool
Body function realizes the explanation referred in embodiment of the method, and details are not described herein.
A kind of computer readable storage medium provided in an embodiment of the present invention stores on the computer readable storage medium
There is computer program, log processing method described in any one technical solution is realized when which is executed by processor.Wherein, institute
Stating computer readable storage medium includes but is not limited to any kind of disk (including floppy disk, hard disk, CD, CD-ROM and magneto-optic
Disk), ROM (Read-Only Memory, read-only memory), RAM (Random AcceSS Memory, immediately memory),
EPROM (EraSable Programmable Read-Only Memory, Erarable Programmable Read only Memory), EEPROM
(Electrically EraSable Programmable Read-Only Memory, Electrically Erasable Programmable Read-Only Memory),
Flash memory, magnetic card or light card.It is, storage equipment includes by equipment (for example, computer, mobile phone) can read
Any medium of form storage or transmission information can be read-only memory, disk or CD etc..
A kind of computer readable storage medium provided in an embodiment of the present invention can intelligently realize above-mentioned log processing side
The embodiment of method, log are replicated away by proprietary network and are analyzed again, and it is excessive to ensure that timeliness also reduces log analysis
It occupies analysis and compression all occupies the resources such as cpu, by the way that log compression is backuped to network memory, and the clothes that will analyze log
Business device hangs the path of the network memory of backup log, enables and stores log memory and analyze the server of log one by one
It is corresponding, when needing to analyze log, the analysis to log is just able to achieve by the network memory path of extension, realizes distributed point
Analysis and compression, solve the performance bottleneck of analysis；A kind of log processing method is also provided in the embodiment of the present invention, and target is talked about
Log backup be engaged in into corresponding network memory, the network memory is assigned internet protocol address；It will be used to analyze institute
The server for stating target traffic log is linked to the internet protocol address, so that the server is to the target traffic day
Will is analyzed；Target traffic log compression after analysis is obtained into compression log, is returned according to what the network memory was specified
Shelves path stores the compression log.In embodiment provided by the invention, by the log of identical product (such as same app)
It copies in network memory in time, and then reduces the memory of end product where log occupies app, while by that will analyze
The server of the product log is suspended to the path of the network memory of storage log, and day can be analyzed in network memory by realizing
Will reduces the resources such as cpu of end product where analysis log occupies app, the log after analysis is compressed, and having divided
The log of analysis and compressed backup is deleted, and the memory that log occupies network memory is reduced, and then reduces memory to analysis
The influence of log speed.Different product or the different types of log of identical product are respectively stored into corresponding network memory
In, and the analysis of progress log and compress backup in its corresponding network memory, realize distributed analysis and compression
Backup log, and then solve the bottleneck directly in terminal analysis log, improve the speed of analysis log.Log compression is standby
Part, the memory that log occupies network memory can be reduced, corresponding log can be found when ensure that the inquiry of subsequent traffic,
After avoiding log in application program from being deleted, the case where appearance cannot carry out traffic inquiry and analysis again.
The embodiment of above-mentioned log processing method may be implemented in computer readable storage medium provided in an embodiment of the present invention,
Concrete function realizes the explanation referred in embodiment of the method, and details are not described herein.
In addition, the present invention also provides a kind of servers, as shown in figure 8, the server process in another embodiment
The devices such as device 503, memory 505, input unit 507 and display unit 509.It will be understood by those skilled in the art that Fig. 8 shows
Structure devices out do not constitute the restriction to Servers-all, may include than illustrating more or fewer components or group
Close certain components.Memory 505 can be used for storing application program 501 and each functional module, and the operation of processor 503 is stored in
The application program 501 of reservoir 505, thereby executing the various function application and data processing of equipment.Memory 505 can be interior
Memory or external memory, or including both built-in storage and external memory.Built-in storage may include read-only memory
(ROM), programming ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash
Device or random access memory.External memory may include hard disk, floppy disk, ZIP disk, USB flash disk, tape etc..It is disclosed in this invention to deposit
Reservoir includes but is not limited to the memory of these types.Memory 505 disclosed in this invention is only used as example rather than as limit
It is fixed.
Input unit 507 be used for receive signal input and user entered keyword section and user input execute inquiry
The operational order of retrieval.Input unit 507 may include touch panel and other input equipments.Touch panel collects client and exists
On or near it touch operation (such as client using any suitable object or attachment such as finger, stylus on touch panel
Or the operation near touch panel), and corresponding attachment device is driven according to a pre-set procedure；Other input equipments can
To include but is not limited to physical keyboard, function key (such as broadcasting control button, switch key etc.), trace ball, mouse, operating stick
Deng one of or it is a variety of.Display unit 509 can be used for showing the information of client's input or be supplied to the information and meter of client
Calculate the various menus of machine equipment.The forms such as liquid crystal display, Organic Light Emitting Diode can be used in display unit 509.Processor 503
It is the control centre of computer equipment, using the various pieces of various interfaces and the entire computer of connection, by running or holding
The software program and/or module that row is stored in memory 503, and the data being stored in memory are called, it executes various
Function and processing data.One or more processors 503 shown in fig. 8 are able to carry out, realize backup module shown in fig. 7
100 function, the function of link module 200, the function of compression module 300, the function of extraction module 210, removing module 310
Function, the building function of module 320, the function of judgment module 101, the function of switching module 102, the function of cutting unit 301.
In one embodiment, the server includes one or more processors 503, and one or more storages
Device 505, one or more application program 501, wherein one or more of application programs 501 are stored in memory 505
And be configured as being executed by one or more of processors 503, one or more of application programs 301 are configured to carry out
Log processing method described in above embodiments.
A kind of server provided in an embodiment of the present invention can intelligently realize the embodiment of above-mentioned log processing method,
Log is replicated away by proprietary network and is analyzed again, ensure that timeliness also reduces the excessive occupancy analysis and pressure of log analysis
Contracting all occupies the resources such as cpu, by the way that log compression is backuped to network memory, and the server for analyzing log is hung backup day
The path of the network memory of will enables storage log memory and the server of analysis log to correspond, is needing
When analyzing log, the analysis to log is just able to achieve by the network memory path of extension, realizes distributed analysis and compression, solution
Determined analysis performance bottleneck；A kind of log processing method is also provided in the embodiment of the present invention, by target traffic Log backup
Into corresponding network memory, the network memory is assigned internet protocol address；The target words will be used to analyze
The server of business log is linked to the internet protocol address, so that the server divides the target traffic log
Analysis；Target traffic log compression after analysis is obtained into compression log, the filing path specified according to the network memory will
The compression log is stored.It is in embodiment provided by the invention, the log of identical product (such as same app) is multiple in time
It makes in network memory, and then reduces the memory of end product where log occupies app, while by the way that the product day will be analyzed
The server of will is suspended to the path of the network memory of storage log, and log can be analyzed in network memory by realizing, and reduces
The resources such as cpu for analyzing end product where log occupies app, the log after analysis is compressed, and analyzed and by
The log of the backup of compression is deleted, and the memory that log occupies network memory is reduced, and then reduces memory to analysis log speed
The influence of degree.Different product or the different types of log of identical product are respectively stored into corresponding network memory, and
Analysis and the compress backup that log is carried out in its corresponding network memory realize distributed analysis and compress backup day
Will, and then solve the bottleneck directly in terminal analysis log, improve the speed of analysis log.Log compression is backed up, it can
To reduce the memory that log occupies network memory, corresponding log can be found when ensure that the inquiry of subsequent traffic, is avoided
After log in application program is deleted, the case where appearance cannot carry out traffic inquiry and analysis again.
The embodiment of the log processing method of above-mentioned offer, specific function may be implemented in server provided in an embodiment of the present invention
It is able to achieve the explanation referred in embodiment of the method, details are not described herein.
The above is only some embodiments of the invention, it is noted that for the ordinary skill people of the art
For member, various improvements and modifications may be made without departing from the principle of the present invention, these improvements and modifications are also answered
It is considered as protection scope of the present invention.
Claims (10)
1. a kind of log processing method characterized by comprising
By target traffic Log backup into corresponding network memory, the network memory is with being assigned network protocol
Location；
The server for being used to analyze the target traffic log is linked to the internet protocol address, so that the server
The target traffic log is analyzed；
Target traffic log compression after analysis is obtained into compression log, the filing path specified according to the network memory will
The compression log is stored.
2. log processing method according to claim 1, which is characterized in that will be used to analyze the target traffic described
The server of log is linked to the internet protocol address, so that the server analyzes the target traffic log
Later, further includes:
The critical field in the target traffic log after extraction and analysis stores the critical field to number with indexed mode
According in library.
3. log processing method according to claim 1, which is characterized in that described specified according to the network memory
Filing path the compression log is stored after, further includes:
The target traffic log after deleting the analysis in the network memory.
4. log processing method according to claim 1, which is characterized in that described specified according to the network memory
Filing path by the compression log stored in after, further includes:
Construct log query corresponding with web-query interface and that the compression log can be got from the network memory
Interface.
5. log processing method according to claim 1, which is characterized in that it is described by target traffic Log backup to pair
Before in the network memory answered, further includes:
Determine whether current log is the target traffic for needing to be backed up according to default key message and preset matching rule
Log；
When determining the current log is the target traffic log for needing to be backed up, the target traffic log will be backed up
Server be switched to server corresponding with the target traffic log.
6. log processing method according to claim 5, which is characterized in that the preset matching rule is canonical matching rule
Then.
7. log processing method according to claim 5, which is characterized in that in the target traffic log by after analysis
Compression obtains among compression log, further includes:
The target traffic log is temporally divided into the target traffic log of unit interval；When two neighboring unit
Between section target traffic log in, the target traffic log of previous unit interval, the mesh comprising the latter unit interval
The target traffic log content of front preset time period in traffic log is marked, and marks the target traffic log content as the time
Continuity check content；The target traffic log of the unit interval is compressed respectively.
8. a kind of log processing device characterized by comprising
Backup module, for, into corresponding network memory, the network memory to be assigned by target traffic Log backup
There is internet protocol address；
Link module, for the server for being used to analyze the target traffic log to be linked to the internet protocol address, with
So that the server analyzes the target traffic log；
Compression module obtains compression log for the target traffic log compression after analyzing, refers to according to the network memory
Fixed filing path stores the compression log.
9. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium
Program realizes claim 1 to 7 described in any item log processing methods when the program is executed by processor.
10. a kind of server characterized by comprising
One or more processors；
Memory；
One or more application program, wherein one or more of application programs are stored in the memory and are configured
To be executed by one or more of processors, one or more of application programs are configured to carry out according to claim 1
The step of to 7 described in any item log processing methods.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810642922.0A CN109062774A (en) | 2018-06-21 | 2018-06-21 | Log processing method, device and storage medium, server |
| PCT/CN2018/108058 WO2019242148A1 (en) | 2018-06-21 | 2018-09-27 | Log processing method and apparatus, and storage medium and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810642922.0A CN109062774A (en) | 2018-06-21 | 2018-06-21 | Log processing method, device and storage medium, server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109062774A true CN109062774A (en) | 2018-12-21 |
Family
ID=64821281
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810642922.0A Pending CN109062774A (en) | 2018-06-21 | 2018-06-21 | Log processing method, device and storage medium, server |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109062774A (en) |
| WO (1) | WO2019242148A1 (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110536031A (en) * | 2019-08-30 | 2019-12-03 | 上海浦东发展银行股份有限公司信用卡中心 | A kind of association of traffic data and integration method |
| CN111414613A (en) * | 2020-03-18 | 2020-07-14 | 杭州迪普科技股份有限公司 | Log processing method and device |
| CN111488314A (en) * | 2020-03-30 | 2020-08-04 | 北京中电华大电子设计有限责任公司 | Simulation log analysis method based on Python |
| CN113064752A (en) * | 2019-12-16 | 2021-07-02 | 华晨宝马汽车有限公司 | Method, system, and computer readable medium for archiving logs |
| CN113190726A (en) * | 2021-04-16 | 2021-07-30 | 珠海格力精密模具有限公司 | Method for reading CAE (computer aided engineering) modular flow analysis data, electronic equipment and storage medium |
| CN113297008A (en) * | 2021-05-19 | 2021-08-24 | 阿里巴巴新加坡控股有限公司 | Data processing method and system |
| CN114490557A (en) * | 2022-02-16 | 2022-05-13 | 平安科技（深圳）有限公司 | Seat duration statistical method, system and device, electronic equipment and storage medium |
| CN114697070A (en) * | 2021-12-31 | 2022-07-01 | 成都思维世纪科技有限责任公司 | Method and system for dynamic compression and storage of HTTP (hyper text transport protocol) traffic |
| CN114936615A (en) * | 2022-07-25 | 2022-08-23 | 南京大数据集团有限公司 | Small sample log information anomaly detection method based on characterization consistency correction |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1199970A (en) * | 1998-05-13 | 1998-11-25 | 广东省邮电科学技术研究院 | Central operation and maintenance system of analog movable communication network B |
| CN101552842A (en) * | 2008-01-09 | 2009-10-07 | 埃森哲环球服务有限公司 | Call center application data and interoperation architecture for a telecommunication service center |
| CN102750196A (en) * | 2011-04-20 | 2012-10-24 | 大连兆阳软件科技有限公司 | Data storage and backup system and method |
| CN103200037A (en) * | 2013-04-11 | 2013-07-10 | 深圳市共进电子股份有限公司 | System log (syslog) storing method |
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
| CN107885817A (en) * | 2017-11-06 | 2018-04-06 | 余帝乾 | A kind of method and apparatus based on big data networks congestion control |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9729671B2 (en) * | 2014-10-05 | 2017-08-08 | YScope Inc. | Systems and processes for computer log analysis |
| CN106649336B (en) * | 2015-10-30 | 2019-10-25 | 华为数字技术（苏州）有限公司 | A kind of log compression method and log processing equipment, log processing system |
| CN106897187A (en) * | 2017-01-23 | 2017-06-27 | 北京思特奇信息技术股份有限公司 | A kind of terminal data location mode and device |
| CN107622084A (en) * | 2017-08-10 | 2018-01-23 | 深圳前海微众银行股份有限公司 | Blog management method, system and computer-readable recording medium |
-
2018
- 2018-06-21 CN CN201810642922.0A patent/CN109062774A/en active Pending
- 2018-09-27 WO PCT/CN2018/108058 patent/WO2019242148A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1199970A (en) * | 1998-05-13 | 1998-11-25 | 广东省邮电科学技术研究院 | Central operation and maintenance system of analog movable communication network B |
| CN101552842A (en) * | 2008-01-09 | 2009-10-07 | 埃森哲环球服务有限公司 | Call center application data and interoperation architecture for a telecommunication service center |
| CN102750196A (en) * | 2011-04-20 | 2012-10-24 | 大连兆阳软件科技有限公司 | Data storage and backup system and method |
| CN103200037A (en) * | 2013-04-11 | 2013-07-10 | 深圳市共进电子股份有限公司 | System log (syslog) storing method |
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
| CN107885817A (en) * | 2017-11-06 | 2018-04-06 | 余帝乾 | A kind of method and apparatus based on big data networks congestion control |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110536031A (en) * | 2019-08-30 | 2019-12-03 | 上海浦东发展银行股份有限公司信用卡中心 | A kind of association of traffic data and integration method |
| CN113064752A (en) * | 2019-12-16 | 2021-07-02 | 华晨宝马汽车有限公司 | Method, system, and computer readable medium for archiving logs |
| CN113064752B (en) * | 2019-12-16 | 2023-11-21 | 华晨宝马汽车有限公司 | Method, system and computer readable medium for archiving logs |
| CN111414613A (en) * | 2020-03-18 | 2020-07-14 | 杭州迪普科技股份有限公司 | Log processing method and device |
| CN111414613B (en) * | 2020-03-18 | 2023-12-26 | 杭州迪普科技股份有限公司 | Log processing method and device |
| CN111488314B (en) * | 2020-03-30 | 2023-06-30 | 北京中电华大电子设计有限责任公司 | Python-based simulation log analysis method |
| CN111488314A (en) * | 2020-03-30 | 2020-08-04 | 北京中电华大电子设计有限责任公司 | Simulation log analysis method based on Python |
| CN113190726A (en) * | 2021-04-16 | 2021-07-30 | 珠海格力精密模具有限公司 | Method for reading CAE (computer aided engineering) modular flow analysis data, electronic equipment and storage medium |
| CN113297008A (en) * | 2021-05-19 | 2021-08-24 | 阿里巴巴新加坡控股有限公司 | Data processing method and system |
| CN113297008B (en) * | 2021-05-19 | 2023-12-12 | 阿里巴巴新加坡控股有限公司 | Data processing method and system |
| CN114697070A (en) * | 2021-12-31 | 2022-07-01 | 成都思维世纪科技有限责任公司 | Method and system for dynamic compression and storage of HTTP (hyper text transport protocol) traffic |
| CN114697070B (en) * | 2021-12-31 | 2024-04-02 | 成都思维世纪科技有限责任公司 | Method and system for dynamically compressing and storing HTTP protocol traffic |
| CN114490557A (en) * | 2022-02-16 | 2022-05-13 | 平安科技（深圳）有限公司 | Seat duration statistical method, system and device, electronic equipment and storage medium |
| CN114936615A (en) * | 2022-07-25 | 2022-08-23 | 南京大数据集团有限公司 | Small sample log information anomaly detection method based on characterization consistency correction |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019242148A1 (en) | 2019-12-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109062774A (en) | Log processing method, device and storage medium, server | |
| CN109034993B (en) | Account checking method, account checking equipment, account checking system and computer readable storage medium | |
| US10462002B2 (en) | Automatically determining requirements for provisioning a hosted computing environment | |
| US10560465B2 (en) | Real time anomaly detection for data streams | |
| US10447772B2 (en) | Managed function execution for processing data streams in real time | |
| US20200285514A1 (en) | Automated reconfiguration of real time data stream processing | |
| US7827191B2 (en) | Discovering web-based multimedia using search toolbar data | |
| CN109800207B (en) | Log analysis method, device and equipment and computer readable storage medium | |
| US20100082774A1 (en) | Distributed File System Consistency Mechanism Extension for Enabling Internet Video Broadcasting | |
| CN108471366A (en) | A kind of stereoscopic monitoring system of facing cloud native applications | |
| US10360133B2 (en) | Analyzing analytic element network traffic | |
| CN101454764A (en) | Independent actionscript analytics tools and techniques | |
| WO2020087082A1 (en) | Trace and span sampling and analysis for instrumented software | |
| Vallentin et al. | {VAST}: A Unified Platform for Interactive Network Forensics | |
| WO2008021459A2 (en) | Software web crawlwer and method thereof | |
| CN106649120A (en) | Data acquisition method, and data analysis method and system | |
| CN112685270A (en) | System monitoring log acquisition method and device, electronic equipment and medium | |
| CN113656673A (en) | Master-slave distributed content crawling robot for advertisement delivery | |
| CN108900547B (en) | Source returning control method and device | |
| EP4363976A1 (en) | Streaming analytics using a serverless compute system | |
| CN114546756A (en) | Method and system for monitoring link data in micro-service architecture system | |
| CN116974948B (en) | Service system testing method, system, equipment and medium | |
| EP3010194B1 (en) | Method of tracing a transaction in a network | |
| CN111782428B (en) | Data calling system and method | |
| US20230138113A1 (en) | System for retrieval of large datasets in cloud environments |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181221 |