CN112115114A - Log processing method, device, equipment and storage medium - Google Patents
Log processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112115114A CN112115114A CN202011023854.3A CN202011023854A CN112115114A CN 112115114 A CN112115114 A CN 112115114A CN 202011023854 A CN202011023854 A CN 202011023854A CN 112115114 A CN112115114 A CN 112115114A
- Authority
- CN
- China
- Prior art keywords
- log
- time
- real
- user behavior
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 6
- 238000012545 processing Methods 0.000 claims abstract description 65
- 230000006399 behavior Effects 0.000 claims description 139
- 238000000034 method Methods 0.000 claims description 45
- 238000013507 mapping Methods 0.000 claims description 20
- 230000015654 memory Effects 0.000 claims description 18
- 238000004140 cleaning Methods 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 5
- 238000007405 data analysis Methods 0.000 abstract description 2
- 238000003780 insertion Methods 0.000 description 11
- 230000037431 insertion Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 7
- 238000012544 monitoring process Methods 0.000 description 7
- 238000013500 data storage Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000004590 computer program Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000012827 research and development Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 230000008676 import Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000000638 solvent extraction Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000002354 daily effect Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/156—Query results presentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Probability & Statistics with Applications (AREA)
- Library & Information Science (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a log processing method, a log processing device, log processing equipment and a log processing storage medium, and relates to the technical field of big data and cloud computing in computer technology, in particular to the fields of distributed storage, intelligent search meters, data statistics, data analysis and the like. The specific implementation scheme is as follows: the user behavior log is stored in a real-time log table of a first database, so that the user behavior log is stored in real time; the user behavior logs in the previous time period are stored in the offline log table of the second database at intervals of a time period, the user behavior logs in the previous time period in the real-time log table are deleted, mass storage of the user behavior logs can be achieved, real-time query of the user behavior logs can be achieved by establishing a virtual view chart in the second database, and the virtual view chart is used for being associated with the real-time log table and the offline log table, so that the real-time storage and real-time query statistics of the logs can be achieved while the mass logs are stored.
Description
Technical Field
The present application relates to the field of big data and cloud computing in computer technology, and more particularly to distributed storage and intelligent search. The application provides a method, a device, equipment and a storage medium for log processing.
Background
With the rapid development of internet technology, various services of the internet have become an indispensable part of people's lives. A large amount of user behavior logs are generated by various application services such as games, e-commerce, verticality, feed flow, search, instant messaging and the like every day, and analysis of the user behavior logs can help enterprises to understand users better, recommend more appropriate information to meet user requirements, and can also meet data statistics requirements of the enterprises.
The conventional real-time storage system can not store too much data, a distributed storage system capable of storing very much data cannot insert query data in real time, the conventional real-time or quasi-real-time log stream system has to abandon detailed data, only stores a result data set or a castrated detailed data set with a small number of dimensions, and has low availability.
Disclosure of Invention
The application provides a method, a device, equipment and a storage medium for log processing.
According to an aspect of the present application, there is provided a method of log processing, including:
storing the user behavior log into a real-time log table of a first database;
storing the user behavior log in the previous time period into an offline log table of a second database at intervals of a time period, and deleting the user behavior log in the previous time period in the real-time log table;
and establishing a virtual view chart in the second database, wherein the virtual view chart is used for associating the real-time log sheet and the off-line log sheet.
According to another aspect of the present application, there is provided an apparatus for log processing, including:
the real-time data processing module is used for storing the user behavior log into a real-time log table of the first database;
the offline data processing module is used for storing the user behavior log in the previous time period into an offline log table of a second database at intervals of one time period and deleting the user behavior log in the previous time period from the real-time log table;
and the virtual view module is used for establishing a virtual view table in the second database, and the virtual view table is used for associating the real-time log table and the offline log table.
According to another aspect of the present application, there is provided an electronic device including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method described above.
According to another aspect of the present application, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method described above.
According to the technology of the application, the storage of a large amount of user behavior logs is realized, and meanwhile, the real-time storage and the real-time query of the user behavior logs are realized.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present application, nor do they limit the scope of the present application. Other features of the present application will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
FIG. 1 is a flowchart of a method for log processing according to a first embodiment of the present application;
FIG. 2 is a flowchart of a method for log processing according to a second embodiment of the present application;
fig. 3 is an overall framework diagram of a processing method of a dotting log according to a second embodiment of the present application;
FIG. 4 is a schematic diagram of error real-time monitoring information for an applet as provided herein;
FIG. 5 is a diagram of an apparatus for log processing according to a third embodiment of the present application;
FIG. 6 is a diagram of an apparatus for log processing according to a fourth embodiment of the present application;
fig. 7 is a block diagram of an electronic device for implementing a method of log processing according to an embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Currently, the mainstream real-time or quasi-real-time log stream schemes include the following: (1) the scheme comprises the following steps of elastic search, Logstash and Kibana, wherein the Logstash acquires data, ES stores the data, and the Kibana displays the data; (2) according to the scheme of Flume + Kafka + Storm + Redis + MySQL, Flume acquires data, Kafka serves as a message queue to transmit data, Storm analyzes the data, a result data set is stored in Redis, and MySQL serves as data persistence storage; (3) the method comprises the following steps of a scheme of Flume + Kafka + Spark Stream + Redis + Hbase, Spark Stream streaming parsing data, a Redis storage result data set, and a Hbase storage result data set with a small number of dimensions.
The current mainstream real-time or quasi-real-time log stream scheme stores the following problems: 1) the data reliability is not high. The log data is subjected to a great number of process links from collection to queue to cleaning to transmission to storage, the more links, the higher the possibility of errors, such as message queue blockage, and the incorrect data seen downstream due to missing. 2) The problem of troubleshooting is costly. Log data is transmitted to storage from the time of collecting to the time of cleaning to the time of storing, a tool link is long, and the storage generally only stores a result data set and does not store a detail data set. 3) The cost of operation and maintenance is very high. Due to the fact that a large number of systems are used, such as message queues, once a block or a fault is required to be processed in time, otherwise, downstream real-time data display is not required, a set of monitoring system needs to be set up for the log system in order to guarantee data, the system cannot depend on the log system, even if a problem is found, time is required for processing completion, and the downstream real-time data display still has the problem in the time. 4) The amount of data stored is limited. The daily log magnitude of a service system is generally large, basically hundred-million level, and the occupied storage is large, wherein each day is hundred G or T level, but a conventional real-time storage system cannot store too much data, and a distributed storage system capable of storing very much data cannot insert data in real time, so that the conventional scheme has to discard detailed data and only stores a result data set or a small-dimension castration detailed data set.
The application provides a log processing method, a device, equipment and a storage medium, which are applied to the technical field of big data and cloud computing in computer technology, and more particularly relate to the fields of distributed storage, intelligent search, data statistics, data analysis and the like so as to realize the storage of massive user behavior logs and simultaneously realize the real-time storage and real-time query of the user behavior logs.
The log processing method provided by the application can be at least applied to the following scenes:
one application scenario is an error real-time monitoring system, for example, for monitoring in real-time whether an application is in error while the application is online. By inquiring the error log in real time, the error log can be reported immediately when being inquired, and the detailed information of the error log is displayed, so that the information of on-duty research personnel can quickly find and locate problems.
The other application scenario is data real-time query statistics, for example, real-time query and statistics of error quantity, error logs under the statistics are queried in real time when the data is online each time, not only can the total quantity change of error data be checked, but also real-time error details can be checked, the positioning problem of research and development personnel can be conveniently checked, and major accidents caused by online are avoided. As another example, real-time query and statistical traffic data, etc.
Fig. 1 is a flowchart of a method for processing a log according to a first embodiment of the present application. The method of this embodiment may be executed by a log processing apparatus, where the log processing apparatus may be a client or a server cluster (hereinafter, collectively referred to as "electronic device") with certain computing power, such as a desktop computer, a tablet computer, a notebook computer, or the like, or the apparatus may also be a chip in the electronic device, and the present embodiment is not limited in particular herein.
As shown in fig. 1, the method comprises the following specific steps:
and step S101, storing the user behavior log into a real-time log table of a first database.
The first database can be a relational database such as MySQL, and a real-time log table in the first database is used for storing user behavior logs in real time.
In the embodiment, the user behavior logs collected in real time are inserted into the real-time log table of the first database in real time, due to the fact that no complex logic exists, no updating and query operations exist in the middle, sequential simple batch insertion operations are adopted, performance is very fast, time consumption is in millisecond level, and therefore the user behavior logs can be stored into the real-time log table of the first database in real time, and real-time insertion of the logs is achieved.
In addition, the user behavior log in this embodiment may be a massive dotting log or other log data, and this embodiment is not limited in this respect.
And S102, storing the user behavior log in the previous time period into an offline log table of a second database at intervals of one time period, and deleting the user behavior log in the previous time period in the real-time log table.
The second database may be a database such as Doris capable of storing mass data and a database supporting multidimensional analysis of data, and an offline log table in the second database is used for storing a user behavior log.
In this embodiment, after the user behavior log collected in real time is inserted into the real-time log table of the first database in real time, the user behavior log may be synchronized into the second database in time intervals. Specifically, at intervals of a time period, importing a user behavior log in the previous time period into an offline log table of a second database; and after the user behavior log in the previous time period is completely imported, deleting the user behavior log in the previous time period from the real-time log table of the first database. The log data is either in a real-time log table of the first database or in an offline log table of the second database. Therefore, the first database only needs to record a small amount of recent user behavior logs, and the data volume of real-time data required to be stored in the first database is greatly reduced.
Step S103, a virtual view chart is established in the second database, and the virtual view chart is used for associating the real-time log sheet and the off-line log sheet.
In the embodiment, in order to realize real-time data query, the virtual view table capable of associating the real-time log table and the offline log table is established in the second database, and a user can directly query the virtual view table when querying the log, so that query of the real-time log table and the offline log table can be realized, the user cannot perceive offline storage of the user log, and all user behavior logs can be accessed in real time.
The embodiment of the application realizes the real-time storage of the user behavior log by storing the user behavior log into a real-time log table of a first database; the user behavior logs in the previous time period are stored in the offline log table of the second database at intervals of a time period, the user behavior logs in the previous time period in the real-time log table are deleted, mass storage of the user behavior logs can be achieved, real-time query of the user behavior logs can be achieved by establishing a virtual view chart in the second database, and the virtual view chart is used for being associated with the real-time log table and the offline log table, so that the real-time storage and real-time query statistics of the logs can be achieved while the mass logs are stored.
Fig. 2 is a flowchart of a method for processing logs according to a second embodiment of the present application. Based on the first embodiment, in this embodiment, the method flow of log processing is exemplarily described in detail by taking the first database as MySQL and the second database as Doris as an example, in other embodiments of the present application, the first database and the second database may also be implemented by using other databases, which is not specifically limited in this embodiment.
MySQL is a relational database management system, is the most popular open source database, is basically used by the first-line back-end services of the Internet companies, is extremely low in learning and using cost, has professional operation and maintenance personnel for each Internet company or service team, is extremely high in stability and reliability, and basically cannot be down and fail because once down, the Internet service is unavailable. MySQL also has drawbacks, however, such as hundreds of millions of data queries per table store that can be poor performing.
Doris is an interactive SQL data warehouse based On Massively Parallel Processing (MPP), and the main objective is to support stable, online, interactive data Reporting (Reporting) and data multidimensional analysis services, such as OLAP (On-Line Analytical Processing) services. Doris supports two levels of data partitioning. For example, in this embodiment, time may be used as a first-layer partition of offline log data, and then bucket partitioning may be performed according to a hash value of a user identifier (e.g., a user id) to ensure that data is uniformly distributed on each Doris distributed data server. However, like most OLAP data, Doris does not support real-time insertion of data, and only supports low-frequency block data import.
In this embodiment, a real-time log table is established in the MySQL database by using the characteristics of MySQL and Doris, an offline log table is established in the Doris database, the real-time log table is mapped to Doris, and the real-time log table and the offline log table are combined through a Doris virtual view table. The real-time log table supports high-concurrency real-time insertion, and is short in storage time and small in relative data volume. The data volume of the historical data stored in the offline log table is huge, a partition of date dimension can be established, and then hash barrel storage is carried out according to user dimension, so that the problem of large data volume storage is solved; the virtual view table is a set of a real-time log table and an off-line log table, and real-time query and statistics of massive logs are realized based on the virtual view table.
As shown in fig. 2, the method comprises the following specific steps:
step S201, a user behavior log is acquired in real time.
In this embodiment, the user behavior log may be a mass dotting log or other log data, and this embodiment is not limited specifically here.
Illustratively, user behavior data is acquired in real time, and first processing is performed on the user behavior data to obtain a corresponding user behavior log.
For example, the first processing performed on the user behavior data may be filtering and converting some fields in the user behavior data, filtering out dirty data, and the like, so as to extract data information that needs to be recorded in the user behavior log, and generate a corresponding user behavior log.
In an optional implementation manner, for the dotting log, the gif picture of one pixel point is adopted by the client or the front end to acquire the user behavior data. For example, when a user browses or clicks an event, a gif request is sent to the back end through the client or the front end, and the gif request records user behavior data such as the identity, a request behavior and a request page of the user, so that the function of the data acquisition module is realized.
And transmitting the gif request of the user into a Nginx server of the server through the network, and configuring the Nginx server to forward the gif request to the server interface. Wherein the server side interface can be a server side web interface written in PHP or other languages. And after receiving the gif request, the server-side interface performs first processing on the user behavior data to obtain a corresponding user behavior log. Then, through step S202, the user behavior log is inserted into the real-time log table of MySQL. Due to the fact that no complex logic exists, the query and update operations cannot occur in the middle of the insertion operation, and the insertion operation is a sequential simple batch insertion operation, performance is very fast, and the whole process from the step of receiving the gif request to the step of inserting the user behavior data in the request into the database table takes about 10ms and is in the millisecond level. And the work of real-time data acquisition and storage is finished.
In other optional embodiments, any manner of acquiring the user behavior log in real time in the prior art may be adopted to acquire the user behavior log in real time, which is not described herein again.
Step S202, storing the user behavior log into a real-time log table of a first database.
After the user behavior log is acquired, the user behavior log can be inserted into the MySQL real-time log table in real time, so that the user behavior log can be stored in real time.
And step S203, storing the user behavior log in the previous time period into an offline log table of the second database at intervals of one time period, and deleting the user behavior log in the previous time period in the real-time log table.
In this embodiment, the step may be specifically implemented as follows:
transmitting the user behavior log in the previous time period to a distributed file system at intervals of one time period; the user behavior logs are imported into an offline log table of a second database from the distributed file system, the user behavior logs can be stored in the offline log table of the second database, and the storage of massive user behavior logs is achieved through the second database.
The time period for transmitting the user behavior log interval may be configured and adjusted according to an actual application scenario, and this embodiment is not specifically limited herein.
For example, it may be configured to transmit a log of user behavior over a period of the previous hour every one hour apart.
Optionally, before the user behavior log is imported into the offline log table of the second database, data cleaning may be performed on the user behavior log according to a service requirement, so as to reduce unnecessary information according to the service requirement, and improve availability of data in the offline log table.
For example, for the dotting log, after the server interface completes processing of data in the gif request and records the user behavior log, the nginnx server may record the user behavior log of the request in the access log (access _ log). The operation is divided once per hour, a log agent (loggent) deployed by an Nginx server transmits a user behavior log in an access _ log to a distributed data storage HDFS in real time, a transmission task is divided according to the hour, and for the user behavior log in each hour, if all the user behavior logs in the hour are transmitted completely, a corresponding completion mark is marked on the HDFS.
Further, when the user behavior log transmission task in the last hour is completed, the downstream computing task starts to be executed, and the user behavior log in the HDFS is subjected to data cleaning through a hadoop or spark offline computing module and is stored in a new HDFS address. The data cleaning is mainly to screen information in the user behavior logs according to service requirements (such as query requirements, statistical requirements and the like), extract useful data, form new user behavior logs and store the new user behavior logs in new HDFS addresses.
Further, before the user behavior logs are imported into the offline log table of the second database, the offline log table is established in the second database and is used for storing massive user behavior logs.
Illustratively, an offline log table similar to the real-time log table in MySQL is built in Doris, so that the fields stored in the offline log table are basically consistent with the real-time log table, and useful fields and other necessary fields are reserved according to business requirements. The fields included in the offline log table may be configured according to an actual application scenario, and this embodiment is not specifically limited herein.
After the offline log table is established, the user behavior log in the new HDFS address is imported into the offline log table of Doris through a brooker data import module of the Doris database according to the new HDFS address.
The user behavior log in the new HDFS address may be text data or compressed text data, and both may be imported into the offline log table through the broker data import module.
Further, after the last hour of user behavior data is imported into the Doris offline log table, the corresponding last hour of data in the MySQL real-time log table is deleted.
In this embodiment, the task of importing the user behavior log in each time period into the offline log table is executed serially, that is, after the user behavior log in one time period is imported into the offline log table, the user behavior log in the next time period is imported, so that the user behavior log is in the MySQL real-time log table or the Doris offline log table, and MySQL only needs to record a small amount of recent user behavior logs, thereby greatly reducing the data volume of the real-time data required to be stored in the first database; doris enables the storage of massive log data.
Alternatively, a reference time per period (per hour) may be set, which may be a start time or an end time of each period. And according to the reference time of each time segment, executing the tasks of transmitting, processing and importing the user behavior log into an offline log table.
And step S204, establishing a virtual view chart in the second database, wherein the virtual view chart is used for associating the real-time log sheet and the off-line log sheet.
In this embodiment, the step may be specifically implemented as follows:
establishing an association mapping table of a real-time log table in a second database; and establishing a virtual view table in the second database, wherein the virtual view table is used for associating the association mapping table with the offline log table, so that the association of the real-time log table and the offline log table is realized through the virtual view table, the query of the real-time log table and the offline log table can be realized through querying the virtual view table, and the real-time query and statistics of the logs are realized.
Illustratively, first, an association mapping table of the real-time log table in MySQL is established in Doris, and fields of the association mapping table are consistent with fields of the real-time log table in MySQL, and data in the real-time log table in MySQL can be queried through the association mapping table. And then establishing a virtual view table, and associating the MySQL real-time log table with the Doris offline log table, wherein the minimum time of the log in the MySQL real-time log table is greater than the maximum time of the log in the Doris offline log table.
The virtual view table includes a part of common fields in the real-time log table and the offline log table, and may be configured according to a service requirement in a specific application scenario, which is not specifically limited in this embodiment.
Therefore, a user can directly query the virtual view table, and the real-time log table of MySQL and/or the offline log table of Doris can be automatically queried according to the query time range during query. Therefore, the high-availability real-time statistics of the dotting logs is realized by utilizing the characteristics of high availability and real-time insertion of MySQL and the characteristics of mass detail data storage of Doris.
In an optional implementation manner, the real-time log table may be implemented in a manner of separate banks and separate tables. If the real-time log flow of the MySQL is too large and even exceeds 100 million QPS (Query Per Second, request number Per Second), the pressure can be shared by adopting a database-partitioning and even multi-cluster mode, each real-time log table is associated to Doris, and the associated MySQL real-time log tables are all in the virtual view table, so that the problem of low Query performance caused by too large data volume of the MySQL real-time log table is solved.
And S205, responding to the log query request, and querying the virtual view table to obtain a query result.
In this embodiment, the user can query the virtual view table to query and count the user behavior log.
Optionally, if the query time range is not specified in the log query request of the user, all data in the real-time log table and the offline log table are queried through the virtual view table, so as to obtain a corresponding query result.
Optionally, the log query request of the user may include a time range, and in response to the log query request, the real-time log table and/or the offline log table are queried according to the time range in the log query request to obtain a query result, so that the user behavior log can be queried and counted in real time.
For example, if the queried user behavior logs are determined to be in the offline log table according to the time range, the offline log table is queried to obtain a corresponding query result. And if the queried user behavior logs are determined to be in the real-time log table according to the time range, querying the real-time log table to obtain a corresponding query result. And if the queried user behavior log part is determined to be in the offline log table and the other part is determined to be in the real-time log table according to the time range, respectively querying the real-time log table and the offline log table, and synthesizing the query results of the two tables to obtain a final query result.
Alternatively, after the query result is obtained, the query result may be displayed in a graph, text, or the like. For example, a corresponding error statistics graph may be generated according to the query result, and further, detailed information of the error log may be displayed.
Fig. 3 is an overall framework diagram of a method for processing a dotting log according to a second embodiment of the present application, and the flow shown in fig. 3 is a preferred implementation manner, and as shown in fig. 3, the method for processing a dotting log may substantially include the following flows: s1, requesting a server gif file through a front-end point burying, wherein the gif file carries dotting log information; s2, the Nginx server forwards the request information to a server-side interface, the server-side interface analyzes the request information in real time to obtain user behavior information, and the user behavior information is inserted into a MySQL real-time log table; s3, the server-side interface finishes real-time storage, the Nginx server logs are transmitted through the LogAgent, and the Nginx server logs are temporarily stored in the HDFS; s4, reading data in the HDFS through Spark or Hadoop, performing simple field splitting, and writing the data into an offline log table of Doris; and S5, automatically and correlatively reading the MySQL real-time log table and the Doris offline log table by the virtual view table of Doris, and directly accessing the virtual view table by a user, namely, accessing all data without perception.
Fig. 4 is a schematic diagram of error real-time monitoring information of a certain applet provided by the present application, and a front-end dotting established applet error real-time monitoring system can report immediately and display log details if an applet has an error, so that an on-duty research and development worker can quickly find and locate the problem. FIG. 4 illustrates one presentation of the output monitoring information of the system, generating and displaying a 24-hour real-time error statistics line graph, and displaying the error log, e.g., time, path, error name, error information, app version, system version, sdk version, number of errors, etc., of the error log.
In an optional implementation mode, by combining with a real-time data stream, the real-time check of the small program error log can be realized, statistics of the lower error log can be inquired when the small program error log is online at each time, not only can the total amount change of error data be checked, but also the real-time error detail can be checked, the positioning problem of research and development personnel is facilitated, and the major accident caused by online is avoided.
In another optional implementation mode, the problem of delay of data report statistics can be solved by combining with real-time data flow, the problem of delay of data report statistics is easily solved due to the fact that the traditional data report statistics is large in data volume and small-scale data statistics easily fails, and a research and development worker on duty often needs to process the problem anywhere at any time (possibly in the middle of night), but after the scheme is used, data can be greatly guaranteed, even if the fault of the offline log module is delayed for more than ten hours, due to the fact that the offline log module is supported by the real-time log table, sufficient recovery time is reserved for fault recovery of the offline log module.
The scheme has strong universality, the selected technical schemes are all combinations of common schemes, the statistics of the high-available dotting log data stream is realized at very low cost, the expansibility is strong, no problem exists in processing million QPS log streams, and the user can obtain the data of real-time statistics conveniently.
The embodiment of the application realizes the real-time storage of the user behavior log by inserting the user behavior log into the real-time log table of MySQL in real time; storing the user behavior log in the previous time period into an offline log table of Doris at intervals of one time period, and deleting the user behavior log in the previous time period from a real-time log table, so that mass storage of the user behavior log can be realized; by establishing an association mapping table of the real-time log table in MySQL in Doris, the fields of the association mapping table are consistent with the fields of the real-time log table in MySQL, and the data in the real-time log table in MySQL can be inquired through the association mapping table. And then establishing a virtual view table, and associating the MySQL real-time log table with the Doris offline log table, wherein the minimum time of the log in the MySQL real-time log table is greater than the maximum time of the log in the Doris offline log table. The user can directly inquire the virtual view table, and the real-time log table of MySQL and/or the offline log table of Doris can be automatically inquired according to the inquiry time range during inquiry. Therefore, by utilizing the characteristics of high availability and real-time insertion of MySQL and the characteristics of mass detail data storage of Doris, the real-time statistics of the high-availability dotting logs is realized.
Fig. 5 is a schematic diagram of an apparatus for processing logs according to a third embodiment of the present application. The log processing device provided by the embodiment of the application can execute the processing flow provided by the method embodiment of the log processing. As shown in fig. 5, the log processing apparatus 30 includes: a real-time data processing module 301, an offline data processing module 302 and a virtual view module 303.
Specifically, the real-time data processing module 301 is configured to store the user behavior log in a real-time log table of the first database.
The offline data processing module 302 is configured to store, every time interval of one time period, the user behavior log in the previous time period in an offline log table of the second database, and delete the user behavior log in the previous time period in the real-time log table.
The virtual view module 303 is configured to establish a virtual view table in the second database, where the virtual view table is used to associate the real-time log table with the offline log table.
The apparatus provided in this embodiment of the present application may be specifically configured to execute the method embodiment provided in the first embodiment, and specific functions are not described herein again.
The embodiment of the application realizes the real-time storage of the user behavior log by storing the user behavior log into a real-time log table of a first database; the user behavior logs in the previous time period are stored in the offline log table of the second database at intervals of a time period, the user behavior logs in the previous time period in the real-time log table are deleted, mass storage of the user behavior logs can be achieved, real-time query of the user behavior logs can be achieved by establishing a virtual view chart in the second database, and the virtual view chart is used for being associated with the real-time log table and the offline log table, so that the real-time storage and real-time query statistics of the logs can be achieved while the mass logs are stored.
Fig. 6 is a schematic diagram of an apparatus for log processing according to a fourth embodiment of the present application. On the basis of the third embodiment, in this embodiment, the virtual view module is further configured to:
establishing an association mapping table of a real-time log table in a second database; and establishing a virtual view table in the second database, wherein the virtual view table is used for associating the association mapping table with the offline log table.
In an optional implementation, the real-time data processing module is further configured to:
and acquiring a user behavior log in real time.
In an optional implementation, the offline data processing module is further configured to:
transmitting the user behavior log in the previous time period to a distributed file system at intervals of one time period; and importing the user behavior log from the distributed file system into an offline log table of the second database.
In an optional implementation, the offline data processing module is further configured to:
and establishing an offline log table in a second database.
In an optional implementation, the offline data processing module is further configured to:
and according to the service requirement, performing data cleaning on the user behavior log.
In an alternative embodiment, as shown in fig. 6, the log processing apparatus 30 further includes: a real-time query statistics module 304 to:
and responding to the log query request, and querying the virtual view table to obtain a query result.
In an optional implementation, the real-time query statistics module is further configured to:
and responding to the log query request, and querying the real-time log list and/or the offline log list according to the time range in the log query request to obtain a query result.
The apparatus provided in the embodiment of the present application may be specifically configured to execute the method embodiment provided in the second embodiment, and specific functions are not described herein again.
The embodiment of the application realizes the real-time storage of the user behavior log by inserting the user behavior log into the real-time log table of MySQL in real time; storing the user behavior log in the previous time period into an offline log table of Doris at intervals of one time period, and deleting the user behavior log in the previous time period from a real-time log table, so that mass storage of the user behavior log can be realized; by establishing an association mapping table of the real-time log table in MySQL in Doris, the fields of the association mapping table are consistent with the fields of the real-time log table in MySQL, and the data in the real-time log table in MySQL can be inquired through the association mapping table. And then establishing a virtual view table, and associating the MySQL real-time log table with the Doris offline log table, wherein the minimum time of the log in the MySQL real-time log table is greater than the maximum time of the log in the Doris offline log table. The user can directly inquire the virtual view table, and the real-time log table of MySQL and/or the offline log table of Doris can be automatically inquired according to the inquiry time range during inquiry. Therefore, by utilizing the characteristics of high availability and real-time insertion of MySQL and the characteristics of mass detail data storage of Doris, the real-time statistics of the high-availability dotting logs is realized.
According to an embodiment of the present application, an electronic device and a readable storage medium are also provided.
Fig. 7 is a block diagram of an electronic device according to a method of log processing according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 7, the electronic apparatus includes: one or more processors Y01, a memory Y02, and interfaces for connecting the various components, including a high speed interface and a low speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). In fig. 7, one processor Y01 is taken as an example.
Memory Y02 is a non-transitory computer readable storage medium as provided herein. The memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method of log processing provided herein. The non-transitory computer-readable storage medium of the present application stores computer instructions for causing a computer to perform the method of log processing provided herein.
Memory Y02 is a non-transitory computer readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the method of log processing in the embodiments of the present application (e.g., real-time data processing module 301, offline data processing module 302, and virtual view module 303 shown in fig. 5). The processor Y01 executes various functional applications of the server and data processing, i.e., a method of implementing log processing in the above-described method embodiments, by executing non-transitory software programs, instructions, and modules stored in the memory Y02.
The memory Y02 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created from use of the electronic device by log processing, and the like. Additionally, the memory Y02 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory Y02 may optionally include memory located remotely from processor Y01, which may be connected to the log processing electronics via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device of the method of log processing may further include: an input device Y03 and an output device Y04. The processor Y01, the memory Y02, the input device Y03, and the output device Y04 may be connected by a bus or other means, and the bus connection is exemplified in fig. 7.
The input device Y03 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic device being logged, such as a touch screen, keypad, mouse, track pad, touch pad, pointer stick, one or more mouse buttons, track ball, joystick or like input device. The output device Y04 may include a display device, an auxiliary lighting device (e.g., LED), a tactile feedback device (e.g., vibration motor), and the like. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme of the embodiment of the application, the user behavior log is inserted into and stored in the MySQL real-time log table in real time, so that the user behavior log is stored in real time; storing the user behavior log in the previous time period into an offline log table of Doris at intervals of one time period, and deleting the user behavior log in the previous time period from a real-time log table, so that mass storage of the user behavior log can be realized; by establishing an association mapping table of the real-time log table in MySQL in Doris, the fields of the association mapping table are consistent with the fields of the real-time log table in MySQL, and the data in the real-time log table in MySQL can be inquired through the association mapping table. And then establishing a virtual view table, and associating the MySQL real-time log table with the Doris offline log table, wherein the minimum time of the log in the MySQL real-time log table is greater than the maximum time of the log in the Doris offline log table. The user can directly inquire the virtual view table, and the real-time log table of MySQL and/or the offline log table of Doris can be automatically inquired according to the inquiry time range during inquiry. Therefore, by utilizing the characteristics of high availability and real-time insertion of MySQL and the characteristics of mass detail data storage of Doris, the real-time statistics of the high-availability dotting logs is realized.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (18)
1. A method of log processing, comprising:
storing the user behavior log into a real-time log table of a first database;
storing the user behavior log in the previous time period into an offline log table of a second database at intervals of a time period, and deleting the user behavior log in the previous time period in the real-time log table;
and establishing a virtual view chart in the second database, wherein the virtual view chart is used for associating the real-time log sheet and the off-line log sheet.
2. The method of claim 1, wherein said creating a virtual view table in said second database, said virtual view table for associating said live and offline log tables, comprises:
establishing an association mapping table of the real-time log table in the second database;
and establishing the virtual view table in the second database, wherein the virtual view table is used for associating the association mapping table with the offline log table.
3. The method of claim 1, wherein prior to storing the user behavior log in the real-time log table of the first database, further comprising:
and acquiring the user behavior log in real time.
4. The method of claim 1, wherein storing the user behavior log in a previous time period to an offline log table of a second database at intervals of one time period comprises:
transmitting the user behavior log in the previous time period to a distributed file system at intervals of one time period;
and importing the user behavior log from the distributed file system into an offline log table of the second database.
5. The method of claim 4, wherein prior to importing the user behavior log into an offline log table of the second database, further comprising:
and establishing the offline log table in the second database.
6. The method of claim 4, wherein prior to importing the user behavior log into an offline log table of the second database, further comprising:
and according to the service requirement, performing data cleaning on the user behavior log.
7. The method of any of claims 1-6, further comprising:
and responding to the log query request, and querying the virtual view table to obtain a query result.
8. The method of claim 7, wherein said querying the virtual view table for query results in response to log query requests comprises:
responding to a log query request, and querying the real-time log table and/or the offline log table according to the time range in the log query request to obtain a query result.
9. An apparatus of log processing, comprising:
the real-time data processing module is used for storing the user behavior log into a real-time log table of the first database;
the offline data processing module is used for storing the user behavior log in the previous time period into an offline log table of a second database at intervals of one time period and deleting the user behavior log in the previous time period from the real-time log table;
and the virtual view module is used for establishing a virtual view table in the second database, and the virtual view table is used for associating the real-time log table and the offline log table.
10. The apparatus of claim 9, wherein the virtual view module is further to:
establishing an association mapping table of the real-time log table in the second database;
and establishing the virtual view table in the second database, wherein the virtual view table is used for associating the association mapping table with the offline log table.
11. The apparatus of claim 9, wherein the real-time data processing module is further to:
and acquiring the user behavior log in real time.
12. The apparatus of claim 9, wherein the offline data processing module is further to:
transmitting the user behavior log in the previous time period to a distributed file system at intervals of one time period;
and importing the user behavior log from the distributed file system into an offline log table of the second database.
13. The apparatus of claim 12, wherein the offline data processing module is further to:
and establishing the offline log table in the second database.
14. The apparatus of claim 12, wherein the offline data processing module is further to:
and according to the service requirement, performing data cleaning on the user behavior log.
15. The apparatus of any of claims 9-14, further comprising: a real-time query statistics module for:
and responding to the log query request, and querying the virtual view table to obtain a query result.
16. The apparatus of claim 15, wherein the real-time query statistics module is further configured to:
responding to a log query request, and querying the real-time log table and/or the offline log table according to the time range in the log query request to obtain a query result.
17. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.
18. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011023854.3A CN112115114A (en) | 2020-09-25 | 2020-09-25 | Log processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011023854.3A CN112115114A (en) | 2020-09-25 | 2020-09-25 | Log processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112115114A true CN112115114A (en) | 2020-12-22 |
Family
ID=73798713
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011023854.3A Pending CN112115114A (en) | 2020-09-25 | 2020-09-25 | Log processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112115114A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112948461A (en) * | 2021-02-26 | 2021-06-11 | 北京百度网讯科技有限公司 | Method, apparatus, storage medium, and program product for schedule data processing |
| CN113157545A (en) * | 2021-05-20 | 2021-07-23 | 京东方科技集团股份有限公司 | Method, device and equipment for processing service log and storage medium |
| CN113312376A (en) * | 2021-05-21 | 2021-08-27 | 福建天泉教育科技有限公司 | Method and terminal for real-time processing and analysis of Nginx logs |
| CN113641670A (en) * | 2021-07-09 | 2021-11-12 | 北京百度网讯科技有限公司 | Data storage and data retrieval method and device, electronic equipment and storage medium |
| CN113691681A (en) * | 2021-08-20 | 2021-11-23 | 北京琥珀创想科技有限公司 | Junk phone data processing method and system |
| CN114253925A (en) * | 2021-12-01 | 2022-03-29 | 北京人大金仓信息技术股份有限公司 | Method, server, terminal and electronic device for accessing database logs |
| US20240095246A1 (en) * | 2022-09-20 | 2024-03-21 | Beijing Volcano Engine Technology Co., Ltd | Data query method and apparatus based on doris, storage medium and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105933736A (en) * | 2016-04-18 | 2016-09-07 | 天脉聚源(北京)传媒科技有限公司 | Log processing method and device |
| CN108733724A (en) * | 2017-04-24 | 2018-11-02 | 北京京东尚科信息技术有限公司 | One kind is across the real-time connection method of heterogeneous data source and device |
| CN111625600A (en) * | 2020-05-21 | 2020-09-04 | 杭州安恒信息技术股份有限公司 | Data storage processing method, system, computer equipment and storage medium |
-
2020
- 2020-09-25 CN CN202011023854.3A patent/CN112115114A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105933736A (en) * | 2016-04-18 | 2016-09-07 | 天脉聚源(北京)传媒科技有限公司 | Log processing method and device |
| CN108733724A (en) * | 2017-04-24 | 2018-11-02 | 北京京东尚科信息技术有限公司 | One kind is across the real-time connection method of heterogeneous data source and device |
| CN111625600A (en) * | 2020-05-21 | 2020-09-04 | 杭州安恒信息技术股份有限公司 | Data storage processing method, system, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 秦敬辉主编: "Delphi程序设计教程", 31 January 2003, 北京：中国电力出版社, pages: 205 - 206 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112948461A (en) * | 2021-02-26 | 2021-06-11 | 北京百度网讯科技有限公司 | Method, apparatus, storage medium, and program product for schedule data processing |
| CN112948461B (en) * | 2021-02-26 | 2024-05-10 | 北京百度网讯科技有限公司 | Method, apparatus, storage medium and program product for calendar data processing |
| CN113157545A (en) * | 2021-05-20 | 2021-07-23 | 京东方科技集团股份有限公司 | Method, device and equipment for processing service log and storage medium |
| CN113312376A (en) * | 2021-05-21 | 2021-08-27 | 福建天泉教育科技有限公司 | Method and terminal for real-time processing and analysis of Nginx logs |
| CN113641670A (en) * | 2021-07-09 | 2021-11-12 | 北京百度网讯科技有限公司 | Data storage and data retrieval method and device, electronic equipment and storage medium |
| CN113641670B (en) * | 2021-07-09 | 2023-08-11 | 北京百度网讯科技有限公司 | Data storage and data retrieval method and device, electronic equipment and storage medium |
| CN113691681A (en) * | 2021-08-20 | 2021-11-23 | 北京琥珀创想科技有限公司 | Junk phone data processing method and system |
| CN114253925A (en) * | 2021-12-01 | 2022-03-29 | 北京人大金仓信息技术股份有限公司 | Method, server, terminal and electronic device for accessing database logs |
| US20240095246A1 (en) * | 2022-09-20 | 2024-03-21 | Beijing Volcano Engine Technology Co., Ltd | Data query method and apparatus based on doris, storage medium and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112115114A (en) | Log processing method, device, equipment and storage medium | |
| CN108073625B (en) | System and method for metadata information management | |
| JP2019503525A (en) | Event batch processing, output sequencing, and log-based state storage in continuous query processing | |
| CN111966677B (en) | Data report processing method and device, electronic equipment and storage medium | |
| CN111045985B (en) | File storage processing method, server, electronic device and storage medium | |
| CN111586126B (en) | Applet pre-downloading method, device, equipment and storage medium | |
| CN113051446A (en) | Topological relation query method, device, electronic equipment and medium | |
| CN109063196A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN111666280B (en) | Comment ordering method, comment ordering device, comment ordering equipment and computer storage medium | |
| CN105144155A (en) | Visually representing queries of multi-source data | |
| CN112732663A (en) | Log information processing method and device | |
| CN112052134A (en) | Service data monitoring method and device | |
| CN112395333B (en) | Method, device, electronic equipment and storage medium for checking data abnormality | |
| CN115335821B (en) | Offloading statistics collection | |
| US11442930B2 (en) | Method, apparatus, device and storage medium for data aggregation | |
| CN112328658A (en) | User profile data processing method, device, equipment and storage medium | |
| CN103595960A (en) | Interesting information monitoring and pushing method and device | |
| CN113760242B (en) | Data processing method, device, server and medium | |
| CN111726402B (en) | User behavior data processing method and device, electronic equipment and storage medium | |
| US8793272B2 (en) | Query transformation | |
| CN112527620A (en) | Database performance analysis method and device, electronic equipment, medium and product | |
| EP2830289B1 (en) | Adaptable reporting in a multi-client telecommunication network | |
| CN110580322B (en) | Independent visitor information processing method and device, electronic equipment and storage medium | |
| CN114168624B (en) | Data analysis method, computing device and storage medium | |
| CN111698109A (en) | Method and device for monitoring log |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |