CN109299225A - Log searching method, system, terminal and computer readable storage medium - Google Patents
Log searching method, system, terminal and computer readable storage medium Download PDFInfo
- Publication number
- CN109299225A CN109299225A CN201811246089.4A CN201811246089A CN109299225A CN 109299225 A CN109299225 A CN 109299225A CN 201811246089 A CN201811246089 A CN 201811246089A CN 109299225 A CN109299225 A CN 109299225A
- Authority
- CN
- China
- Prior art keywords
- cluster
- interim
- daily record
- log
- record data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000008676 import Effects 0.000 claims abstract description 24
- 238000004590 computer program Methods 0.000 claims description 22
- 238000013508 migration Methods 0.000 claims description 10
- 230000005012 migration Effects 0.000 claims description 10
- 238000012544 monitoring process Methods 0.000 abstract description 7
- 238000009434 installation Methods 0.000 description 18
- 238000013507 mapping Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 11
- 238000012545 processing Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 238000007792 addition Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000012216 screening Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 101100018928 Drosophila melanogaster InR gene Proteins 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of log searching method, system, terminal and computer readable storage medium.The log searching method includes: to store the specified daily record data in ELK log system to a distributed file system from ES cluster backup；A Kubernetes cluster is created, the Kubernetes cluster is independently of the ES cluster and the distributed file system；It receives to be directed to and store to the inquiry request of the specified daily record data of the distributed file system, and the Kubernetes cluster is called to create an interim ES cluster；And the specified daily record data for storing the distributed file system imports the interim ES cluster, to execute the inquiry request by the interim ES cluster.The present invention is based on distributed deployments to store specified daily record data to HDFS, by creating and carrying out log data searching operation using interim ES cluster, and after the completion of monitoring user search operation, discharges the interim ES cluster of creation, is not take up system resource.
Description
Technical field
The present invention relates to daily record data process field more particularly to a kind of log searching method, system, terminal and computers
Readable storage medium storing program for executing.
Background technique
This part intends to provides background for the embodiments of the present invention stated in claims and specific embodiment
Or context.Description herein recognizes it is the prior art not because not being included in this section.
At present most of the log system of enterprise be all based on ELK (Elasticsearch, Logstash and Kibana's
It referred to as) develops, for certain important daily record datas, user can be selected the important data in this part from ELK's
Backed up in Elasticsearch (ES) cluster distributed file system (Hadoop Distributed File System,
HDFS), again by the data guiding back ES cluster in HDFS when needing to inquire, but the load of ES cluster is usually all compared
Height, and the space stored is limited, once carrying out the importing of data, largely can cause shadow to other users on platform
Ring, will lead to when serious entirely service it is unavailable.
Summary of the invention
In view of above-mentioned, the present invention provides a kind of log searching method, system, terminal and computer readable storage medium,
It may be implemented in daily record data importing process on original ES cluster without influence, ensure that the stability of ELK system.
One embodiment of the application provides a kind of log searching method, which comprises
Specified daily record data in ELK log system is stored from ES cluster backup to a distributed file system；
A Kubernetes cluster is created, wherein ES of the Kubernetes cluster independently of the ELK log system
Cluster and the distributed file system；
It receives to be directed to and store to the inquiry request of the specified daily record data of the distributed file system, and described in calling
Kubernetes cluster creates an interim ES cluster；And
The specified daily record data that the distributed file system is stored imports the interim ES cluster, to be faced by described
When ES cluster execute the inquiry request.
Preferably, the specified daily record data by ELK log system is stored from ES cluster backup to a distributed text
The step of part system includes:
The specified daily record data in the ELK log system is stored from the ES cluster backup using snapshot migration algorithm
To the distributed file system；Or
An ES-Hadoop frame is established, and will be specified in the ELK log system using the ES-Hadoop frame
Daily record data is stored from the ES cluster backup to the distributed file system.
Preferably, the interim ES cluster is independently of the ES cluster in the ELK log system.
Preferably, described the step of calling the Kubernetes cluster to create an interim ES cluster, includes:
Acquisition is stored to the data volume size of the specified daily record data of the distributed file system；And
It include the interim ES cluster of corresponding node quantity according to the creation of the data volume size of the specified daily record data；
Wherein, the interim ES cluster includes an at least ES-Client node, at least an ES-Data node and at least one
ES-Master node.
Preferably, the specified daily record data that the distributed file system is stored imports the interim ES cluster
Step includes:
An ES-Hadoop frame is established, and stored the distributed file system using the ES-Hadoop frame
Specified daily record data imports the interim ES cluster.
Preferably, the specified daily record data that the distributed file system is stored imports the interim ES cluster,
To include the step of executing the inquiry request by the interim ES cluster；
The specified daily record data that the distributed file system is stored imports the interim ES cluster, and imports in data
The address of service of the interim ES cluster is returned to after the completion of operation；And
The interim ES cluster is linked to according to the address of service, to execute the inquiry by the interim ES cluster
Request.
Preferably, the specified daily record data that the distributed file system is stored imports the interim ES cluster,
After the step of executing the inquiry request by the interim ES cluster further include:
Monitor whether the operation that the interim ES cluster executes the inquiry request terminates；And
When the operation of the inquiry request has terminated, the interim ES cluster is discharged.
One embodiment of the application provides a kind of log searching system, the system comprises:
Backup module, for storing the specified daily record data in ELK log system from ES cluster backup to a distribution
File system；
First creation module, for creating a Kubernetes cluster, wherein the Kubernetes cluster is independently of institute
State ELK log system ES cluster and the distributed file system；
Second creation module is stored for receiving to be directed to the inquiry of the specified daily record data of the distributed file system
Request, and the Kubernetes cluster is called to create an interim ES cluster；And
Execution module, the specified daily record data for storing the distributed file system import the interim ES collection
Group, to execute the inquiry request by the interim ES cluster.
One embodiment of the application provides a kind of terminal, and the terminal includes processor and memory, on the memory
Several computer programs are stored with, are realized when the processor is for executing the computer program stored in memory such as front institute
The step of log searching method stated.
One embodiment of the application provides a kind of computer readable storage medium, is stored thereon with computer program, described
The step of log searching method as elucidated before is realized when computer program is executed by processor.
Above-mentioned log searching method, system, terminal and computer readable storage medium, by creating independently of HDFS, ES
The Kubernetes cluster of cluster, and calling Kubernetes cluster according to user search demand is user's creation and the ES collection
The completely isolated interim ES cluster of group, and backup log data on HDFS are imported into the interim ES collection automatically after the completion of creation
In group, so that user can carry out log data searching operation on the interim ES cluster, and monitoring user's
After the completion of search operaqtion, the interim ES cluster of creation is discharged, so that whole process does not influence original ES cluster, what is be created that is interim
ES cluster is only effective to active user, will not occupying system resources, ensure that the uniqueness of user's operation, entire ELK days aspiration
Resource of uniting has good allocability, does not influence the operating experience of other users.
Detailed description of the invention
It, below will be to required in embodiment description in order to illustrate more clearly of the technical solution of embodiment of the present invention
The attached drawing used is briefly described, it should be apparent that, the accompanying drawings in the following description is some embodiments of the present invention, for
For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other
Attached drawing.
Fig. 1 is the step flow chart of log searching method in one embodiment of the invention.
Fig. 2 is the step flow chart of log searching method in another embodiment of the present invention.
Fig. 3 is the functional block diagram of log searching system in one embodiment of the invention.
Fig. 4 is computer schematic device in one embodiment of the invention.
Specific embodiment
To better understand the objects, features and advantages of the present invention, with reference to the accompanying drawing and specific real
Applying mode, the present invention will be described in detail.It should be noted that in the absence of conflict, presently filed embodiment and reality
The feature applied in mode can be combined with each other.
In the following description, numerous specific details are set forth in order to facilitate a full understanding of the present invention, described embodiment
Only some embodiments of the invention, rather than whole embodiments.Based on the embodiment in the present invention, this field
Those of ordinary skill's every other embodiment obtained without making creative work, belongs to guarantor of the present invention
The range of shield.
Unless otherwise defined, all technical and scientific terms used herein and belong to technical field of the invention
The normally understood meaning of technical staff is identical.Term as used herein in the specification of the present invention is intended merely to description tool
The purpose of the embodiment of body, it is not intended that in the limitation present invention.
Preferably, log searching method of the invention is applied in one or more computer installation.The computer
Device is that one kind can be according to the instruction for being previously set or storing, the automatic equipment for carrying out numerical value calculating and/or information processing,
Hardware includes but is not limited to microprocessor, specific integrated circuit (Application Specific Integrated
Circuit, ASIC), programmable gate array (Field-Programmable Gate Array, FPGA), digital processing unit
(Digital Signal Processor, DSP), embedded device etc..
The computer installation can be the calculating such as desktop PC, laptop, tablet computer, server and set
It is standby.The computer installation can carry out people by modes such as keyboard, mouse, remote controler, touch tablet or voice-operated devices with user
Machine interaction.
Embodiment one:
Fig. 1 is the step flow chart of log searching method preferred embodiment of the present invention.The process according to different requirements,
The sequence of step can change in figure, and certain steps can be omitted.
As shown in fig.1, the log searching method specifically includes following steps.
Step S11, the specified daily record data in ELK log system is stored from ES cluster backup to a distributed field system
It unites (HDFS).
It in one embodiment, include ES cluster, Logstash frame and Kibana frame in the ELK log system
Frame.The Logstash frame can be distributed on each node of ES cluster, for collecting correlation log data, and through excessive
ES cluster is sent to after analysis, filtering to be stored, then is presented to the user daily record data by Kibana frame, for example provide a variety of
API is for user query, operation.The specified daily record data can be the log number that the user in ELK log system artificially specifies
According to being also possible to according to presetting the daily record data that screens of screening conditions.The default screening conditions can be specified pass through
Period, keyword etc. are screened.For example, utilizing instruction: sed-n'/2014-12-17 16:17:20/ ,/2014-12-17
16:17:36/p'test.log finds out the daily record data of 2014-12-17 16:17:20~2014-12-17 16:17:36；
Utilizing instruction cat-n test.log | grep " insurance " obtains the line number of the daily record data comprising keyword " insurance ".
The HDFS is used for storage and management file.File in HDFS is physically piecemeal storage (block),
The size of block can provide by configuration parameter (dfs.blocksize), default size 128M.HDFS file system
Can provide a unified abstract directory tree, client can access file by specified path, such as: hdfs: //
namenode:port/dir-a/dir-b/dir-c/file.data.Bibliographic structure and file block location information (metadata)
Management undertaken by name-node node, name-node is HDFS cluster host node, is responsible for the entire hdfs file system of maintenance
Directory tree and each path (file) corresponding to data block information (block identification and the data-node at place service
Device).The storage management of each block of file is undertaken by data-node node, and data-node is HDFS cluster from node, often
One block can store multiple copies on multiple data-node, and (copy amount can also pass through parameter setting
3) dfs.replication, default are.Data-node can periodically report the file block itself saved to Name-node
Information, and name-node can then be responsible for keeping the copy amount of file, it can be by name-node when requesting access to HDFS
Apply to carry out.
In one embodiment, the specified daily record data can be backed up from ES cluster in such a way that snapshot migrates
It stores to the HDFS, specifically: adding a repository-hdfs plug-in unit in ES cluster and modify matching for each node
Set file；A warehouse is established, for creating and storing snapshot, wherein the ES cluster can create multiple warehouses, Ke Yixuan
A specified warehouse creation snapshot is selected, may include multiple indexes inside each snapshot, default is the entire ES cluster of backup
Index also can specify the index for only backing up specified daily record data, then restore snapshot in the HDFS, can be realized will refer to
Determine daily record data to store from ES cluster backup to the HDFS.
In one embodiment, it can also be realized by establishing an ES-Hadoop frame by the finger in ELK log system
Determine daily record data to store from ES cluster backup to the HDFS, specifically: created in ES cluster document (in ES cluster data with
Document is that basic unit is stored), the document includes the information such as index, type, id, and index is used to indicate document ownership
Indexed in which, type is used to indicate which type document belongs to, and id is the identifier of document, then create from ES cluster to
The value of the Data Migration mapping table of Hadoop, the key of the mapping table parameter is the id value of document in ES cluster, and input parameter is
The content of document in ES cluster；Create the Job class of Data Migration from ES cluster to Hadoop, the Job class can will be from ES
Data are read in cluster, and are converted into the input parameter of the mapping table, are finally started MapReduce task and are realized specified day
Will data are stored from ES cluster backup to HDFS.
Step S12, a Kubernetes cluster is created, wherein the Kubernetes cluster is independently of the ELK log
The ES cluster and the HDFS of system.
In one embodiment, the Kubernetes cluster can be created in ELK log system.Specifically, it creates
Kubernetes cluster can be accomplished by the following way: setting creates cloud virtual machine used in the Kubernetes cluster
Quantity and specification, and complete the creation of required cloud virtual machine；Obtain the IP information and ssh (Secure Shell, peace of cloud virtual machine
Full shell protocol) information；Binary file required for deployment Kubernetes cluster is copied to created using ssh tool
On the cloud virtual machine, and Kubernetes collection swarm parameter is set；Finally Kubernetes cluster is disposed using kubectl tool
All components.
In one embodiment, the ES cluster in the Kubernetes cluster and the HDFS, the ELK log system
Mutually decoupling, i.e., the described Kubernetes cluster, the ES cluster, the HDFS are mutually indepedent.
Step S13, it receives to be directed to and store to the inquiry request of the specified daily record data of the HDFS, and described in calling
Kubernetes cluster creates an interim ES cluster.
It in one embodiment, can be in ELK days aspirations when user, which wants inquiry, has backed up to the daily record data of HDFS
The essential information of Backup Data is filled on the page of system and issues inquiry request, and the ELK log system can call
Kubernetes cluster is that user creates an interim ES cluster, and the interim ES cluster is independently of the ES collection of ELK log system
Group.
In one embodiment, during carrying out the creation of interim ES cluster, it can first obtain and store to the HDFS
Specified daily record data data volume size, and according to the data volume size of the specified daily record data creation include corresponding node
The interim ES cluster of quantity.In other words, it if the data volume of specified daily record data is bigger, needs to create for the interim ES cluster
Greater number of ES-Client node, ES-Data node and ES-Master node, if the data volume of specified daily record data is got over
It is small, small number of ES-Client node, ES-Data node and ES-Master section can be created for the interim ES cluster
Point.It is understood that the interim ES cluster includes at least an ES-Client node, an ES-Data node and an ES-
Master node.
In one embodiment, ELK log system has the function of authority managing and controlling, the ELK log power that each user can check
Limit is different, and the data volume that user can back up to the daily record data of HDFS is also different.The ELK log system is calling
The interface of Kubernetes cluster is that can previously back up to HDFS's according to user during user creates interim ES cluster
Daily record data amount size creates the ES-Client node, ES-Data node and ES-Master node of respective numbers.Difference is used
Log amount size in family corresponds to the ES-Client node, ES-Data node and ES-Master node of different number.
The ES-Data node is mainly used for storing index data, can change to document progress additions and deletions and look into operation, converging operation
Deng.The ES-Master node such as creates or deletes index for carrying out the relevant content of cluster operation, which node tracked
It is a part of cluster, and determines which fragment distributes to relevant node.The ES-Client node tunable ES-
Master node and ES-Data node, the ES-Client node are added the state of available cluster after cluster and can bases
The state direct routing of cluster is requested.When ES-Master node and the configuration of ES-Data node are both configured to false,
ES-Client node can handle route requests, processing search, the operation such as distribution index.
Step S14, the specified daily record data for storing the distributed file system imports the interim ES cluster, with logical
It crosses the interim ES cluster and executes the inquiry request.
In one embodiment, it again may be by establishing an ES-Hadoop frame, and utilize the ES-Hadoop frame
The specified daily record data that frame stores the distributed file system imports the interim ES cluster.When completing the distribution
After the specified daily record data of file system storage imports the interim ES cluster, the address of service of the interim ES cluster is returned to,
And then user can link to the interim ES cluster according to the address of service, described in being executed by the interim ES cluster
Inquiry request.
In one embodiment, it can be accomplished by the following way and establish an ES-Hadoop frame, and described in utilization
The specified daily record data that the distributed file system stores is imported the interim ES cluster by ES-Hadoop frame: being created first
The mapping table from Hadoop to the Data Migration of the interim ES cluster is built, the input of the mapping table is a HDFS file (institute
Stating HDFS file includes daily record data to be imported), the output of the mapping table the result is that the Text of a json format；So
The Job class from Hadoop to the Data Migration of the interim ES cluster is created afterwards, and the Job class can be by the defeated of MapReduce
Result (Text of a json format) is converted to the id of the interim ES cluster and the content of document out；Finally start
Data are imported the interim ES cluster from HDFS by MapReduce task.
In one embodiment, it can also be accomplished by the following way and be specified what the distributed file system stored
Daily record data imports the interim ES cluster: being first loaded into specified daily record data in Spark SQL and with RDD from HDFS
The storage of (set of distributed Java object) form；Then addition preset data structural information to be to update the RDD, and according to
Updated RDD creates DataFrame (set of distributed Row object), to be connected to the interim ES cluster；Finally by
DataFrame creation index, and the interim ES cluster is written.
Please refer to Fig. 2, compared with log searching method shown in fig. 1, Fig. 2 shows log searching method further include
Step S15 and step S16.
Step S15, monitors whether the operation that the interim ES cluster executes the inquiry request terminates；
Step S16 discharges the interim ES cluster when the operation of the inquiry request has terminated.
In one embodiment, when ELK log system monitors that user completes daily record data on the interim ES cluster
Inquiry operation after, discharge the interim ES cluster, and then can be created that the interim ES cluster only to active user
Effectively, will not occupying system resources, ensure that the uniqueness of user's operation, entire ELK log system resource have it is good can
Distributivity does not influence the operating experience of other users.When the inquiry request for receiving the daily record data for HDFS storage again
When, need to re-create interim ES cluster to realize data query operation.
Embodiment two:
Fig. 3 is the functional block diagram of log searching system preferred embodiment of the present invention.
As shown in fig.3, the log searching system 10 may include backup module 101, the first creation module 102,
Two creation modules 103, execution module 104, monitoring modular 105 and release module 106.
The backup module 101 from ES cluster backup for storing the specified daily record data in ELK log system to one
Distributed file system (HDFS).
It in one embodiment, include ES cluster, Logstash frame and Kibana frame in the ELK log system
Frame.The Logstash frame can be distributed on each node of ES cluster, for collecting correlation log data, and through excessive
ES cluster is sent to after analysis, filtering to be stored, then is presented to the user daily record data by Kibana frame, for example provide a variety of
API is for user query, operation.The specified daily record data can be the log number that the user in ELK log system artificially specifies
According to being also possible to according to presetting the daily record data that screens of screening conditions.The default screening conditions can be specified pass through
Period, keyword etc. are screened.For example, utilizing instruction: sed-n'/2014-12-17 16:17:20/ ,/2014-12-17
16:17:36/p'test.log finds out the daily record data of 2014-12-17 16:17:20~2014-12-17 16:17:36；
Utilizing instruction cat-n test.log | grep " insurance " obtains the line number of the daily record data comprising keyword " insurance ".
The HDFS is used for storage and management file.File in HDFS is physically piecemeal storage (block),
The size of block can provide by configuration parameter (dfs.blocksize), default size 128M.HDFS file system
Can provide a unified abstract directory tree, client can access file by specified path, such as: hdfs: //
namenode:port/dir-a/dir-b/dir-c/file.data.Bibliographic structure and file block location information (metadata)
Management undertaken by name-node node, name-node is HDFS cluster host node, is responsible for the entire hdfs file system of maintenance
Directory tree and each path (file) corresponding to data block information (block identification and the data-node at place service
Device).The storage management of each block of file is undertaken by data-node node, and data-node is HDFS cluster from node, often
One block can store multiple copies on multiple data-node, and (copy amount can also pass through parameter setting
3) dfs.replication, default are.Data-node can periodically report the file block itself saved to Name-node
Information, and name-node can then be responsible for keeping the copy amount of file, it can be by name-node when requesting access to HDFS
Apply to carry out.
In one embodiment, the backup module 101 can be in such a way that snapshot migrates by the specified log number
It stores according to from backup in ES cluster to the HDFS, specifically: a repository-hdfs plug-in unit is added in ES cluster simultaneously
Modify the configuration file of each node；A warehouse is established, for creating and storing snapshot, wherein the ES cluster can create
Multiple warehouses can choose a specified warehouse creation snapshot, may include multiple indexes inside each snapshot, default is standby
The index of the entire ES cluster of part, also can specify the index for only backing up specified daily record data, then recovery is fast in the HDFS
According to can be realized and store specified daily record data to the HDFS from ES cluster backup.
In one embodiment, the backup module 101 can also be realized by establishing an ES-Hadoop frame by
Specified daily record data in ELK log system is stored from ES cluster backup to the HDFS, and specifically: creation is literary in ES cluster
Shelves (data are stored by basic unit of document in ES cluster), the document includes the information such as index, type, id,
Index is used to indicate which index document belongs to, and type is used to indicate which type document belongs to, and id is the mark of document
Symbol, then creates the Data Migration mapping table from ES cluster to Hadoop, the value of the key of the mapping table parameter is in ES cluster
The id value of document, input parameter are the contents of document in ES cluster；Create the Job of the Data Migration from ES cluster to Hadoop
Class, the Job class can will read data from ES cluster, and be converted into the input parameter of the mapping table, finally start
The realization of MapReduce task stores specified daily record data to HDFS from ES cluster backup.
First creation module 102 is for creating a Kubernetes cluster, wherein the Kubernetes cluster is only
Stand on the ES cluster and the HDFS of the ELK log system.
In one embodiment, the Kubernetes cluster can be created in ELK log system.Specifically, described
First creation module 102 creation Kubernetes cluster can be accomplished by the following way: setting creates the Kubernetes
Cloud virtual machine quantity and specification used in cluster, and complete the creation of required cloud virtual machine；Obtain the IP information of cloud virtual machine
With ssh (Secure Shell, safety shell protocol) information；It is copied required for deployment Kubernetes cluster using ssh tool
Binary file to the cloud virtual machine created on, and Kubernetes collection swarm parameter is set；Finally use kubectl
The all components of tool deployment Kubernetes cluster.
In one embodiment, the ES cluster in the Kubernetes cluster and the HDFS, the ELK log system
Mutually decoupling, i.e., the described Kubernetes cluster, the ES cluster, the HDFS are mutually indepedent.
Second creation module 103 is asked for receiving to be directed to store to the inquiry of the specified daily record data of the HDFS
It asks, and the Kubernetes cluster is called to create an interim ES cluster.
It in one embodiment, can be in ELK days aspirations when user, which wants inquiry, has backed up to the daily record data of HDFS
The essential information of Backup Data is filled on the page of system and issues inquiry request, and the ELK log system can call
Kubernetes cluster is that user creates an interim ES cluster, and the interim ES cluster is independently of the ES collection of ELK log system
Group.
In one embodiment, second creation module 103 is during carrying out the creation of interim ES cluster, Ke Yixian
Acquisition is stored to the data volume size of the specified daily record data of the HDFS, and big according to the data volume of the specified daily record data
Small creation includes the interim ES cluster of corresponding node quantity.In other words, if the data volume of specified daily record data is bigger, need for
The interim ES cluster creates greater number of ES-Client node, ES-Data node and ES-Master node, if specified
The data volume of daily record data is smaller, can create small number of ES-Client node, ES-Data for the interim ES cluster
Node and ES-Master node.It is understood that the interim ES cluster includes at least an ES-Client node, an ES-
Data node and an ES-Master node.
In one embodiment, ELK log system has the function of authority managing and controlling, the ELK log power that each user can check
Limit is different, and the data volume that user can back up to the daily record data of HDFS is also different.The ELK log system is calling
The interface of Kubernetes cluster is that can previously back up to HDFS's according to user during user creates interim ES cluster
Daily record data amount size creates the ES-Client node, ES-Data node and ES-Master node of respective numbers.Difference is used
Log amount size in family corresponds to the ES-Client node, ES-Data node and ES-Master node of different number.
The ES-Data node is mainly used for storing index data, can change to document progress additions and deletions and look into operation, converging operation
Deng.The ES-Master node such as creates or deletes index for carrying out the relevant content of cluster operation, which node tracked
It is a part of cluster, and determines which fragment distributes to relevant node.The ES-Client node tunable ES-
Master node and ES-Data node, the ES-Client node are added the state of available cluster after cluster and can bases
The state direct routing of cluster is requested.When ES-Master node and the configuration of ES-Data node are both configured to false,
ES-Client node can handle route requests, processing search, the operation such as distribution index.
The specified daily record data that the execution module 104 is used to store the distributed file system imports described interim
ES cluster, to execute the inquiry request by the interim ES cluster.
In one embodiment, the execution module 104 again may be by establishing an ES-Hadoop frame, and utilize
The specified daily record data that the ES-Hadoop frame stores the distributed file system imports the interim ES cluster.When
After completing the specified daily record data for the storing distributed file system importing interim ES cluster, the interim ES is returned
The address of service of cluster, and then user can link to the interim ES cluster according to the address of service, to be faced by described
When ES cluster execute the inquiry request.
In one embodiment, the execution module 104, which can be accomplished by the following way, establishes an ES-Hadoop frame
Frame, and it is described interim using the specified daily record data importing that the ES-Hadoop frame stores the distributed file system
ES cluster: from Hadoop to the mapping table of the Data Migration of the interim ES cluster, the input of the mapping table is for creation first
One HDFS file (the HDFS file includes daily record data to be imported), the output of the mapping table the result is that a json
The Text of format；Then the Job class from Hadoop to the Data Migration of the interim ES cluster is created, the Job class can incite somebody to action
The output result (Text of a json format) of MapReduce is converted to the id of the interim ES cluster and the content of document；
Finally data are imported the interim ES cluster from HDFS by starting MapReduce task.
In one embodiment, the execution module 104 can also be accomplished by the following way the distributed text
The specified daily record data of part system storage imports the interim ES cluster: being first loaded into specified daily record data from HDFS
It is stored in Spark SQL and in the form of RDD (set of distributed Java object)；Then addition preset data structural information with
The RDD is updated, and DataFrame (set of distributed Row object) is created according to updated RDD, to be connected to
State interim ES cluster；It is finally created and is indexed by DataFrame, and the interim ES cluster is written.
The monitoring modular 105 is for monitoring whether the operation that the interim ES cluster executes the inquiry request terminates.
In one embodiment, the log searching system 10 can execute the inquiry by monitoring the interim ES cluster
Whether the operation of request terminates, to determine whether needing to discharge the interim ES cluster.
The release module 106 is used to terminate in the operation of the inquiry request, discharges the interim ES cluster.
In one embodiment, when ELK log system monitors that user completes daily record data on the interim ES cluster
Inquiry operation after, the release module 106 discharges the interim ES cluster, and then can be created that the interim ES collection
Group is only effective to active user, will not occupying system resources, ensure that the uniqueness of user's operation, entire ELK log system money
Source has good allocability, does not influence the operating experience of other users.When receive again for HDFS storage log
When the inquiry request of data, need to re-create interim ES cluster to realize data query operation.
Fig. 4 is the schematic diagram of computer installation preferred embodiment of the present invention.
The computer installation 1 includes memory 20, processor 30 and is stored in the memory 20 and can be in institute
State the computer program 40 run on processor 30, such as log searching program.The processor 30 executes the computer journey
Realize the step in above-mentioned log searching embodiment of the method when sequence 40, such as step S11~S14 shown in FIG. 1, shown in Fig. 2
Step S11~S16.Alternatively, the processor 30 realizes that above-mentioned log searching system is implemented when executing the computer program 40
The function of each module in example, such as the module 101~106 in Fig. 3.
Illustratively, the computer program 40 can be divided into one or more module/units, it is one or
Multiple module/units are stored in the memory 20, and are executed by the processor 30, to complete the present invention.Described one
A or multiple module/units can be the series of computation machine program instruction section that can complete specific function, and described instruction section is used
In implementation procedure of the description computer program 40 in the computer installation 1.For example, the computer program 40 can be with
Backup module 101, the first creation module 102, the second creation module 103, execution module 104, the monitoring mould being divided into Fig. 3
Block 105, release module 106.Each module concrete function is referring to embodiment two.
The computer installation 1 can be the calculating such as desktop PC, notebook, palm PC and cloud server and set
It is standby.It will be understood by those skilled in the art that the schematic diagram is only the example of computer installation 1, do not constitute to computer
The restriction of device 1 may include perhaps combining certain components or different components, example than illustrating more or fewer components
Such as described computer installation 1 can also include input-output equipment, network access equipment, bus.
Alleged processor 30 can be central processing unit (Central Processing Unit, CPU), can also be
Other general processors, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor 30 is also possible to any conventional processing
Device etc., the processor 30 are the control centres of the computer installation 1, utilize various interfaces and the entire computer of connection
The various pieces of device 1.
The memory 20 can be used for storing the computer program 40 and/or module/unit, and the processor 30 passes through
Operation executes the computer program and/or module/unit being stored in the memory 20, and calls and be stored in memory
Data in 20 realize the various functions of the computer installation 1.The memory 20 can mainly include storing program area and deposit
Store up data field, wherein storing program area can application program needed for storage program area, at least one function (for example sound is broadcast
Playing function, image player function etc.) etc.；Storage data area, which can be stored, uses created data (ratio according to computer installation 1
Such as audio data, phone directory) etc..In addition, memory 20 may include high-speed random access memory, it can also include non-easy
The property lost memory, such as hard disk, memory, plug-in type hard disk, intelligent memory card (Smart Media Card, SMC), secure digital
(Secure Digital, SD) card, flash card (Flash Card), at least one disk memory, flush memory device or other
Volatile solid-state part.
If the integrated module/unit of the computer installation 1 is realized in the form of SFU software functional unit and as independence
Product when selling or using, can store in a computer readable storage medium.Based on this understanding, of the invention
It realizes all or part of the process in above-described embodiment method, can also instruct relevant hardware come complete by computer program
At the computer program can be stored in a computer readable storage medium, and the computer program is held by processor
When row, it can be achieved that the step of above-mentioned each embodiment of the method.Wherein, the computer program includes computer program code, institute
Stating computer program code can be source code form, object identification code form, executable file or certain intermediate forms etc..It is described
Computer-readable medium may include: any entity or device, recording medium, U that can carry the computer program code
Disk, mobile hard disk, magnetic disk, CD, computer storage, read-only memory (ROM, Read-Only Memory), arbitrary access
Memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium etc..It needs
It is bright, the content that the computer-readable medium includes can according in jurisdiction make laws and patent practice requirement into
Row increase and decrease appropriate, such as do not include electric load according to legislation and patent practice, computer-readable medium in certain jurisdictions
Wave signal and telecommunication signal.
In several embodiments provided by the present invention, it should be understood that disclosed computer installation and method, it can be with
It realizes by another way.For example, computer installation embodiment described above is only schematical, for example, described
The division of unit, only a kind of logical function partition, there may be another division manner in actual implementation.
It, can also be in addition, each functional unit in each embodiment of the present invention can integrate in same treatment unit
It is that each unit physically exists alone, can also be integrated in same unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of hardware adds software function module.
It is obvious to a person skilled in the art that invention is not limited to the details of the above exemplary embodiments, Er Qie
In the case where without departing substantially from spirit or essential attributes of the invention, the present invention can be realized in other specific forms.Therefore, no matter
From the point of view of which point, the present embodiments are to be considered as illustrative and not restrictive, and the scope of the present invention is by appended power
Benefit requires rather than above description limits, it is intended that all by what is fallen within the meaning and scope of the equivalent elements of the claims
Variation is included in the present invention.Any reference signs in the claims should not be construed as limiting the involved claims.This
Outside, it is clear that one word of " comprising " does not exclude other units or steps, and odd number is not excluded for plural number.It is stated in computer installation claim
Multiple units or computer installation can also be implemented through software or hardware by the same unit or computer installation.The
One, the second equal words are used to indicate names, and are not indicated any particular order.
Finally it should be noted that the above examples are only used to illustrate the technical scheme of the present invention and are not limiting, although reference
Preferred embodiment describes the invention in detail, those skilled in the art should understand that, it can be to of the invention
Technical solution is modified or equivalent replacement, without departing from the spirit and scope of the technical solution of the present invention.
Claims (10)
1. a kind of log searching method, which is characterized in that the described method includes:
Specified daily record data in ELK log system is stored from ES cluster backup to a distributed file system；
A Kubernetes cluster is created, wherein ES cluster of the Kubernetes cluster independently of the ELK log system
And the distributed file system；
It receives to be directed to and store to the inquiry request of the specified daily record data of the distributed file system, and described in calling
Kubernetes cluster creates an interim ES cluster；And
The specified daily record data that the distributed file system is stored imports the interim ES cluster, to pass through the interim ES
Cluster executes the inquiry request.
2. log searching method as described in claim 1, which is characterized in that the specified log by ELK log system
Data are stored from ES cluster backup to the step of distributed file system
The specified daily record data in the ELK log system is stored from the ES cluster backup to institute using snapshot migration algorithm
State distributed file system；Or
An ES-Hadoop frame is established, and utilizes the ES-Hadoop frame by the specified log in the ELK log system
Data are stored from the ES cluster backup to the distributed file system.
3. log searching method as described in claim 1, which is characterized in that the interim ES cluster is independently of ELK days described
ES cluster in aspiration system.
4. log searching method as claimed in any one of claims 1-3, which is characterized in that described in the calling
Kubernetes cluster create an interim ES cluster the step of include:
Acquisition is stored to the data volume size of the specified daily record data of the distributed file system；And
It include the interim ES cluster of corresponding node quantity according to the creation of the data volume size of the specified daily record data；
Wherein, the interim ES cluster includes an at least ES-Client node, at least an ES-Data node and an at least ES-
Master node.
5. log searching method as described in claim 1, which is characterized in that described to store the distributed file system
Specified daily record data imports the step of interim ES cluster and includes:
An ES-Hadoop frame is established, and is specified using the ES-Hadoop frame by what the distributed file system stored
Daily record data imports the interim ES cluster.
6. log searching method as claimed in any one of claims 1-3, which is characterized in that described by the distributed document
The specified daily record data of system storage imports the interim ES cluster, to execute the inquiry request by the interim ES cluster
The step of include；
The specified daily record data that the distributed file system is stored imports the interim ES cluster, and in data import operation
The address of service of the interim ES cluster is returned to after the completion；And
The interim ES cluster is linked to according to the address of service, is asked with executing the inquiry by the interim ES cluster
It asks.
7. log searching method as claimed in any one of claims 1-3, which is characterized in that described by the distributed document
The specified daily record data of system storage imports the interim ES cluster, to execute the inquiry request by the interim ES cluster
The step of after further include:
Monitor whether the operation that the interim ES cluster executes the inquiry request terminates；And
When the operation of the inquiry request has terminated, the interim ES cluster is discharged.
8. a kind of log searching system, which is characterized in that the system comprises:
Backup module, for storing the specified daily record data in ELK log system from ES cluster backup to a distributed document
System；
First creation module, for creating a Kubernetes cluster, wherein the Kubernetes cluster is independently of the ELK
The ES cluster of log system and the distributed file system；
Second creation module is asked for receiving to be directed to store to the inquiry of the specified daily record data of the distributed file system
It asks, and the Kubernetes cluster is called to create an interim ES cluster；And
Execution module, the specified daily record data for storing the distributed file system import the interim ES cluster, with
The inquiry request is executed by the interim ES cluster.
9. a kind of terminal, the terminal includes processor and memory, and several computer programs are stored on the memory,
It is characterized in that, is realized when the processor is for executing the computer program stored in memory as any in claim 1-7
The step of log searching method described in one.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program
The step of log searching method as described in any one of claim 1-7 is realized when being executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811246089.4A CN109299225B (en) | 2018-10-24 | 2018-10-24 | Log retrieval method, system, terminal and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811246089.4A CN109299225B (en) | 2018-10-24 | 2018-10-24 | Log retrieval method, system, terminal and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109299225A true CN109299225A (en) | 2019-02-01 |
| CN109299225B CN109299225B (en) | 2024-05-28 |
Family
ID=65158639
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811246089.4A Active CN109299225B (en) | 2018-10-24 | 2018-10-24 | Log retrieval method, system, terminal and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109299225B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110633265A (en) * | 2019-08-22 | 2019-12-31 | 达疆网络科技(上海)有限公司 | Method for realizing ES (ES) cross-multi-version data migration |
| CN112948188A (en) * | 2021-01-28 | 2021-06-11 | 苏州浪潮智能科技有限公司 | Log file screening method, system and medium |
| CN113515409A (en) * | 2021-03-04 | 2021-10-19 | 浪潮云信息技术股份公司 | Log timing backup method and system based on ELK |
| CN113761043A (en) * | 2021-08-17 | 2021-12-07 | 紫金诚征信有限公司 | Data extraction method and device, computer equipment and storage medium |
| CN114647559A (en) * | 2022-03-21 | 2022-06-21 | 北京百度网讯科技有限公司 | Storage usage amount statistical method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080086480A1 (en) * | 2006-10-06 | 2008-04-10 | Sybase, Inc. | System and Methods For Temporary Data Management in Shared Disk Cluster |
| CN102937980A (en) * | 2012-10-18 | 2013-02-20 | 亿赞普(北京)科技有限公司 | Method for inquiring data of cluster database |
| US8719415B1 (en) * | 2010-06-28 | 2014-05-06 | Amazon Technologies, Inc. | Use of temporarily available computing nodes for dynamic scaling of a cluster |
| CN104239532A (en) * | 2014-09-19 | 2014-12-24 | 浪潮(北京)电子信息产业有限公司 | Method and device for self-making user extraction information tool in Hive |
| CN107547653A (en) * | 2017-09-11 | 2018-01-05 | 华北水利水电大学 | A kind of distributed file storage system |
| CN107766386A (en) * | 2016-08-22 | 2018-03-06 | 中兴通讯股份有限公司 | A kind of solr data migration methods and device |
-
2018
- 2018-10-24 CN CN201811246089.4A patent/CN109299225B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080086480A1 (en) * | 2006-10-06 | 2008-04-10 | Sybase, Inc. | System and Methods For Temporary Data Management in Shared Disk Cluster |
| US8719415B1 (en) * | 2010-06-28 | 2014-05-06 | Amazon Technologies, Inc. | Use of temporarily available computing nodes for dynamic scaling of a cluster |
| CN102937980A (en) * | 2012-10-18 | 2013-02-20 | 亿赞普(北京)科技有限公司 | Method for inquiring data of cluster database |
| CN104239532A (en) * | 2014-09-19 | 2014-12-24 | 浪潮(北京)电子信息产业有限公司 | Method and device for self-making user extraction information tool in Hive |
| CN107766386A (en) * | 2016-08-22 | 2018-03-06 | 中兴通讯股份有限公司 | A kind of solr data migration methods and device |
| CN107547653A (en) * | 2017-09-11 | 2018-01-05 | 华北水利水电大学 | A kind of distributed file storage system |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110633265A (en) * | 2019-08-22 | 2019-12-31 | 达疆网络科技(上海)有限公司 | Method for realizing ES (ES) cross-multi-version data migration |
| CN112948188A (en) * | 2021-01-28 | 2021-06-11 | 苏州浪潮智能科技有限公司 | Log file screening method, system and medium |
| CN113515409A (en) * | 2021-03-04 | 2021-10-19 | 浪潮云信息技术股份公司 | Log timing backup method and system based on ELK |
| CN113761043A (en) * | 2021-08-17 | 2021-12-07 | 紫金诚征信有限公司 | Data extraction method and device, computer equipment and storage medium |
| CN113761043B (en) * | 2021-08-17 | 2024-08-16 | 紫金诚征信有限公司 | Data extraction method, device, computer equipment and storage medium |
| CN114647559A (en) * | 2022-03-21 | 2022-06-21 | 北京百度网讯科技有限公司 | Storage usage amount statistical method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109299225B (en) | 2024-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11422853B2 (en) | Dynamic tree determination for data processing | |
| CN109299225A (en) | Log searching method, system, terminal and computer readable storage medium | |
| US9244951B2 (en) | Managing tenant-specific data sets in a multi-tenant environment | |
| US8832130B2 (en) | System and method for implementing on demand cloud database | |
| US9996593B1 (en) | Parallel processing framework | |
| US8990243B2 (en) | Determining data location in a distributed data store | |
| US9684702B2 (en) | Database redistribution utilizing virtual partitions | |
| US10257109B2 (en) | Cloud-based content management system | |
| CN110019251A (en) | A kind of data processing system, method and apparatus | |
| CN105468720A (en) | Method for integrating distributed data processing systems, corresponding systems and data processing method | |
| CN109902114A (en) | ES company-data multiplexing method, system, computer installation and storage medium | |
| US11151081B1 (en) | Data tiering service with cold tier indexing | |
| WO2016169237A1 (en) | Data processing method and device | |
| US11232000B1 (en) | Moving database partitions from replica nodes | |
| CN112685499A (en) | Method, device and equipment for synchronizing process data of work service flow | |
| US11100129B1 (en) | Providing a consistent view of associations between independently replicated data objects | |
| JP2023534656A (en) | Archiving of accelerator-only database tables | |
| US11500874B2 (en) | Systems and methods for linking metric data to resources | |
| CN105550342B (en) | A kind of data processing method of the distributed data base of all-transparent | |
| US20230099501A1 (en) | Masking shard operations in distributed database systems | |
| US8862544B2 (en) | Grid based replication | |
| CN113448939A (en) | Data archiving and storing method and device and storage medium | |
| US11757703B1 (en) | Access requests processing and failover handling across multiple fault tolerance zones | |
| US11727022B2 (en) | Generating a global delta in distributed databases | |
| CN113448775B (en) | Multi-source heterogeneous data backup method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |