CN102937984B - A kind of collect the system of data, client and method - Google Patents
A kind of collect the system of data, client and method Download PDFInfo
- Publication number
- CN102937984B CN102937984B CN201210404918.3A CN201210404918A CN102937984B CN 102937984 B CN102937984 B CN 102937984B CN 201210404918 A CN201210404918 A CN 201210404918A CN 102937984 B CN102937984 B CN 102937984B
- Authority
- CN
- China
- Prior art keywords
- field
- data
- value
- key
- merger
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000008569 process Effects 0.000 claims abstract description 29
- 238000004220 aggregation Methods 0.000 claims abstract description 24
- 230000002776 aggregation Effects 0.000 claims abstract description 24
- 238000004519 manufacturing process Methods 0.000 claims abstract description 15
- 238000012935 Averaging Methods 0.000 claims description 22
- 238000012545 processing Methods 0.000 claims description 14
- 230000001186 cumulative effect Effects 0.000 claims description 12
- 238000013481 data capture Methods 0.000 claims description 9
- 238000009826 distribution Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 abstract description 9
- 238000003860 storage Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 241000894007 species Species 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000007667 floating Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 101100297538 Caenorhabditis elegans php-3 gene Proteins 0.000 description 1
- 241001269238 Data Species 0.000 description 1
- 238000005267 amalgamation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000013479 data entry Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000007257 malfunction Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses and a kind of collect the system of data, client and method, belong to Internet technical field. The solution of the present invention includes: difference deploying client on different production servers, acquired data are stored classifiedly by client according to corresponding different classes of business, when the timing cycle that every kind business is corresponding terminates, client is by the data of the category business preserved, and the aggregation of data being identified with the value of the field of key identical is be sent to server end after a data. Technical scheme, the data comprising any number field can be transmitted so that the transmission of data no longer has many restrictions, and in client, data has been carried out merger process, therefore avoid because transmitting substantial amounts of same or similar data, the problem causing network blockage and delay.
Description
Technical field
The present invention relates to Internet technical field, be specifically related to a kind of collect the system of data, client and method.
Background technology
Internet era back-end data extremely important and huge, such as daily record data and statistical data. These back-end datas are probably backstage engineer and analyze the firsthand information of program operation conditions, it is also possible to the first reference that service operation decision-making relies on. But the website of flow generally has up to ten million to produce servers greatly, and it is distributed in each different machine room. Journal file or statistical data will leave these network isomeries in and be distributed on scattered production server, to the collection of daily record, transmits, collects and analyzes and bring very big difficulty. There are some softwares increased income to can be used to collect these daily records at present, but there is also a lot of indeterminable situation.
Some open source softwares such as Scribe comparatively conventional at present, it is possible to reach the purpose of simple collection daily record data.
Scribe is the result collection system increased income of a current large-scale social, gets a lot of applications inside the social that this is large-scale. It from collector journal various Log Sources, can store to a central storage system (can be NFS, distributed file system HDFS etc.), in order to carries out concentrating statistical analysis to process. It for " distributed collection is uniformly processed " of daily record provide one extendible, high fault-tolerant scheme. When the network of central storage system or machine break down, daily record can be dumped to this locality or another position by scribe, when, after centralized storage system recovery, the daily record of unloading can be retransmitted to central storage system by scribe.It is generally combined use with Hadoop, and scribe is used for push daily record in HDFS, and Hadoop is regularly processed by MapReduce operation.
Fig. 1 is the schematic diagram of existing Scribe collector journal. As it is shown in figure 1, Scribe collects data from as each application of kind of data source, being put in a shared queue, then push is to the central storage system of rear end. When centralized storage system malfunctions, scribe can temporarily write daily record in local file, and after central storage system restorability, scribe resumes local daily record on central storage system.
Each data source must by THRIFT(owing to have employed THRIFT, and client can adopt various language to write and transmit data to scribe, and every data record comprises an a category and message). Can at the scribe configuration THRIFT Thread Count (being defaulted as 3) for listening port. In rear end, the data of different category can be stored in different directories by scribe, in order to processes respectively. The daily record storage mode of rear end can be various store, including: file(file), buffer(bilayer stores, a main storage, a secondary storage), another scribe server of network() etc.
But scribe has a disadvantage in that
(1) shortcoming of scribe is that front group organization data is dumb, two fields can only be used, i.e. catagory and message, in the application program producing server, if it is desired to send data with scribe, then every data can only have two fields of catagory and message, if it is desired to transmit multiple field, then oneself organising data necessary, the multiple data that will transmit are merged into inside message. Also want when post analysis data oneself to resolve message, obtain original multiple fields. This causes many restrictions and inconvenience to the transmission of data.
(2) another disadvantage is that, scribe can receive each data, and they are verily recorded at local cache, with certain frequency Batch sending data, even if category and the message of two data is living. When this is very big in volume of transmitted data, transmission frequency is significantly high, it is easy to cause serious network blockage and delay.
Summary of the invention
In view of the above problems, it is proposed that the present invention is to provide a kind of and overcome the problems referred to above or solve the system of collection data of the problems referred to above, client and the corresponding method collecting data at least in part.
According to one aspect of the present invention, it is provided that a kind of system collecting data, wherein, this system includes: server end and the multiple clients being deployed on different production server,
Described client, is suitable to obtain the data of corresponding different classes of business produced by production server, acquired data is stored classifiedly according to corresponding different classes of business;
Wherein, every data comprises more than one field, and different fields has different types, and at least one field identification of every data has key; Every kind business has a corresponding timing cycle;
Described client, is further adapted for when the timing cycle that every kind business is corresponding terminates, and by the data of the category business preserved, the aggregation of data being identified with the value of the field of key identical is be sent to described server end after a data;
Described server end, is suitable to receive data from each client, and carries out storing or forwarding.
Alternatively, described client, it is further adapted for when the timing cycle that every kind business is corresponding terminates, by in the data of the category business preserved, it is identified with the value of the field of key identical aggregation of data when being a data, to the field not identifying key, carry out different merger according to different types and process.
Alternatively, described client, it is further adapted for when the field not identifying key being carried out different merger process according to different types, adopts the combination of one or more of:
For the field of sum-type, the numerical value in this field of each data identical for the value being identified with the field of key is added, itself and as the value of this field after merger;
For being averaging the field of type, being averaging by the numerical value in this field of each data identical for the value being identified with the field of key, its average is as the value of this field after merger;
For the field of maximum type, from the value this field of the identical each data of the value of the field being identified with key, find out maximum, as the value of this field after merger;
For the field of normal character types, from each data that the value of the field being identified with key is identical, take the value in this field of Article 1 data, as the value of this field after merger;
For the field of cumulative character types, after the character in this field of each data identical for the value being identified with the field of key is concatenated in a designated order, as the value of this field after merger.
Alternatively, described server end, be suitable to be transmitted to received data other server, or be forwarded to database facility, or preserve into local file.
According to a further aspect in the invention, provide a kind of client collecting data, wherein, this client includes: data capture unit, merger processing unit and multiple memory element, the plurality of memory element distinguishes corresponding different classes of business, and each memory element has a corresponding timing cycle;
Described data capture unit, is suitable to, from the data producing the server corresponding different classes of business of acquisition, acquired data be preserved to corresponding memory element according to corresponding different classes of distribution of services; Wherein, every data comprises more than one field, and different fields has different types, and at least one field identification of every data has key;
Each memory element, is suitable to preserve the data from data capture unit;
Described merger processing unit, is suitable to when the timing cycle that each memory element is corresponding terminates, and the aggregation of data that the value of the field being identified with key in the data preserve this memory element is identical is be sent to server end after a data.
Alternatively, described merger processing unit, it is further adapted for when the timing cycle that every kind business is corresponding terminates, when being a data by aggregation of data identical for the value of the field being identified with key in the data of the category business preserved, to the field not identifying key, carry out different merger according to different types and process.
Alternatively, merger processing unit, it is further adapted for when the field not identifying key being carried out different merger process according to different types, adopts the combination of one or more of:
For the field of sum-type, the numerical value in this field of each data identical for the value being identified with the field of key is added, itself and as the value of this field after merger;
For being averaging the field of type, being averaging by the numerical value in this field of each data identical for the value being identified with the field of key, its average is as the value of this field after merger;
For the field of maximum type, from the value this field of the identical each data of the value of the field being identified with key, find out maximum, as the value of this field after merger;
For the field of normal character types, from each data that the value of the field being identified with key is identical, take the value in this field of Article 1 data, as the value of this field after merger;
For the field of cumulative character types, after the character in this field of each data identical for the value being identified with the field of key is concatenated in a designated order, as the value of this field after merger.
According to another aspect of the invention, it is provided that a kind of method collecting data, wherein, the method includes:
It is deployed in the client produced on server and obtains the data of corresponding different classes of business produced by this production server; Wherein, every data comprises more than one field, and different fields has different types, and at least one field identification of every data has key;
Acquired data are stored classifiedly by described client according to corresponding different classes of business; Wherein, every kind business has a corresponding timing cycle;
For every kind business, aggregation of data identical for the value of the field being identified with key in the data of the category business preserved, when corresponding timing cycle terminates, is be sent to server end after a data by described client.
Alternatively, described is that a data includes by aggregation of data identical for the value being identified with the field of key:
For not identifying the field of key, carry out different merger according to different types and process.
Alternatively, the described field for not identifying key, carry out different merger according to different field types and process the combination including one or more of:
For the field of sum-type, the numerical value in this field of each data identical for the value being identified with the field of key is added, itself and as the value of this field after merger;
For being averaging the field of type, being averaging by the numerical value in this field of each data identical for the value being identified with the field of key, its average is as the value of this field after merger;
For the field of maximum type, from the value this field of the identical each data of the value of the field being identified with key, find out maximum, as the value of this field after merger;
For the field of normal character types, from each data that the value of the field being identified with key is identical, take the value in this field of Article 1 data, as the value of this field after merger;
For the field of cumulative character types, after the character in this field of each data identical for the value being identified with the field of key is concatenated in a designated order, as the value of this field after merger.
This difference deploying client on different production servers according to the present invention, the data of collection are issued server end by each client, wherein acquired data are stored classifiedly by client according to corresponding different classes of business, every data comprises more than one different types of field, when the timing cycle that every kind business is corresponding terminates, by in the data of the category business preserved, the aggregation of data being identified with the value of the field of key identical is the technical scheme being sent to server end after a data, the data of an arbitrarily field can be transmitted, and just carried out aggregation of data process in client, thus solving existing scribe only allows every data can only have two fields of catagory and message, so that the problem that the transmission of data has many restrictions, and solve existing scribe and simply verily record data in front end, do not carry out merger process, cause that volume of transmitted data is big, transmission frequency is high, the problem easily causing network blockage and delay.
Described above is only the general introduction of technical solution of the present invention, in order to better understand the technological means of the present invention, and can be practiced according to the content of description, and in order to above and other objects of the present invention, feature and advantage can be become apparent, below especially exemplified by the specific embodiment of the present invention.
Accompanying drawing explanation
By reading hereafter detailed description of the preferred embodiment, various other advantage and benefit those of ordinary skill in the art be will be clear from understanding. Accompanying drawing is only for illustrating the purpose of preferred implementation, and is not considered as limitation of the present invention. And in whole accompanying drawing, it is denoted by the same reference numerals identical parts. In the accompanying drawings:
Fig. 1 is the schematic diagram of existing Scribe collector journal;
Fig. 2 illustrates the block diagram of a kind of according to an embodiment of the invention system collecting data;
Fig. 3 illustrates the structure chart of a kind of according to an embodiment of the invention client collecting data;
Fig. 4 illustrates the flow chart of a kind of according to an embodiment of the invention method collecting data.
Detailed description of the invention
It is more fully described the exemplary embodiment of the disclosure below with reference to accompanying drawings. Although accompanying drawing showing the exemplary embodiment of the disclosure, it being understood, however, that may be realized in various forms the disclosure and should do not limited by embodiments set forth here. On the contrary, it is provided that these embodiments are able to be best understood from the disclosure, and complete for the scope of the present disclosure can be conveyed to those skilled in the art.
Fig. 2 illustrates the block diagram of a kind of according to an embodiment of the invention system collecting data. As in figure 2 it is shown, this system includes: server end 202 and multiple client 201. Multiple clients 201 are respectively deployed in different needs and collect on the production server of various data. Each client 201 is collected Data Concurrent produced by self place production server and is given server end 202, and server end 202 receives the data that each client 201 is beamed back, and carries out server that is locally stored or that be transmitted to other. Specifically:
Each client 201, is suitable to obtain the data of corresponding different classes of business produced by production server, acquired data is stored classifiedly according to corresponding different classes of business. Wherein, every data comprises more than one field, and different fields has different types, and at least one field identification of every data has key; Every kind business has a corresponding timing cycle;
Each client 201, when the timing cycle that every kind business is corresponding terminates, by the data of the category business preserved, the aggregation of data being identified with the value of the field of key identical is be sent to described server end 202 after a data;
Server end 202, is suitable to receive data from each client 201, and carries out storing or forwarding.
Here, the data form of the data of one species various-service is identical, and the field number that namely data comprise is all identical with the type of each field. The form that can define data according to the actual requirements of business of all categories, the field number that namely can comprise according to practical business requirement definition one data and the type of each field. For example, it is possible to define following types of field: sum-type (SUM_INT), be averaging type (AVG_INT), maximum type (MAX_INT), normal character types (CONST_STRING) and cumulative character types (CONST_STRING) etc.
Arrange these fields in order that the data for various different kinds of business do various optimization process, making small data quantity, in analyzing and processing data hourly velocity faster, committed memory is less, it is easier to identify the implication of each field, and is that merger below is ready.
Each client 201 is according to configuring maintenance multiple " box ", and the data of same format deposited by each box. That is corresponding one " box " of a kind of category traffic, the data of category business are stored in the box of this correspondence. The data that client 201 will be collected, according to which box is its data format determination leave in. When an end cycle of certain box, the data in this box are done a merger and are processed by client 201, are then sent to server end 202.
In the present invention, as the foundation of merger, some field of data identifies key (" Key " attribute), when doing aggregation of data, comparing the field being identified with key in data, the data being only identified with the value of the field of key identical can merger be just a data.
Client 201 is when the timing cycle that every kind business is corresponding terminates, by in the data of the category business preserved, it is identified with the value of the field of key identical aggregation of data when being a data, to the field not identifying key, carry out different merger according to different types and process. Namely field type is different, and its merger mode is also different.
Client 201, when the field not identifying key being carried out different merger process according to different types, it is possible to adopt the combination of one or more of:
(1) for the field of sum-type: when merger, the numerical value in the sum-type field of each data identical for the value that is identified with the field of key is added, itself and as the value of the sum-type field of data after merger;
(2) for being averaging the field of type: when merger, being averaging by the numerical value being averaging on type field of each data identical for the value being identified with the field of key, its average is as the value being averaging type field of data after merger;
(3) for the field of maximum type: when merger, from the value the maximum type field of the identical each data of the value of the field being identified with key, maximum is found out, as the value of the maximum type field of data after merger;
(4) for the field of normal character types: when merger, from each data that the value of the field being identified with key is identical, the value in the normal character types field of Article 1 data is taken, as the value of the normal character types field of data after merger;
(5) for the field of cumulative character types: when merger, after the character in the cumulative character types field of each data identical for the value being identified with the field of key is concatenated in a designated order, as the value of the cumulative character types field of data after merger.
More than schematically illustrate 5 kinds of field types and its each corresponding merger mode. But the field type in the present invention is not limited to above 5 kind, it is possible to according to the further types of field of practical business requirement definition and its merger mode. Such as can also define floating number and be averaging type (AVG_FLOAT), minima type (MIN_INT) and floating number sum-type (SUM_FLOAT) etc., no longer describe in detail one by one here.
Data are carried out merger process a object lesson is given below.
Define a data form logging in business, this data form, for the data logging in the business number of times of a page " user access " are recorded, are safeguarded one " box " accordingly on the client, is " Login ", cycle is 300 seconds, then data form particularly as follows:
Login (300): user_idKEY_STR, scriptKEY_STR, numberSUM_INT, datetimeTIME_FLOOR;
This data form includes 4 fields, and the first two field user_id and script above identifies type respectively sum-type (SUM_INT) and floor time type (TIME_FLOOR) of relevant key KEY_STR, latter two field number and datetime.
After having defined, it is possible to send, producing, the data meeting each field type on server, the data sent are collected by the client being deployed on this production server. The data that such as client is collected between 2012-09-2100:00:00 to 2012-09-2100:04:59 are as shown in table 1:
| ZK | Index.php | 1 | 2012-09-2100:00:00 |
| ZK | Index.php | 1 | 2012-09-2100:01:03 |
| ZK | Index.php | 5 | 2012-09-2100:01:23 |
| ZK | Login.php | 2 | 2012-09-2100:02:14 |
| HGL | Login.php | 2 | 2012-09-2100:02:14 |
| ZK | Index.php | 3 | 2012-09-2100:03:19 |
| HGL | Index.php | 7 | 2012-09-2100:04:10 |
| HGL | Index.php | 10 | 2012-09-2100:04:34 |
Table 1
Data shown in table 1 are the data belonging to the business of logging in, and its form is identical, are therefore put in " Login " box by client. After the cycle of 300 seconds has arrived, the data in " Login " this box can be done a merger by client, and merger result is as shown in table 2:
| ZK | Index.php | 10 | 2012-09-2100:00:00 | Article 1,2,3,6, merger result |
| ZK | Login.php | 2 | 2012-09-2100:00:00 | Article 4, merger result |
| HGL | Index.php | 17 | 2012-09-2100:00:00 | Article 7,8, merger result |
| HGL | Login.php | 2 | 2012-09-2100:00:00 | Article 5, merger result |
Table 2
The last string of table 2 is the explanation to merger. Visible, due to the 1st in table 1,2,3 identical with the content of the first two field being identified with key of 6 data, a data therefore can be merged into, the data after merging: the first two field is still original value; 3rd field is sum-type, therefore its value be the 1st in table 1,2,3 and 6 data the 3rd field in the sum of data, be specially 10; 4th field is floor time type, and therefore its value is the initial time in this cycle. By that analogy, the 4th data merging in table 1, the 7th and the 8th data in table 1 can merge, and the 5th data in table 1 merges. Amalgamation result is referring to table 2.
So, the Data Entry inputting " Login " box in a cycle (2012-09-2100:00:00 to 2012-09-2100:04:59) is 8 data, only have sent 4 data when being sent to server end 202.
Server end 202, is suitable to receive the data that each client 201 sends, and received data is transmitted to other server, or is forwarded to database facility (such as MySQL server), or preserves into local file.
Visible, server end 202 receives the data that each client is beamed back, and server end 202 can also be transmitted to other server or database facility after receiving data, namely plays the part of " agency " role. So go for network environment or the machine room of isomery.
From above-mentioned, the system of this collection data of the present invention, owing to having carried out processing flexibly and merger to data in client, therefore the collection to daily record can be realized, application can be used for again and get statistics ready.
The composition structure of client 201 is described below.
Fig. 3 illustrates the structure chart of a kind of according to an embodiment of the invention client collecting data. As it is shown on figure 3, this client includes: data capture unit 301, merger processing unit 303 and multiple memory element 302, multiple memory element 302 distinguish corresponding different classes of business, and each memory element 302 has a corresponding timing cycle.Wherein:
Data capture unit 301 is suitable to, from the data producing the server corresponding different classes of business of acquisition, acquired data be preserved to corresponding memory element 302 according to corresponding different classes of distribution of services; Wherein, every data comprises more than one field, and different fields has different types, and at least one field identification of every data has key;
Each memory element 302 is suitable to preserve the data from data capture unit 301;
Merger processing unit 303 is suitable to when the timing cycle of each memory element 302 correspondence terminates, and the aggregation of data that the value of the field being identified with key in the data preserve this memory element 302 is identical is be sent to server end after a data.
Here, the data form of the data of one species various-service is identical, and the field number that namely data comprise is all identical with the type of each field. The form that can define data according to the actual requirements of business of all categories, the field number that namely can comprise according to practical business requirement definition one data and the type of each field.
In one embodiment of the invention, merger processing unit 303 is further adapted for when the timing cycle that every kind business is corresponding terminates, when being a data by aggregation of data identical for the value of the field being identified with key in the data of the category business preserved, to the field not identifying key, carry out different merger according to different types and process.
In one embodiment of the invention, merger processing unit 303 is further adapted for when the field not identifying key being carried out different merger process according to different types, adopts the combination of one or more of:
For the field of sum-type, the numerical value in this field of each data identical for the value being identified with the field of key is added, itself and as the value of this field after merger;
For being averaging the field of type, being averaging by the numerical value in this field of each data identical for the value being identified with the field of key, its average is as the value of this field after merger;
For the field of maximum type, from the value this field of the identical each data of the value of the field being identified with key, find out maximum, as the value of this field after merger;
For the field of normal character types, from each data that the value of the field being identified with key is identical, take the value in this field of Article 1 data, as the value of this field after merger;
For the field of cumulative character types, after the character in this field of each data identical for the value being identified with the field of key is concatenated in a designated order, as the value of this field after merger.
Fig. 4 illustrates the flow chart of a kind of according to an embodiment of the invention method collecting data. As shown in Figure 4, the method includes:
Step S410, is deployed in the client produced on server and obtains the data of corresponding different classes of business produced by this production server; Wherein, every data comprises more than one field, and different fields has different types, and at least one field identification of every data has key;
Here, the data form of the data of one species various-service is identical, and the field number that namely data comprise is all identical with the type of each field.
Step S420, acquired data are stored classifiedly by client according to corresponding different classes of business; Wherein, every kind business has a corresponding timing cycle;
Step S430, for every kind business, aggregation of data identical for the value of the field being identified with key in the data of the category business preserved, when corresponding timing cycle terminates, is be sent to server end after a data by client.
Wherein, in step S430, it is that a data includes by aggregation of data identical for the value being identified with the field of key: for not identifying the field of key, carries out different merger according to different types and process. This field for not identifying key, carries out different merger according to different field types and processes the combination including one or more of:
For the field of sum-type, the numerical value in this field of each data identical for the value being identified with the field of key is added, itself and as the value of this field after merger;
For being averaging the field of type, being averaging by the numerical value in this field of each data identical for the value being identified with the field of key, its average is as the value of this field after merger;
For the field of maximum type, from the value this field of the identical each data of the value of the field being identified with key, find out maximum, as the value of this field after merger;
For the field of normal character types, from each data that the value of the field being identified with key is identical, take the value in this field of Article 1 data, as the value of this field after merger;
For the field of cumulative character types, after the character in this field of each data identical for the value being identified with the field of key is concatenated in a designated order, as the value of this field after merger.
In sum, this difference deploying client on different production servers of the present invention, the data of collection are issued server end by each client, wherein acquired data are stored classifiedly by client according to corresponding different classes of business, every data comprises more than one different types of field, when the timing cycle that every kind business is corresponding terminates, by in the data of the category business preserved, the aggregation of data being identified with the value of the field of key identical is the technical scheme being sent to server end after a data, the data of an arbitrarily field can be transmitted, and just carried out aggregation of data process in client, thus solving existing scribe only allows every data can only have two fields of catagory and message, so that the problem that the transmission of data has many restrictions, and solve existing scribe and simply verily record data in front end, do not carry out merger process, cause that volume of transmitted data is big, transmission frequency is high, the problem easily causing network blockage and delay. technical scheme, it is possible to save bandwidth, dispose simple, easy maintenance and performance efficiency, flexible and changeable demand when technical scheme meets network data transmission to a greater extent, to log transmission.
It should be understood that
Not intrinsic to any certain computer, virtual system or miscellaneous equipment relevant in algorithm and the display of this offer. Various general-purpose systems can also with use based on together with this teaching. As described above, the structure constructed required by this kind of system is apparent from. Additionally, the present invention is also not for any certain programmed language. It is understood that, it is possible to utilize various programming language to realize the content of invention described herein, and the description above language-specific done is the preferred forms in order to disclose the present invention.
In description mentioned herein, describe a large amount of detail. It is to be appreciated, however, that embodiments of the invention can be put into practice when not having these details. In some instances, known method, structure and technology it are not shown specifically, in order to do not obscure the understanding of this description.
Similarly, it is to be understood that, one or more in order to what simplify that the disclosure helping understands in each inventive aspect, herein above in the description of the exemplary embodiment of the present invention, each feature of the present invention is grouped together in single embodiment, figure or descriptions thereof sometimes. But, the method for the disclosure should be construed to and reflect an intention that namely the present invention for required protection requires feature more more than the feature being expressly recited in each claim. More precisely, as the following claims reflect, inventive aspect is in that all features less than single embodiment disclosed above. Therefore, it then follows claims of detailed description of the invention are thus expressly incorporated in this detailed description of the invention, wherein each claim itself as the independent embodiment of the present invention.
Those skilled in the art are appreciated that, it is possible to carry out the module in the equipment in embodiment adaptively changing and they being arranged in one or more equipment different from this embodiment. Module in embodiment or unit or assembly can be combined into a module or unit or assembly, and multiple submodule or subelement or sub-component can be put them in addition. Except at least some in such feature and/or process or unit excludes each other, it is possible to adopt any combination that all processes or the unit of all features disclosed in this specification (including adjoint claim, summary and accompanying drawing) and so disclosed any method or equipment are combined. Unless expressly stated otherwise, each feature disclosed in this specification (including adjoint claim, summary and accompanying drawing) can be replaced by the alternative features providing purpose identical, equivalent or similar.
In addition, those skilled in the art it will be appreciated that, although embodiments more described herein include some feature included in other embodiments rather than further feature, but the combination of the feature of different embodiment means to be within the scope of the present invention and form different embodiments. Such as, in the following claims, the one of any of embodiment required for protection can mode use in any combination.
The all parts embodiment of the present invention can realize with hardware, or realizes with the software module run on one or more processor, or realizes with their combination. It will be understood by those of skill in the art that the some or all functions of the some or all parts of the client and server end that microprocessor or digital signal processor (DSP) can be used in practice to realize in data gathering system according to embodiments of the present invention. The present invention is also implemented as part or all the equipment for performing method as described herein or device program (such as, computer program and computer program). The program of such present invention of realization can store on a computer-readable medium, or can have the form of one or more signal. Such signal can be downloaded from internet website and obtain, or provides on carrier signal, or provides with any other form.
The present invention will be described rather than limits the invention to it should be noted above-described embodiment, and those skilled in the art can design alternative embodiment without departing from the scope of the appended claims. In the claims, any reference marks that should not will be located between bracket is configured to limitations on claims. Word " comprises " and does not exclude the presence of the element or step not arranged in the claims. Word "a" or "an" before being positioned at element does not exclude the presence of multiple such element. The present invention by means of including the hardware of some different elements and can realize by means of properly programmed computer. In the unit claim listing some devices, several in these devices can be through same hardware branch and specifically embody. Word first, second and third use do not indicate that any order. Can be title by these word explanations.
Claims (7)
1. collecting a system for data, wherein, this system includes: server end and the multiple clients being deployed on different production server,
Described client, is suitable to obtain the data of corresponding different classes of business produced by production server, acquired data is stored classifiedly according to corresponding different classes of business;
Wherein, every data comprises more than one field, and different fields has different types, and at least one field identification of every data has key; Every kind business has a corresponding timing cycle; Business of all categories defines data form according to the actual requirements, and the data form of the data of one species various-service is identical;
Described client, is further adapted for when the timing cycle that every kind business is corresponding terminates, and by the data of the category business preserved, the aggregation of data being identified with the value of the field of key identical is be sent to described server end after a data;
Described server end, is suitable to receive data from each client, and carries out storing or forwarding;
Described client, it is further adapted for when the timing cycle that every kind business is corresponding terminates, by in the data of the category business preserved, it is identified with the value of the field of key identical aggregation of data when being a data, to the field not identifying key, carry out different merger according to different types and process.
2. the system as claimed in claim 1, it is characterised in that
Described client, is further adapted for when the field not identifying key being carried out different merger process according to different types, adopts the combination of one or more of:
For the field of sum-type, the numerical value in this field of each data identical for the value being identified with the field of key is added, itself and as the value of this field after merger;
For being averaging the field of type, being averaging by the numerical value in this field of each data identical for the value being identified with the field of key, its average is as the value of this field after merger;
For the field of maximum type, from the value this field of the identical each data of the value of the field being identified with key, find out maximum, as the value of this field after merger;
For the field of normal character types, from each data that the value of the field being identified with key is identical, take the value in this field of Article 1 data, as the value of this field after merger;
For the field of cumulative character types, after the character in this field of each data identical for the value being identified with the field of key is concatenated in a designated order, as the value of this field after merger.
3. the system as according to any one of claim 1 to 2, it is characterised in that
Described server end, is suitable to be transmitted to received data other server, or is forwarded to database facility, or preserve into local file.
4. the client collecting data, wherein, this client includes: data capture unit, merger processing unit and multiple memory element, the business that the plurality of memory element is corresponding different classes of respectively, and each memory element has a corresponding timing cycle;
Described data capture unit, is suitable to, from the data producing the server corresponding different classes of business of acquisition, acquired data be preserved to corresponding memory element according to corresponding different classes of distribution of services; Wherein, every data comprises more than one field, and different fields has different types, and at least one field identification of every data has key; Business of all categories defines data form according to the actual requirements, and the data form of the data of one species various-service is identical;
Each memory element, is suitable to preserve the data from data capture unit;
Described merger processing unit, is suitable to when the timing cycle that each memory element is corresponding terminates, and the aggregation of data that the value of the field being identified with key in the data preserve this memory element is identical is be sent to server end after a data;
Described merger processing unit, it is further adapted for when the timing cycle that every kind business is corresponding terminates, when being a data by aggregation of data identical for the value of the field being identified with key in the data of the category business preserved, to the field not identifying key, carry out different merger according to different types and process.
5. client as claimed in claim 4, wherein,
Merger processing unit, is further adapted for when the field not identifying key being carried out different merger process according to different types, adopts the combination of one or more of:
For the field of sum-type, the numerical value in this field of each data identical for the value being identified with the field of key is added, itself and as the value of this field after merger;
For being averaging the field of type, being averaging by the numerical value in this field of each data identical for the value being identified with the field of key, its average is as the value of this field after merger;
For the field of maximum type, from the value this field of the identical each data of the value of the field being identified with key, find out maximum, as the value of this field after merger;
For the field of normal character types, from each data that the value of the field being identified with key is identical, take the value in this field of Article 1 data, as the value of this field after merger;
For the field of cumulative character types, after the character in this field of each data identical for the value being identified with the field of key is concatenated in a designated order, as the value of this field after merger.
6. the method collecting data, wherein, the method includes:
It is deployed in the client produced on server and obtains the data of corresponding different classes of business produced by this production server; Wherein, every data comprises more than one field, and different fields has different types, and at least one field identification of every data has key; Business of all categories defines data form according to the actual requirements, and the data form of the data of one species various-service is identical;
Acquired data are stored classifiedly by described client according to corresponding different classes of business; Wherein, every kind business has a corresponding timing cycle;
For every kind business, aggregation of data identical for the value of the field being identified with key in the data of the category business preserved, when corresponding timing cycle terminates, is be sent to server end after a data by described client;
Described is that a data includes by aggregation of data identical for the value being identified with the field of key:
For not identifying the field of key, carry out different merger according to different types and process.
7. method as claimed in claim 6, wherein, the described field for not identifying key, carry out different merger according to different field types and process the combination including one or more of:
For the field of sum-type, the numerical value in this field of each data identical for the value being identified with the field of key is added, itself and as the value of this field after merger;
For being averaging the field of type, being averaging by the numerical value in this field of each data identical for the value being identified with the field of key, its average is as the value of this field after merger;
For the field of maximum type, from the value this field of the identical each data of the value of the field being identified with key, find out maximum, as the value of this field after merger;
For the field of normal character types, from each data that the value of the field being identified with key is identical, take the value in this field of Article 1 data, as the value of this field after merger;
For the field of cumulative character types, after the character in this field of each data identical for the value being identified with the field of key is concatenated in a designated order, as the value of this field after merger.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610302731.0A CN105930502B (en) | 2012-10-22 | 2012-10-22 | System, client and method for collecting data |
| CN201210404918.3A CN102937984B (en) | 2012-10-22 | 2012-10-22 | A kind of collect the system of data, client and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210404918.3A CN102937984B (en) | 2012-10-22 | 2012-10-22 | A kind of collect the system of data, client and method |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610302731.0A Division CN105930502B (en) | 2012-10-22 | 2012-10-22 | System, client and method for collecting data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102937984A CN102937984A (en) | 2013-02-20 |
| CN102937984B true CN102937984B (en) | 2016-06-08 |
Family
ID=47696881
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610302731.0A Expired - Fee Related CN105930502B (en) | 2012-10-22 | 2012-10-22 | System, client and method for collecting data |
| CN201210404918.3A Active CN102937984B (en) | 2012-10-22 | 2012-10-22 | A kind of collect the system of data, client and method |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610302731.0A Expired - Fee Related CN105930502B (en) | 2012-10-22 | 2012-10-22 | System, client and method for collecting data |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN105930502B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104699718B (en) * | 2013-12-10 | 2019-04-12 | 阿里巴巴集团控股有限公司 | Method and apparatus for being rapidly introduced into business datum |
| CN104090938A (en) * | 2014-06-26 | 2014-10-08 | 广州金山网络科技有限公司 | Method and device for submitting data |
| CN109064317B (en) * | 2018-08-10 | 2021-04-02 | 玖富金科控股集团有限责任公司 | Data receiving and forwarding method, electronic equipment and readable storage medium |
| CN109491815A (en) * | 2018-10-17 | 2019-03-19 | 深圳壹账通智能科技有限公司 | Based on multistage data creation method, device and computer equipment |
| CN110826307A (en) * | 2019-10-31 | 2020-02-21 | 北京字节跳动网络技术有限公司 | Method and device for creating business object |
| CN110995839B (en) * | 2019-12-03 | 2022-09-20 | 北京搜狐新媒体信息技术有限公司 | Method and device for analyzing performance of advertisement system and computer storage medium |
| CN112416972A (en) * | 2020-09-25 | 2021-02-26 | 上海哔哩哔哩科技有限公司 | Real-time data stream processing method, device, equipment and readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5737600A (en) * | 1994-09-12 | 1998-04-07 | International Business Machines Corporation | Method and system for log management in a coupled data processing system |
| CN1949214A (en) * | 2006-09-26 | 2007-04-18 | 北京北大方正电子有限公司 | Information merging method and system |
| CN102637142A (en) * | 2012-04-13 | 2012-08-15 | 浪潮(北京)电子信息产业有限公司 | Computer system and method for realizing log management |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060129415A1 (en) * | 2004-12-13 | 2006-06-15 | Rohit Thukral | System for linking financial asset records with networked assets |
| CN101566986A (en) * | 2008-04-21 | 2009-10-28 | 阿里巴巴集团控股有限公司 | Method and device for processing data in online business processing |
-
2012
- 2012-10-22 CN CN201610302731.0A patent/CN105930502B/en not_active Expired - Fee Related
- 2012-10-22 CN CN201210404918.3A patent/CN102937984B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5737600A (en) * | 1994-09-12 | 1998-04-07 | International Business Machines Corporation | Method and system for log management in a coupled data processing system |
| CN1949214A (en) * | 2006-09-26 | 2007-04-18 | 北京北大方正电子有限公司 | Information merging method and system |
| CN102637142A (en) * | 2012-04-13 | 2012-08-15 | 浪潮(北京)电子信息产业有限公司 | Computer system and method for realizing log management |
Non-Patent Citations (1)
| Title |
|---|
| 统一网络安全管理中数据采集代理的设计和实现;庄欣;《中国优秀硕士学位论文全文数据库信息科技辑》;20091115(第11期);第16-61页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105930502A (en) | 2016-09-07 |
| CN102937984A (en) | 2013-02-20 |
| CN105930502B (en) | 2020-04-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102902813B (en) | Result collection system | |
| CN102937984B (en) | A kind of collect the system of data, client and method | |
| US11182098B2 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| US11582123B2 (en) | Distribution of data packets with non-linear delay | |
| US20210279265A1 (en) | Optimization for Real-Time, Parallel Execution of Models for Extracting High-Value Information from Data Streams | |
| CN106815338A (en) | A kind of real-time storage of big data, treatment and inquiry system | |
| CN107895011B (en) | Session information processing method, system, storage medium and electronic equipment | |
| CN103605651A (en) | Data processing showing method based on on-line analytical processing (OLAP) multi-dimensional analysis | |
| CN104216989A (en) | Method for storing transmission line integrated data based on HBase | |
| CN104036025A (en) | Distribution-base mass log collection system | |
| CN105490854A (en) | Real-time log collection method and system, and application server cluster | |
| CN108052679A (en) | A kind of Log Analysis System based on HADOOP | |
| US10110419B2 (en) | Alarm to event tracing | |
| CN105471670A (en) | Flow data classification method and device | |
| CN103886508A (en) | Mass farmland data monitoring method and system | |
| CN109977125A (en) | A kind of big data safety analysis plateform system based on network security | |
| US10127617B2 (en) | System for analyzing social media data and method of analyzing social media data using the same | |
| CN106682206A (en) | Method and system for big data processing | |
| CN109800133A (en) | A kind of method, one-stop monitoring alarm platform and the system of unified monitoring alarm | |
| CN112417050A (en) | Data synchronization method and device, system, storage medium and electronic device | |
| CN107257289A (en) | A kind of risk analysis equipment, monitoring system and monitoring method | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| CN109446167A (en) | A kind of storage of daily record data, extracting method and device | |
| EP3380906A1 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| CN116610531B (en) | Method for collecting data embedded points and requesting image uploading data based on code probe |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220714 Address after: Room 801, 8th floor, No. 104, floors 1-19, building 2, yard 6, Jiuxianqiao Road, Chaoyang District, Beijing 100015 Patentee after: BEIJING QIHOO TECHNOLOGY Co.,Ltd. Address before: 100088 room 112, block D, 28 new street, new street, Xicheng District, Beijing (Desheng Park) Patentee before: BEIJING QIHOO TECHNOLOGY Co.,Ltd. Patentee before: Qizhi software (Beijing) Co.,Ltd. |