CN105760236A - Data collection method and system of distributed computer cluster - Google Patents
Data collection method and system of distributed computer cluster Download PDFInfo
- Publication number
- CN105760236A CN105760236A CN201610319520.8A CN201610319520A CN105760236A CN 105760236 A CN105760236 A CN 105760236A CN 201610319520 A CN201610319520 A CN 201610319520A CN 105760236 A CN105760236 A CN 105760236A
- Authority
- CN
- China
- Prior art keywords
- data
- incremental
- module
- memory queue
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a data collection method and system of a distributed computer cluster. The method comprises the steps that a plurality of message collecting modules detect the data states of a plurality of application servers respectively, and when the data state of any application server changes, the corresponding message collecting module collects the incremental data on the application server; a message forwarding module conducts supplementing treatment on the incremental data sent by the message collecting modules and then combines the incremental data into updated data; a storage module compares the updated data with stored historical data, combines the updated data and the stored historical data and then stores the combined data according to data types. By means of concurrence, the data transmission efficiency is improved; succession, integrity and reliability are provided with crowded data transmission through an internal storage queue. A data flow is accessed in a read-only and monitoring mode, and the incremental data can be collected as long as readable data flows exist no matter how a business system is designed; the coupling degree of the system and all business systems is greatly reduced.
Description
Technical field
The present invention relates to data transfer of computer field, collecting method under espespecially a kind of distributed computer cluster and system.
Background technology
Along with the development of the technology such as Distributed Calculation, grid computing and parallel computation, bank computer system increasingly develops towards distributed type assemblies direction, with the different types of computer cooperation of multiple stage, completes the process to business.
In the computer cluster of banking system, data are disperseed on the respective computers, and the concentrated collection of data is relatively difficult, such as log information.Collecting method general at present is for setting up special application system and goes concentrated collection data, for avoiding bringing performance pressures to computer cluster, such data collecting system is typically all regularly, batch mode goes to obtain the source data on each computer, and data acquisition is not in time；Meanwhile, in order to enable accurately to obtain Various types of data, it is necessary to artificial distinguish the source path of Various types of data, frequency acquisition etc., flow process is complicated and easily makes mistakes, once corrupt data, is difficult to again to find error source is from which link.Therefore, the collecting method under existing distributed computer cluster environment exists not in time, error-prone defect, cannot meet practical application needs.
Summary of the invention
The present invention proposes the collecting method under a kind of distributed computer cluster and system, computers different in the cluster in time can obtain multiple derived data, significantly shorten the time of data acquisition, improve the efficiency of data acquisition, adopt the technological means such as Load Balance Unit and memory queue simultaneously, do not affect the stability of computer cluster, under the premise with versatility, isolate with operation system, reduce the improvement cost of operation system.
For reaching above-mentioned purpose, collecting method under distributed computer cluster provided by the present invention, specifically comprise: a plurality of message collection modules detect the data mode of a plurality of application server respectively, when the described data mode of arbitrary described application server changes, corresponding described message collection module gathers the incremental data on described application server；The a plurality of incremental datas described message collection module sent by message transmitting module are supplemented respectively after processing and are merged into more new data；Memory module stores according to data type after being combined by the historical data ratio of described more new data with storage.
In collecting method under above-mentioned distributed computer cluster, preferably, a plurality of incremental datas described message collection module sent supplement respectively merge into after processing more new data packets containing: a plurality of incremental data in chronological sequence order sent after supplementary process and for same message collection module is merged a more new data and/or a plurality of incremental data in chronological sequence order sent after supplementary process and for a plurality of message collection modules is merged a more new data.
In collecting method under above-mentioned distributed computer cluster, it is preferred that described incremental data comprises traffic type information and transaction data.
In collecting method under above-mentioned distributed computer cluster, preferably, when described data mode changes, the incremental data that corresponding described message collection module gathers on described application server comprises: described information collection module obtains the Data Source of transaction data after initializing；In the way of data stream, access data source record data stream state according to described Data Source and read position；Monitor described data flow state, obtain described incremental data when described data change.
In collecting method under above-mentioned distributed computer cluster, preferably, the supplementary process respectively of the described a plurality of incremental datas described message collection module sent by message transmitting module is comprised: described message transmitting module obtains the daily record of described incremental data；According to described incremental data, each row record in described daily record is supplemented transmitting terminal information and the transmission time of the application server sending described incremental data.
In collecting method under above-mentioned distributed computer cluster, preferably, merge into more new data after the supplementary process respectively of the described a plurality of incremental datas described message collection module sent by message transmitting module also to comprise: the described record after supplementary process is merged into more new data in chronological order；Inquire about the use state of current all memory queues, according to described memory queue use state assignment at least two memory queue send described in more new data；Said two memory queue comprises the first memory queue and the second memory queue；With time order and function order, all records in described more new data are write described first memory queue successively, when writing the first memory queue success, write successfully record is write in the second memory queue, after writing the second memory queue success, is transferred to memory module.
In collecting method under above-mentioned distributed computer cluster, preferably, the described a plurality of incremental datas described message collection module sent by message transmitting module are merged into more new data after supplementing process respectively and are also comprised: when the record in described more new data writes the first memory queue failure or when writing the second memory queue failure, re-write described first memory queue by writing failed record.
The present invention also provides for the data collecting system under a kind of distributed computer cluster, and described system comprises: message transmitting module, memory module and a plurality of information collection module；Described a plurality of message collection module is respectively arranged on the application server of a plurality of operation system, for detecting the data mode of described application server, when described data mode changes, gathers the incremental data on described application server；Described message transmitting module is connected with described a plurality of message collection modules, for collecting the incremental data that described a plurality of message collection modules obtain, a plurality of incremental datas is supplemented respectively and merges into more new data after processing；Described memory module is connected with described message transmitting module, for storing according to data type after being combined by the historical data ratio of described more new data with storage.
In data collecting system under above-mentioned distributed computer cluster, it is preferred that described system also comprises data mart modeling module, and described data mart modeling module is connected with described memory module, for described in analyzing and processing more new data and historical data merge after data.
In data collecting system under above-mentioned distributed computer cluster, it is preferred that described message transmitting module also comprises Load Balance Unit and plural number platform memory queue unit；Described Load Balance Unit is connected with described information collection module and memory queue unit respectively, state is used for receiving described incremental data and a plurality of memory queue unit of cyclic check, after described Load Balance Unit receives described incremental data, use at least two memory queue unit of state assignment that described incremental data is processed according to described memory queue unit；Described memory queue unit is for being forwarded to described memory module by described incremental data.
The Advantageous Effects of the present invention is in that: by concurrently improving the efficiency of data transmission；Succession, integrity and reliability is provided by the data transmission that memory queue is big cluster；Accessing data stream in the way of " read-only+monitoring ", no matter how operation system designs, as long as there being readable data stream, can realize the data collection of increment；Greatly reduce the degree of coupling of native system and each operation system.
Accompanying drawing explanation
Accompanying drawing described herein is used for providing a further understanding of the present invention, constitutes the part of the application, is not intended that limitation of the invention.In the accompanying drawings:
Figure 1A is the collecting method schematic flow sheet under distributed computer cluster provided by the present invention；
Figure 1B is the collecting method one example process schematic diagram under distributed computer cluster provided by the present invention；
Fig. 2 is the data collecting system structural representation under distributed computer cluster provided by the present invention；
Fig. 3 is that in the collecting method under distributed computer cluster provided by the present invention, increment obtains data flow diagram；
Fig. 4 is message forwarding process schematic diagram in the collecting method under distributed computer cluster provided by the present invention.
Detailed description of the invention
For making the purpose of the embodiment of the present invention, technical scheme and advantage clearly understand, below in conjunction with embodiment and accompanying drawing, the present invention is described in further details.At this, the schematic description and description of the present invention is used for explaining the present invention, but not as a limitation of the invention.
Refer to shown in Figure 1A, the invention provides the collecting method under a kind of distributed computer cluster, specifically comprise: a plurality of message collection modules detect the data mode of a plurality of application server respectively, when the described data mode of arbitrary described application server changes, corresponding described message collection module gathers the incremental data on described application server；The a plurality of incremental datas described message collection module sent by message transmitting module are supplemented respectively after processing and are merged into more new data；Memory module stores according to data type after being combined by the historical data ratio of described more new data with storage.
In the above-described embodiments, when described data mode changes, the incremental data that corresponding described message collection module gathers on described application server comprises: described information collection module obtains the Data Source of transaction data after initializing；In the way of data stream, access data source record data stream state according to described Data Source and read position；Monitor described data flow state, obtain described incremental data when described data change.
Wherein a plurality of incremental datas are supplemented respectively merge into after processing more new data packets containing: a plurality of incremental data in chronological sequence order sent after supplementary process and for same message collection module is merged a more new data and/or a plurality of incremental data in chronological sequence order sent after supplementary process and for a plurality of message collection modules is merged a more new data；Such as, according to time sequencing successively, multiple incremental datas are integrated into a data, transmission data volume is reduced with this, or the identification information according to application server, integrates respectively by the incremental data that each application server sends so that it is a plurality of incremental data of transmission is classified as one, when supplementing mark with this later stage, it is only necessary to supplement the transmission time of incremental data；Wherein, above-mentioned incremental data comprises traffic type information and transaction data.
Understand above-described embodiment for being more convenient for, refer to shown in Figure 1B, the collecting method under distributed computer cluster provided by the present invention is applied in real work, specifically can operate as follows:
Step 201: dispose information collection module on the application server and detect the data movement state of server, whenever there being variation event, i.e. trigger data acquisition process；
Step 202: information collection module obtains the incremental data of change, and in the way of data stream, incremental data is sent to message transmitting module.
Step 203: message transmitting module collects the incremental data sent from each Application Server information collection module, and these data is merged in chronological order in a data stream.
Step 204: in message transmitting module, it is necessary to the data collected are processed, supplements source-information, the source of labeled data, time, type etc. principally for pieces of data.
Step 205: memory module reads a piece of news from message transmitting module, with informed source, time for major key, by message data-in storehouse, does persistence.
Data in an asynchronous manner, are analyzed and calculate by step 206: data mart modeling module is according to service needed.
In the present invention one preferred embodiment, the described a plurality of incremental datas described message collection module sent by message transmitting module supplement respectively merge into after processing more new data packets containing: described message transmitting module obtains the daily record of described incremental data；According to described incremental data, each row record in described daily record is supplemented transmitting terminal information and the transmission time of the application server sending described incremental data；Thereafter, more new data is merged in chronological order in the described record after supplementary process；Inquire about the use state of current all memory queues, according to described memory queue use state assignment at least two memory queue send described in more new data；Said two memory queue comprises the first memory queue and the second memory queue；With time order and function order, all records in described more new data are write described first memory queue successively, when writing the first memory queue success, write successfully record is write in the second memory queue, after writing the second memory queue success, is transferred to memory module；Wherein, when the record in described more new data writes the first memory queue failure or when writing the second memory queue failure, re-write described first memory queue by writing failed record.
In the above-described embodiments, the acquisition flow process of described incremental data can as it is shown on figure 3, specifically comprise the following steps that
Step 301: information collection module need to initialize after starting, to remove non-essential data remanence, clearly needs thereafter the Data Source of monitoring.
Step 302: information collection module searches for each target data source, accesses each data source in the way of data stream, and record data stream state and read position.
Step 303: information collection module constantly monitors the state of data stream and the state of data: if the operating system support of destination server, it is possible to monitoring data movement event；Otherwise so that whether polling mode inquiry has data variation.
Step 304: if data change, it is meant that have new data to write in same data stream, need to obtain incremental data, then return step 302 record data stream state again and reading position is monitored.
Step 305: if data stream there occurs change, it is meant that legacy data stream is closed, and generate new data stream；Then monitoring is transferred on new data stream, reacquire and record data stream state and Data Position, and return step 302.
In the above-described embodiments, the workflow of described message transmitting module can as shown in Figure 4, specifically comprise the following steps that
Step 403: message transmitting module obtains the daily record of increment from message source by row, and each row daily record is a record.
Step 404: message transmitting module is the information that every record supplements transmitting terminal, is mainly MAC and the time of transmission of distribution of machine.
Step 405: message transmitting module passes through Load Balance Unit, it is thus achieved that at least two available memory queues.
Step 406: message transmitting module attempts record is write memory queue 1, and such as failure, then this record sends unsuccessfully；Such as success, then continue to attempt to write message memory queue 2.
Step 407: message transmitting module attempts record is write memory queue 2, and such as failure, then this record sends unsuccessfully；Such as success, then it is assumed that queue charged in this record.
Step 408: message is failure when charging to memory queue, waits a moment and continues to send this message by trial.
Step 409: message successfully charges to memory queue, waits and dumps to memory module.
Wherein, memory queue quantity can preset varying number according to actual needs, workflow still operates as above, and in daily record, each record flows through the first memory queue, the second memory queue etc. successively, is finally transferred to memory module again.
Refer to shown in Fig. 2, the present invention also provides for the data collecting system under a kind of distributed computer cluster, and described system comprises: message transmitting module, memory module and a plurality of information collection module；Described a plurality of message collection module is respectively arranged on the application server of a plurality of operation system, for detecting the data mode of described application server, when described data mode changes, gathers the incremental data on described application server；Described message transmitting module is connected with described a plurality of message collection modules, for collecting the incremental data that described a plurality of message collection modules obtain, a plurality of incremental datas is supplemented respectively and merges into more new data after processing；Described memory module is connected with described message transmitting module, for storing according to data type after being combined by the historical data ratio of described more new data with storage.
In the above-described embodiments, described system also comprises data mart modeling module, and described data mart modeling module is connected with described memory module, for described in analyzing and processing more new data and historical data merge after data.Wherein, described data mart modeling module can select to add according to actual needs, its role is to the data helping to process storage further, it is simple to when other operation systems of later stage use, it is possible to comparatively fast read.
In the above-described embodiments, described message transmitting module also comprises Load Balance Unit and plural number platform memory queue unit；Described Load Balance Unit is connected with described information collection module and memory queue unit respectively, state is used for receiving described incremental data and a plurality of memory queue unit of cyclic check, after described Load Balance Unit receives described incremental data, use at least two memory queue unit of state assignment that described incremental data is processed according to described memory queue unit；Described memory queue unit is for being forwarded to described memory module by described incremental data.
For clearer explanation above-described embodiment, below in conjunction with Fig. 2, the data collecting system under distributed computer cluster provided by the present invention is described further.
System provided by the present invention is made up of information collection module 101, message transmitting module 102, memory module 103, data mart modeling module 104, wherein:
Information collection module 101: be deployed on the application server of multiple stage operation system, for obtaining the transaction data etc. of this server；Information collection module 101 reads data message from server log, and is each daily record data supplementary data source information, such as server info, traffic type information, then is forwarded to message transmitting module 102；This information collection module 101 is by monitoring state and the Data Position of data stream, it is achieved on each application server, the increment of data is obtained, with less cost, it is thus achieved that newly-increased data, decreases decision content and the transmission quantity of data；Be conducive to collection data in large-scale cluster.
Message transmitting module 102: be responsible for receiving information collection module 101 and send the message data of coming, and forward；Message transmitting module 102 realizes short connection by Load Balance Unit with the information collection module 101 of each application server, is provided with multiple stage message queue unit, to support the connection request of magnanimity after Load Balance Unit.The data message received can be recorded internal memory and on nonmagnetic disk by each message queue, to ensure quick readwrite performance.A piece of news can recorded in two different queues, to ensure the reliability of message.
Memory module 103: obtain incremental data from message transmitting module 102, and incremental data is saved in outside storage, preservation process need to carry out merging of incremental data and data on stock.
Data mart modeling module 104: be responsible for being processed the data of collection processing, including analyzing, calculate and filing, to be supplied to follow-up system use data.
Further, above-mentioned message forwarder 102 includes Load Balance Unit, memory queue unit, wherein:
Load Balance Unit is connected with information collection apparatus 101 and memory queue unit respectively, makes regular check on the availability of each memory queue in memory queue unit；When Load Balance Unit receives the information that information collection apparatus 101 sends, namely from available memory queue, random choose is idle or load pressure is less two, will forward information to this two memory queues；So can the information that information collection module 101 sends be shared on multiple stage memory queue equably, can the handling capacity of significant increase message transmitting module 102.
Memory queue is made up of the Distributed Message Queue customized, and the example of each message queue obtains new information from information data collection device 101, and is forwarded to memorizer 103.This memory queue safeguards one piece of internal memory voluntarily, and is saved in internal memory by the message received, to ensure readwrite performance.For the information drop-out that the power traction that prevents from delaying rises in internal memory, when writing message, message is write on two different memory queues.
Collecting method under distributed computer cluster provided by the present invention and system are compared with existing transmission means, and its advantage is mainly from the following aspects:
1) data transmission efficiency is improved.Large-scale computer cluster often has various types of server, network, operating system and application system, due to communication protocol, data form, application version disunity, big assemblage classification can only be group by business by traditional method, realizes data collection and analysis in group.The invention provides a kind of method of concurrently-transmitted data in mainframe computer cluster, by concurrently improving the efficiency of data transmission.
2) ensure that the succession of information and integrity.Large-scale cluster transmits data, the order of data and completely extremely important, data incorrect order, omit or repeat to obtain wrong analysis result.Traditional method is difficult to avoid that incorrect order, omission or the repetition that personal error, systematic error bring.This method provides succession, integrity and reliability by the data transmission that memory queue is big cluster.
Achieve and the seamless combination of each application system.Traditional method needs to close data stream could obtain incremental data, otherwise just need to be read by the full detail in data stream and refilter；Cause that the operation system that need to stop on application server can obtain data, or operation system and collection system have tightly coupled design, are unfavorable for O&M, deployment and upgrading.One great feature of the present invention is to access data stream in the way of " read-only+monitoring ", and no matter how operation system designs, as long as there being readable data stream, can realize the data collection of increment；Greatly reduce the degree of coupling of native system and each operation system.
Particular embodiments described above; the purpose of the present invention, technical scheme and beneficial effect have been further described; it is it should be understood that; the foregoing is only specific embodiments of the invention; the protection domain being not intended to limit the present invention; all within the spirit and principles in the present invention, any amendment of making, equivalent replacement, improvement etc., should be included within protection scope of the present invention.
Claims (10)
1. the collecting method under a distributed computer cluster, it is characterised in that described method comprises:
A plurality of message collection modules detect the data mode of a plurality of application server respectively, and when the described data mode of arbitrary described application server changes, corresponding described message collection module gathers the incremental data on described application server；
The a plurality of incremental datas described message collection module sent by message transmitting module are supplemented respectively after processing and are merged into more new data；
Memory module stores according to data type after being combined by the historical data ratio of described more new data with storage.
2. the collecting method under distributed computer cluster according to claim 1, it is characterized in that, a plurality of incremental datas that described message collection module is sent supplement respectively merge into after processing more new data packets containing: a plurality of incremental data in chronological sequence order sent after supplementary process and for same message collection module is merged a more new data and/or a plurality of incremental data in chronological sequence order sent after supplementary process and for a plurality of message collection modules is merged a more new data.
3. the collecting method under distributed computer cluster according to claim 1, it is characterised in that described incremental data comprises traffic type information and transaction data.
4. the collecting method under distributed computer cluster according to claim 3, it is characterised in that when described data mode changes, the incremental data that corresponding described message collection module gathers on described application server comprises:
Described information collection module obtains the Data Source of transaction data after initializing；
In the way of data stream, access data source record data stream state according to described Data Source and read position；
Monitor described data flow state, obtain described incremental data when described data change.
5. the collecting method under distributed computer cluster according to claim 1, it is characterized in that, the described a plurality of incremental datas described message collection module sent by message transmitting module are supplemented process respectively and are comprised: described message transmitting module obtains the daily record of described incremental data；According to described incremental data, each row record in described daily record is supplemented transmitting terminal information and the transmission time of the application server sending described incremental data.
6. the collecting method under distributed computer cluster according to claim 5, it is characterised in that the described a plurality of incremental datas described message collection module sent by message transmitting module are merged into more new data after supplementing process respectively and also comprised:
More new data is merged in chronological order in described record after supplementary process；
Inquire about the use state of current all memory queues, according to described memory queue use state assignment at least two memory queue send described in more new data；
Said two memory queue comprises the first memory queue and the second memory queue；
With time order and function order, all records in described more new data are write described first memory queue successively, when writing the first memory queue success, write successfully record is write in the second memory queue, after writing the second memory queue success, is transferred to memory module.
7. the collecting method under distributed computer cluster according to claim 6, it is characterized in that, the described a plurality of incremental datas described message collection module sent by message transmitting module are merged into more new data after supplementing process respectively and are also comprised: when the record in described more new data writes the first memory queue failure or when writing the second memory queue failure, re-write described first memory queue by writing failed record.
8. the data collecting system under a distributed computer cluster, it is characterised in that described system comprises: message transmitting module, memory module and a plurality of information collection module；
Described a plurality of message collection module is respectively arranged on the application server of a plurality of operation system, for detecting the data mode of described application server, when described data mode changes, gathers the incremental data on described application server；
Described message transmitting module is connected with described a plurality of message collection modules, for collecting the incremental data that described a plurality of message collection modules obtain, a plurality of incremental datas is supplemented respectively and merges into more new data after processing；
Described memory module is connected with described message transmitting module, for storing according to data type after being combined by the historical data ratio of described more new data with storage.
9. the data collecting system under distributed computer cluster according to claim 8, it is characterized in that, described system also comprises data mart modeling module, and described data mart modeling module is connected with described memory module, for described in analyzing and processing more new data and historical data merge after data.
10. the data collecting system under distributed computer cluster according to claim 8, it is characterised in that described message transmitting module also comprises Load Balance Unit and plural number platform memory queue unit；
Described Load Balance Unit is connected with described information collection module and memory queue unit respectively, state is used for receiving described incremental data and a plurality of memory queue unit of cyclic check, after described Load Balance Unit receives described incremental data, use at least two memory queue unit of state assignment that described incremental data is processed according to described memory queue unit；Described memory queue unit is for being forwarded to described memory module by described incremental data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610319520.8A CN105760236A (en) | 2016-05-13 | 2016-05-13 | Data collection method and system of distributed computer cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610319520.8A CN105760236A (en) | 2016-05-13 | 2016-05-13 | Data collection method and system of distributed computer cluster |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105760236A true CN105760236A (en) | 2016-07-13 |
Family
ID=56322990
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610319520.8A Pending CN105760236A (en) | 2016-05-13 | 2016-05-13 | Data collection method and system of distributed computer cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105760236A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106485180A (en) * | 2016-09-05 | 2017-03-08 | 天津中兴智联科技有限公司 | For improving the method and system of electronic license plate rower middleware system automatic identification rate |
| CN107329834A (en) * | 2017-07-04 | 2017-11-07 | 北京百度网讯科技有限公司 | Method and apparatus for performing calculating task |
| CN107958010A (en) * | 2016-10-18 | 2018-04-24 | 北京京东尚科信息技术有限公司 | Method and system for online data migration |
| CN108052567A (en) * | 2017-12-06 | 2018-05-18 | 吉旗（成都）科技有限公司 | A kind of method that increment caches displaying with pulling time series data and exhaustive |
| CN109388481A (en) * | 2018-09-21 | 2019-02-26 | 网易（杭州）网络有限公司 | A kind of transmission method of transaction information, device, calculates equipment and medium at system |
| CN110704466A (en) * | 2019-09-27 | 2020-01-17 | 武汉极意网络科技有限公司 | Black product data storage method and device |
| CN111259081A (en) * | 2020-02-04 | 2020-06-09 | 杭州数梦工场科技有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN112506870A (en) * | 2020-12-18 | 2021-03-16 | 上海哔哩哔哩科技有限公司 | Data warehouse increment updating method and device and computer equipment |
| CN113031873A (en) * | 2021-03-26 | 2021-06-25 | 山东英信计算机技术有限公司 | IO request processing method, device, equipment and readable storage medium |
| CN113326015A (en) * | 2021-06-23 | 2021-08-31 | 北京字跳网络技术有限公司 | Information display method and device and electronic equipment |
| CN114138196A (en) * | 2021-11-26 | 2022-03-04 | 南方电网大数据服务有限公司 | Power system data storage method and device, computer equipment and storage medium |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100386755C (en) * | 2001-06-22 | 2008-05-07 | 英维思系统股份有限公司 | Method and system for collecting and retrieving time-series, real-time and non-real-time data |
| CN101334791A (en) * | 2007-06-29 | 2008-12-31 | 塞门铁克公司 | Techniques for performing intelligent content indexing |
| CN102118395A (en) * | 2011-03-11 | 2011-07-06 | 北京神舟航天软件技术有限公司 | Simple object access protocol (SOAP)-based reliable cross-domain data transmission component and method |
| CN102123413A (en) * | 2011-03-29 | 2011-07-13 | 杭州电子科技大学 | Network monitoring and protocol analysis system of wireless sensor network |
| CN102598584A (en) * | 2009-10-16 | 2012-07-18 | 瑞尔创新国际公司 | System And Method For Providing Real-time Data |
| CN102648465A (en) * | 2009-08-26 | 2012-08-22 | 杰维斯股份公司 | Remote data collection systems and methods |
| CN102970353A (en) * | 2012-11-08 | 2013-03-13 | 大唐软件技术股份有限公司 | Method and system for business data processing |
| CN103077070A (en) * | 2012-12-31 | 2013-05-01 | 无锡城市云计算中心有限公司 | Cloud computing management system and management method for cloud computing systems |
| CN104199790A (en) * | 2014-08-21 | 2014-12-10 | 北京奇艺世纪科技有限公司 | Data processing method and device |
| CN104572505A (en) * | 2015-02-03 | 2015-04-29 | 济南大学 | System and method for ensuring eventual consistency of mass data caches |
| CN104866602A (en) * | 2015-06-01 | 2015-08-26 | 走遍世界（北京）信息技术有限公司 | Queue processing method and device |
| CN105022671A (en) * | 2015-07-20 | 2015-11-04 | 中国科学院计算技术研究所 | Load balancing method for parallel processing of stream data |
| US20150332280A1 (en) * | 2014-05-16 | 2015-11-19 | Microsoft Technology Licensing, Llc | Compliant auditing architecture |
| CN105159657A (en) * | 2015-06-12 | 2015-12-16 | 北京京东尚科信息技术有限公司 | Method and system for processing message |
| CN206021244U (en) * | 2016-05-13 | 2017-03-15 | 中国工商银行股份有限公司 | A kind of data collecting system under distributed computer cluster |
-
2016
- 2016-05-13 CN CN201610319520.8A patent/CN105760236A/en active Pending
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100386755C (en) * | 2001-06-22 | 2008-05-07 | 英维思系统股份有限公司 | Method and system for collecting and retrieving time-series, real-time and non-real-time data |
| CN101334791A (en) * | 2007-06-29 | 2008-12-31 | 塞门铁克公司 | Techniques for performing intelligent content indexing |
| CN102648465A (en) * | 2009-08-26 | 2012-08-22 | 杰维斯股份公司 | Remote data collection systems and methods |
| CN102598584A (en) * | 2009-10-16 | 2012-07-18 | 瑞尔创新国际公司 | System And Method For Providing Real-time Data |
| CN102118395A (en) * | 2011-03-11 | 2011-07-06 | 北京神舟航天软件技术有限公司 | Simple object access protocol (SOAP)-based reliable cross-domain data transmission component and method |
| CN102123413A (en) * | 2011-03-29 | 2011-07-13 | 杭州电子科技大学 | Network monitoring and protocol analysis system of wireless sensor network |
| CN102970353A (en) * | 2012-11-08 | 2013-03-13 | 大唐软件技术股份有限公司 | Method and system for business data processing |
| CN103077070A (en) * | 2012-12-31 | 2013-05-01 | 无锡城市云计算中心有限公司 | Cloud computing management system and management method for cloud computing systems |
| US20150332280A1 (en) * | 2014-05-16 | 2015-11-19 | Microsoft Technology Licensing, Llc | Compliant auditing architecture |
| CN104199790A (en) * | 2014-08-21 | 2014-12-10 | 北京奇艺世纪科技有限公司 | Data processing method and device |
| CN104572505A (en) * | 2015-02-03 | 2015-04-29 | 济南大学 | System and method for ensuring eventual consistency of mass data caches |
| CN104866602A (en) * | 2015-06-01 | 2015-08-26 | 走遍世界（北京）信息技术有限公司 | Queue processing method and device |
| CN105159657A (en) * | 2015-06-12 | 2015-12-16 | 北京京东尚科信息技术有限公司 | Method and system for processing message |
| CN105022671A (en) * | 2015-07-20 | 2015-11-04 | 中国科学院计算技术研究所 | Load balancing method for parallel processing of stream data |
| CN206021244U (en) * | 2016-05-13 | 2017-03-15 | 中国工商银行股份有限公司 | A kind of data collecting system under distributed computer cluster |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106485180A (en) * | 2016-09-05 | 2017-03-08 | 天津中兴智联科技有限公司 | For improving the method and system of electronic license plate rower middleware system automatic identification rate |
| CN107958010A (en) * | 2016-10-18 | 2018-04-24 | 北京京东尚科信息技术有限公司 | Method and system for online data migration |
| CN107958010B (en) * | 2016-10-18 | 2020-09-01 | 北京京东尚科信息技术有限公司 | Method and system for online data migration |
| CN107329834A (en) * | 2017-07-04 | 2017-11-07 | 北京百度网讯科技有限公司 | Method and apparatus for performing calculating task |
| CN108052567A (en) * | 2017-12-06 | 2018-05-18 | 吉旗（成都）科技有限公司 | A kind of method that increment caches displaying with pulling time series data and exhaustive |
| CN109388481B (en) * | 2018-09-21 | 2021-08-17 | 网易（杭州）网络有限公司 | Transaction information transmission method, system, device, computing equipment and medium |
| CN109388481A (en) * | 2018-09-21 | 2019-02-26 | 网易（杭州）网络有限公司 | A kind of transmission method of transaction information, device, calculates equipment and medium at system |
| CN110704466A (en) * | 2019-09-27 | 2020-01-17 | 武汉极意网络科技有限公司 | Black product data storage method and device |
| CN110704466B (en) * | 2019-09-27 | 2021-12-17 | 武汉极意网络科技有限公司 | Black product data storage method and device |
| CN111259081A (en) * | 2020-02-04 | 2020-06-09 | 杭州数梦工场科技有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN112506870A (en) * | 2020-12-18 | 2021-03-16 | 上海哔哩哔哩科技有限公司 | Data warehouse increment updating method and device and computer equipment |
| CN113031873A (en) * | 2021-03-26 | 2021-06-25 | 山东英信计算机技术有限公司 | IO request processing method, device, equipment and readable storage medium |
| CN113031873B (en) * | 2021-03-26 | 2023-02-28 | 山东英信计算机技术有限公司 | IO request processing method, device, equipment and readable storage medium |
| CN113326015A (en) * | 2021-06-23 | 2021-08-31 | 北京字跳网络技术有限公司 | Information display method and device and electronic equipment |
| CN114138196A (en) * | 2021-11-26 | 2022-03-04 | 南方电网大数据服务有限公司 | Power system data storage method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105760236A (en) | Data collection method and system of distributed computer cluster | |
| CN102546730B (en) | Data processing method, Apparatus and system | |
| CN102024044B (en) | Distributed file system | |
| US7624118B2 (en) | Data processing over very large databases | |
| CN101997823B (en) | Distributed file system and data access method thereof | |
| US11403269B2 (en) | Versioning validation for data transfer between heterogeneous data stores | |
| US12111794B2 (en) | Replication barriers for dependent data transfers between data stores | |
| CN101187888A (en) | Method for coping database data in heterogeneous environment | |
| CN206021244U (en) | A kind of data collecting system under distributed computer cluster | |
| US11221785B2 (en) | Managing replication state for deleted objects | |
| CN104933114A (en) | Mass log management cloud platform | |
| US11567899B2 (en) | Managing dependent delete operations among data stores | |
| CN102750326A (en) | Log management optimization method of cluster system based on downsizing strategy | |
| CN107515933A (en) | Change method, server, system and the storage medium of source data database table structure | |
| CN103778133A (en) | Database object changing method and device | |
| CN106850258A (en) | A kind of Log Administration System, method and device | |
| CN103716174A (en) | Test log acquisition system and method for the same | |
| CN107272669B (en) | A kind of airborne Fault Management System | |
| CN104584524A (en) | Aggregating data in a mediation system | |
| CN104158757A (en) | GPS (Global Positioning System) data loading and processing method and system | |
| CN105843936A (en) | Service data report form method and system | |
| CN110389933A (en) | Blog management method and device between a kind of process | |
| CN113067717A (en) | Network request log chain tracking method, full link call monitoring system and medium | |
| CN104484131A (en) | Device and corresponding method for processing data of multi-disk servers | |
| CN117041269A (en) | Real-time bidirectional synchronization system for synchronizing sub-database data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160713 |