CN113032375A - Data acquisition and aggregation method based on Flume - Google Patents
Data acquisition and aggregation method based on Flume Download PDFInfo
- Publication number
- CN113032375A CN113032375A CN201911350150.4A CN201911350150A CN113032375A CN 113032375 A CN113032375 A CN 113032375A CN 201911350150 A CN201911350150 A CN 201911350150A CN 113032375 A CN113032375 A CN 113032375A
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- flume
- layers
- acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Quality & Reliability (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a Flume-based data acquisition and aggregation method, which comprises the following steps: distributing data sources with different formats to source layers in different acquisition layers, wherein the source layers send acquired original data to channel layers for processing data with different formats; the channel layer sets a data cleaning rule and performs data cleaning, the cleaned data is sent to sink layers of different format data of the acquisition layer Flume, and the sink layers are sent to source layers of convergence layer Flume of the different format data; sending the data acquired by the source layer of the convergence layer Flume to channel layers of the convergence layer flumes with different formats of data; the channel layer sends the acquired convergence data to a sink layer of the convergence layer Flume, and the sink layer sends the converged data to different data receivers or actively pulls the sink data of the convergence layer by different receivers. The invention respectively combines the Flume into the acquisition layer and the convergence layer, flexibly processes through the corresponding relation configuration of the acquisition layer and the convergence layer, realizes the filtration of various data and supports the receiving requirements of the receiving party on the various data.
Description
Technical Field
The invention belongs to the technical field of big data, and particularly relates to a Flume-based data acquisition and aggregation method.
Background
In the field of big data technology, data is the basis. Data collection and aggregation are the main ways and channels for collecting data from various sources. Data from different sources has multiple data formats and encodings, and the data quality is also uneven. In order to acquire data as comprehensive and detailed as possible, multiple data from various sources are generally acquired, so that a data acquisition system needs to be capable of supporting high-concurrency, high-throughput and high-quality data acquisition services; different platforms using big data need specific service data respectively, and the required data formats are also very different. Therefore, there is a need for an acquisition system that can support not only acquisition and cleaning of data in multiple formats, but also output of data in multiple formats, and also support flexible linear expansion.
The traditional data acquisition technology based on the flash generally only uses the flash as an efficient cache module, or uses the flash as a module for converting a plurality of input data into streaming data to be output, for example, the flash is docked with spark streaming to realize real-time acquisition and processing of the streaming data, and the flash is docked with Hadoop to realize distributed storage of the data. As shown in fig. 1, SparkStreaming and Hadoop both belong to a data receiver, and the support of multiple data sources is realized by means of a source layer of flash; the channel layer is used for realizing the high-efficiency caching of the data of multiple data sources; and outputting data of different receivers by means of the sink layer.
In the actual use process of the traditional data acquisition technology using the Flume, a plurality of source are configured to realize the acquisition of a plurality of data formats, and a plurality of sink are configured to realize the output support of a plurality of receivers. The following problems are encountered in a production environment:
(1) when a performance bottleneck occurs, the capacity expansion is inconvenient;
(2) when data sources with different formats need different data cleaning rules, the configuration and the implementation are inconvenient and flexible;
(3) it is difficult to implement when different data receivers require full or differentiated data.
Disclosure of Invention
In order to solve the technical problems, the invention provides a Flume-based data acquisition and aggregation method, which comprises a data source, an acquisition layer Flume, an aggregation layer Flume and a receiver, wherein the acquisition layer Flume comprises a source layer, a channel layer and a sink layer, the aggregation layer Flume also comprises a source layer, a channel layer and a sink layer, and the Flume-based data acquisition and aggregation method comprises the following steps:
s1: distributing data sources with different formats to source layers in different independent acquisition layers, wherein the source layers with different formats send acquired original data to channel layers for processing data with different formats;
s2: an interceptor is arranged between source layers of data in different formats and channel layers of data in different formats for processing, sets data cleaning rules for the data of different data sources and cleans the data, classifies the cleaned and filtered data, outputs invalid data which do not accord with the cleaning and filtering rules to an invalid data processing type channel layer for discarding, and outputs valid data which accord with the cleaning and filtering rules to a channel layer of a valid data processing type;
s3: the method comprises the steps that channel layers of different format data processing of an acquisition layer Flume clean data and send the cleaned data to sink layers of different format data of the acquisition layer Flume, and the sink layers of the different format data of the acquisition layer Flume acquire the cleaned data and push the data to source layers of one or more convergence layers Flume;
s4: the method comprises the steps that a source layer of a convergence layer of data with different formats acquires cleaned data from sink layers of a plurality of acquisition layers and sends the data to the source layers of the convergence layer of the data with different formats, and the source layers of the convergence layer of the data with different formats screen and combine the acquired data according to configuration rules and send the screened and combined data to channel layers of the convergence layer of the data with different formats;
s5: the channel layers of the convergence layers of the Flume with the data in different formats converge the acquired data, the data are divided into streaming data and non-streaming data according to the requirements of a receiver, the streaming data and the non-streaming data are respectively divided into sink layers of the corresponding convergence layers of the Flume, and the sink layers of the convergence layers of the Flume send the converged data to different data receivers.
Preferably, the cleansing rule is to preset corresponding data cleansing rules for data sources in different formats.
Preferably, the sink layer of the acquisition layer Flume with the data in different formats acquires the cleaned data and sends the data to the source layer of the convergence layer Flume with the data in different formats, rather than directly sending the data to other data storage parties or file systems.
Preferably, the source layer of the aggregation layer Flume with the data in different formats supports distributed acquisition cluster service to automatically output the aggregated data, so that the cleaned data can be acquired from the sink layers of a plurality of acquisition layers, one-stage, two-stage or even multi-stage longitudinal expansion of the acquisition layer Flume is realized, a data acquisition matrix can be formed by the acquisition layer Flume and the transversely expanded Flume, and higher-performance expansion and support of data acquisition are realized.
Preferably, the sink layer of the convergence layer Flume with the data in different formats supports providing different convergence data for different data receivers, so that the converged data can be sent to different data receivers.
Preferably, the number of the acquisition layer Flume and the number of the convergence layer Flume are both multiple, the acquisition layer Flume and the convergence layer Flume support the transverse expansion, the configuration can be carried out according to different data convergence output requirements and concurrency requirements, the linear expansion of the data acquisition capacity can be conveniently realized, the enhancement of the support of new data types can be quickly realized, and the promotion of the data acquisition capacity can be conveniently realized.
Compared with the prior art, the invention has the beneficial effects that: the Flume is respectively combined into the acquisition layer and the convergence layer, and the corresponding relation between the acquisition layer and the convergence layer is flexibly processed through configuration, so that the respective clearness and filtration of various data are realized, and the requirements of a receiver on the reception of various data are supported.
Drawings
FIG. 1 is a flow chart of conventional Flume data collection of internal data;
FIG. 2 is a big data collection flow chart of a FLUME-based data collection and aggregation method.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention is further described below:
example (b):
as shown in fig. 2, a big data collection and aggregation method supporting high concurrency, linear expansion, multiple data format input and cleaning, and multiple data receivers is provided: the method comprises a data source, an acquisition layer Flume, a convergence layer Flume and a receiver, wherein the acquisition layer Flume comprises a source layer, a channel layer and a sink layer, the convergence layer Flume also comprises a source layer, a channel layer and a sink layer, and the steps of the Flume-based data acquisition and convergence method are as follows:
s1: distributing data sources with different formats to source layers in different acquisition layer flumes, pointing http-format data to http-format data source directories of the source layers in configuration files of the acquisition layer flumes, and monitoring changes of files in the http protocol data acquisition directories by the source layers in real time to automatically acquire the http-format data; the data in the ftp format point to an ftp format data source directory of a source layer in a configuration file of an acquisition layer Flume, and the source layer monitors the change of a file in an ftp protocol data acquisition directory in real time to automatically acquire the data in the ftp format; the data of the database source points to a database source data source directory of a source layer in a configuration file of the acquisition layer Flume, and the source layer monitors the change of a file under a database content pull directory in real time to automatically acquire the data in the database; the source layers of the data in different formats send the acquired original data to channel layers for processing the data in different formats;
s2: an interceptor exists between source layers of data in different formats and channel layers for processing data in different formats, the interceptor sets corresponding data cleaning rules for data sources in different formats in advance, performs data cleaning, classifies the cleaned and filtered data, outputs invalid data which do not accord with the cleaning and filtering rules to an invalid data processing type channel layer for discarding, and outputs valid data which accord with the cleaning and filtering rules to a channel layer of a valid data processing type;
s3: the method comprises the steps that channel layers of different format data processing of an acquisition layer Flume clean data and send the cleaned data to sink layers of different format data of the acquisition layer Flume, and the sink layers of the different format data of the acquisition layer Flume acquire the cleaned data and push the data to source layers of one or more convergence layers Flume;
s4: the method comprises the steps that a source layer of a convergence layer Flume with different format data in a distributed file storage HDFS system obtains cleaned data from a sink layer of the convergence layer Flume with different format data, the source layer of the convergence layer Flume with different format data screens and combines the obtained data according to configuration rules, the screened and combined data are sent to channel layers of the convergence layer Flume with different format data, the channel layers of the convergence layer Flume with different format data filter and cache the gathered various data and send the data to the sink layer of the convergence layer Flume with different format data, the sink layer of the convergence layer Flume with different format data receives the data pushed by the channel layers of the convergence layer Flume with different format data and writes the gathered data into the HDFS file system, and the function that the collected data of multiple data sources are all gathered and then output to the distributed file system is achieved;
s5: the method comprises the steps that a source layer of a convergence layer Flume with different format data in a real-time statistical analysis system obtains cleaned real-time data from a sink layer of a collection layer Flume with different format data, the source layer of the convergence layer Flume with different format data screens and combines the obtained data according to configuration rules, the screened and combined data are sent to channel layers of the convergence layer Flume with different format data, the channel layers of the convergence layer Flume with different format data filter and cache the gathered real-time data, the filtered and buffered real-time data are sent to the sink layers of the convergence layer Flume with different format data, the sink layers of the convergence layer Flume with different format data receive the real-time data pushed by the channel layers of the convergence layer Flume with different format data, the gathered real-time data are subjected to statistical analysis, and a statistical analysis chart used for a statistical analysis data receiver to display real-time source data is obtained.
Specifically, a convergence agent composed of a source layer, a channel layer and a sink layer of a convergence layer Flume of the HDFS file system converges data of various sources of all acquisition agents and outputs the data to the distributed file system; the aggregation agent consisting of the source layer, the channel layer and the sink layer of the aggregation layer Flume subjected to real-time statistical analysis only aggregates data from all http sources and outputs the data to the real-time statistical analysis module in real time; through the difference example of the two different data receivers for the required data, it is described that the convergence layer Flume can be flexibly configured according to the data requirements of the data receivers, can converge all the data of the collection agent, and can also converge only the data of a specific type.
Specifically, the sink layer of the acquisition layer Flume with the different format data acquires the cleaned data and sends the data to the source layer of the convergence layer Flume with the different format data, rather than directly sending the data to other data storage parties or file systems.
Specifically, the source layer of the aggregation layer Flume with the data in different formats supports distributed acquisition cluster service to automatically output the aggregated data, so that the cleaned data can be acquired from sink layers of a plurality of acquisition layers, one-stage, two-stage or even multi-stage longitudinal expansion of the acquisition layer Flume is realized, a data acquisition matrix can be formed by the acquisition layer Flume and transversely expanded Flume, and higher-performance expansion and support of data acquisition are realized.
Specifically, the sink layer of the convergence layer Flume with the data in different formats supports providing differentiated convergence data for different data receivers, so that the converged data can be sent to different data receivers.
Specifically, the quantity of acquisition layer Flume and the quantity of convergence layer Flume are a plurality of, and the quantity of acquisition layer Flume and the quantity of convergence layer Flume support lateral expansion, can assemble output demand and concurrency needs configuration according to different data, and the linear dilatation of realization data collection ability that can be very convenient not only can be quick to the reinforcing that new data type supported, the promotion of realization data collection ability that moreover can be convenient.
According to the method and the idea of the embodiment of the present invention, a technician can flexibly adjust the butt-joint relationship between the acquisition Flume and the convergence Flume, and can design various changes or variations of applicable scenes, and all of the changes and variations based on the technical idea of the present invention should belong to the protection scope of the claims of the present invention.
It should be noted that, in this document, moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (6)
1. The data acquisition and aggregation method based on the Flume is characterized by comprising a data source, an acquisition layer Flume, an aggregation layer Flume and a receiver, wherein the acquisition layer Flume comprises a source layer, a channel layer and a sink layer, the aggregation layer Flume also comprises a source layer, a channel layer and a sink layer, and the steps of the data acquisition and aggregation method based on the Flume are as follows:
s1: distributing data sources with different formats to source layers in different acquisition layers, wherein the source layers with different formats send acquired original data to channel layers for processing data with different formats;
s2: an interceptor exists between the source layer of the data in different formats and the channel layer of the data in different formats, and the interceptor sets data cleaning rules for the data of different data sources and performs data cleaning;
s3: the method comprises the steps that channel layers of different format data processing of an acquisition layer Flume clean data and send the cleaned data to sink layers of different format data of the acquisition layer Flume, and the sink layers of the different format data of the acquisition layer Flume acquire the cleaned data and send the cleaned data to source layers of convergence layer Flume of the different format data;
s4: the source layers of the convergence layers of the data with different formats acquire cleaned data from sink layers of the multiple acquisition layers and send the data to the source layers of the convergence layers of the data with different formats, and the source layers of the convergence layers of the data with different formats send the acquired data to channel layers of the convergence layers of the data with different formats;
s5: the channel layers of the convergence layers of the data with different formats send the acquired convergence data to sink layers of the convergence layers of the data with different formats, and the sink layers of the convergence layers of the data with different formats send the converged data to different data receivers.
2. The Flume-based data collection and aggregation method according to claim 1, wherein the cleansing rules are corresponding data cleansing rules preset for data sources with different formats.
3. The Flume-based data acquisition and aggregation method according to claim 1, wherein the sink layer of the acquisition layer Flume with the data in different formats acquires the cleaned data and sends the cleaned data to the source layer of the convergence layer Flume with the data in different formats, rather than directly sending the cleaned data to other data storage parties or file systems.
4. The Flume-based data acquisition and aggregation method according to claim 1, wherein the source layer of the aggregation layer Flume with different format data supports a distributed acquisition cluster service to automatically output aggregated data, so that cleaned data can be acquired from a sink layer of a plurality of acquisition layers.
5. The Flume-based data acquisition and aggregation method according to claim 1, wherein the sink layer of the Flume with different format data supports providing different aggregation data for different data receivers, so that the aggregated data can be sent to different data receivers.
6. The Flume-based data acquisition and convergence method according to claim 1, wherein the number of the acquisition layer Flume and the convergence layer Flume is multiple, and the number of the acquisition layer Flume and the convergence layer Flume supports horizontal expansion and can be configured according to different data convergence output requirements and concurrency requirements.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911350150.4A CN113032375A (en) | 2019-12-24 | 2019-12-24 | Data acquisition and aggregation method based on Flume |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911350150.4A CN113032375A (en) | 2019-12-24 | 2019-12-24 | Data acquisition and aggregation method based on Flume |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113032375A true CN113032375A (en) | 2021-06-25 |
Family
ID=76451925
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911350150.4A Pending CN113032375A (en) | 2019-12-24 | 2019-12-24 | Data acquisition and aggregation method based on Flume |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113032375A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150317231A1 (en) * | 2010-03-31 | 2015-11-05 | Cloudera, Inc. | Collecting and aggregating log data with fault tolerance |
| CN107590182A (en) * | 2017-08-03 | 2018-01-16 | 华南理工大学 | A kind of distributed information log collection method |
| CN110502491A (en) * | 2019-07-25 | 2019-11-26 | 北京神州泰岳智能数据技术有限公司 | A kind of Log Collect System and its data transmission method, device |
-
2019
- 2019-12-24 CN CN201911350150.4A patent/CN113032375A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150317231A1 (en) * | 2010-03-31 | 2015-11-05 | Cloudera, Inc. | Collecting and aggregating log data with fault tolerance |
| CN107590182A (en) * | 2017-08-03 | 2018-01-16 | 华南理工大学 | A kind of distributed information log collection method |
| CN110502491A (en) * | 2019-07-25 | 2019-11-26 | 北京神州泰岳智能数据技术有限公司 | A kind of Log Collect System and its data transmission method, device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101569170B (en) | Encoding device, encoding method, decoding device, and decoding method | |
| JP5567501B2 (en) | Synchronization method between customer PVR and network-based DVR | |
| US20110026521A1 (en) | Apparatus and methods for forwarding data packets captured from a network | |
| CN101534213B (en) | Acquisition method of log and log server | |
| CN101543077A (en) | Information processing device and method | |
| US20110026406A1 (en) | Apparatus and methods for capturing data packets from a network | |
| CN105472400A (en) | Message pushing method and system | |
| CN102100082A (en) | Information processing device and method | |
| CN103595577B (en) | System and method for monitoring whether traffic of interconnection interfaces between ISPs exceeds threshold valve | |
| CN106791889A (en) | A kind of method for processing video frequency and system, distributed objects storage system | |
| US20100278068A1 (en) | Method and System for Filtering IP Traffic in Mobile IP Networks | |
| CN106250410A (en) | A kind of data processing method based on flume system and device thereof | |
| CN111209467A (en) | Data real-time query system under multi-concurrency multi-channel environment | |
| CN102543087A (en) | Audio mixing method applied to multi-point mobile audio communication system | |
| CN113032375A (en) | Data acquisition and aggregation method based on Flume | |
| KR101408032B1 (en) | Distribution System for analysing massive traffic in real time and method thereof | |
| CN1881931A (en) | Multicast forwarding route aggregating method | |
| CN103347199B (en) | A kind of live audio method and system | |
| CN107085579A (en) | A kind of data acquisition distribution method and device | |
| CN105359539A (en) | Content provision device, content provision method, program, terminal device, and content provision system | |
| CN101663876A (en) | Be used in the method and apparatus of network components deal with data and the system that comprises this device | |
| CN106507134A (en) | A kind of method of streaming media server transcoding on demand | |
| CN205693815U (en) | A kind of ultra high-definition transmission of video and storage system | |
| CN104618266A (en) | Method and device for transferring messages among a plurality of ports | |
| CN102547789A (en) | Early warning method, device and system for quality of peer-to-peer service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210625 |
|
| RJ01 | Rejection of invention patent application after publication |