CN110704442A - Real-time acquisition method and device for big data - Google Patents
Real-time acquisition method and device for big data Download PDFInfo
- Publication number
- CN110704442A CN110704442A CN201910933843.XA CN201910933843A CN110704442A CN 110704442 A CN110704442 A CN 110704442A CN 201910933843 A CN201910933843 A CN 201910933843A CN 110704442 A CN110704442 A CN 110704442A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- data table
- table structure
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a real-time acquisition method and a real-time acquisition device for big data, wherein the method comprises the following steps: acquiring target data and a source data table structure corresponding to a target database through a data configuration center, wherein the target database comprises the target data; performing first processing on the target data to obtain target warehousing data, and synchronizing a storage data table structure corresponding to a target storage database with the source data table structure to obtain a target storage data table structure; and writing the target warehousing data and the target storage data table structure into a target storage database after second processing. The invention can synchronize the table structure and the big data in real time and intelligently acquire the big data in real time.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a method and a device for acquiring big data in real time.
Background
Big data systems are generally divided into several major layers of data acquisition, data computation, data service, and data application. In a data acquisition layer, log acquisition and data source data synchronization are mainly divided. Big data refers to a huge data set collected from many sources in a multivariate manner, and is often real-time. In the case of business-to-business sales, this data may be obtained from social networks, e-commerce websites, and customer visit records, among many other sources. These data are not normal data sets of the corporate customer relationship management database. Technically, the relation between big data and cloud computing is as inseparable as the front and back of a coin. The big data cannot be processed by a single computer necessarily, and a distributed computing architecture must be adopted. The method is characterized by mining mass data, but the method must rely on distributed processing, distributed databases, cloud storage and/or virtualization technologies of cloud computing. The existing big data acquisition or synchronization program has the following problems: firstly, a common data acquisition program for big data is not intelligent enough to use, and the maintenance workload is large; secondly, real-time synchronous data are complex in configuration and cannot automatically identify and synchronize a table structure; finally, the existing real-time synchronization procedures are fewer, and the synchronization efficiency is lower.
Disclosure of Invention
The embodiment of the invention provides a method and a device for acquiring big data in real time, and aims to solve the problems that in the prior art, a big data acquisition method cannot achieve real-time and intelligent acquisition, the data and table structure synchronization efficiency is low, and the maintenance workload is large.
In order to solve the above technical problem, a first technical solution adopted in the embodiments of the present invention is as follows:
a method of real-time acquisition of big data, comprising: acquiring target data and a source data table structure corresponding to a target database through a data configuration center, wherein the target database comprises the target data; performing first processing on the target data to obtain target warehousing data, and synchronizing a storage data table structure corresponding to a target storage database with the source data table structure to obtain a target storage data table structure; performing second processing on the target warehousing data and the target storage data table structure and writing the target warehousing data and the target storage data table structure into a target storage database; the data configuration center is configured before the target data is acquired, and the data stored by the data configuration center comprises related parameter data, a server address and a database address which are involved in the process of acquiring the big data in real time.
Optionally, the obtaining, by the data configuration center, the target data and the source data table structure corresponding to the target database includes: acquiring a target database address corresponding to the target data through a data configuration center; and accessing the target database address to enter a target database, and acquiring the target data and the source data table structure corresponding to the target database from the target database.
Optionally, the performing a first process on the target data to obtain target warehousing data includes: analyzing the designated field of the target data, and cleaning the non-standard data of the target data to obtain the target warehousing data.
Optionally, the synchronizing a storage data table structure corresponding to the target storage database with the source data table structure to obtain a target storage data table structure includes: judging whether the structure of the storage data table is the same as that of the source data table; and if not, updating the storage data table structure by taking the source data table structure as a standard table structure to obtain a target storage data table structure, otherwise, keeping the storage data table structure unchanged, wherein the target storage data table structure is the same as the source data table structure.
Optionally, the determining whether the storage data table structure is the same as the source data table structure includes: obtaining a latest cache data table structure from a cache, wherein the latest cache data table structure is the same as the storage data table structure; judging whether the structure of the storage data table is the same as that of the cache data table or not; and if so, judging that the structure of the storage data table is the same as that of the source data table, otherwise, judging that the structure of the storage data table is different from that of the source data table.
Optionally, the writing the target warehousing data and the target storage data table structure into the target storage database after the second processing includes: and writing the target warehousing data and the target storage data table structure into a cache, and writing the target warehousing data and the target storage data table structure which are written into the cache into the target storage database.
Optionally, before the obtaining, by the data configuration center, the target data and the source data table structure corresponding to the target database, the method includes: acquiring a target data code corresponding to a new service every preset time, wherein the target data code is used for connecting the data configuration center; and acquiring the target data and the source data table structure corresponding to the target database through the data configuration center according to the target data code.
In order to solve the above technical problem, a second technical solution adopted in the embodiments of the present invention is as follows:
a device for real-time acquisition of big data, comprising: the data acquisition module is used for acquiring target data and a source data table structure corresponding to a target database through a data configuration center, wherein the target database comprises the target data; the first processing module is used for carrying out first processing on the target data to obtain target warehousing data, and synchronizing a storage data table structure corresponding to a target storage database with the source data table structure to obtain a target storage data table structure; the second processing module is used for writing the target warehousing data and the target storage data table structure into a target storage database after second processing; the data configuration center is configured before the target data is acquired, and the data stored by the data configuration center comprises related parameter data, a server address and a database address which are involved in the process of acquiring the big data in real time.
In order to solve the above technical problem, a third technical solution adopted in the embodiments of the present invention is as follows:
a computer-readable storage medium, on which a computer program is stored, which, when executed, implements a real-time acquisition method of big data as described above.
In order to solve the above technical problem, a fourth technical solution adopted in the embodiments of the present invention is as follows:
a computer device comprising a processor, a memory and a computer program stored on the memory and executable on the processor, the processor implementing the method of real-time acquisition of big data as described above when executing the computer program.
The embodiment of the invention has the beneficial effects that: different from the situation in the prior art, the embodiment of the invention obtains the target data and the source data table structure corresponding to the target database through the data configuration center, performs the first processing on the target data to obtain the target warehousing data, then synchronizes the storage data table structure corresponding to the target storage database with the source data table structure to obtain the target storage data table structure, and finally performs the second processing on the target warehousing data and the target storage data table structure and writes the target warehousing data and the target storage data table structure into the target storage database.
Drawings
Fig. 1 is an implementation flowchart of an embodiment of a method for acquiring big data in real time according to a first embodiment of the present invention;
FIG. 2 is a partial structural framework diagram of an embodiment of a real-time big data acquisition apparatus according to a second embodiment of the present invention;
FIG. 3 is a partial structural framework diagram of an embodiment of a computer-readable storage medium according to a third embodiment of the present invention;
fig. 4 is a partial structural framework diagram of an embodiment of a computer device according to a fourth embodiment of the present invention.
Detailed Description
Example one
Referring to fig. 1, fig. 1 is a flowchart illustrating an implementation of a method for acquiring big data in real time according to an embodiment of the present invention, which can be obtained by referring to fig. 1, and the method for acquiring big data in real time according to the present invention includes:
step S101: and acquiring target data and a source data table structure corresponding to a target database through a data configuration center, wherein the target database comprises the target data.
Step S102: and performing first processing on the target data to obtain target warehousing data, and synchronizing a storage data table structure corresponding to a target storage database with the source data table structure to obtain a target storage data table structure.
Step S103: and writing the target warehousing data and the target storage data table structure into a target storage database after second processing.
The data configuration center is configured before the target data is acquired, and the data stored by the data configuration center comprises related parameter data, a server address and a database address which are involved in the process of acquiring the big data in real time.
In this embodiment, optionally, various configuration operations are completed through a unified configuration center, and various parameters, function switches, server addresses and the like corresponding to the big data acquisition method are extracted and placed in the configuration center, so that the method is simple and easy to maintain; the configuration center is divided into development, test and production environments, the configuration modification is effective in real time corresponding to the configuration of each environment, and after the configuration modification, the configuration center can update the version of the big data acquisition method in real time.
In this embodiment, optionally, the obtaining, by the data configuration center, the target data and the source data table structure corresponding to the target database includes:
firstly, a target database address corresponding to the target data is obtained through a data configuration center.
Secondly, accessing the target database address to enter a target database, and acquiring the target data and the source data table structure corresponding to the target database from the target database.
In this embodiment, optionally, the performing a first process on the target data to obtain target warehousing data includes:
analyzing the designated field of the target data, and cleaning the non-standard data of the target data to obtain the target warehousing data.
In this embodiment, optionally, the synchronizing the storage data table structure corresponding to the target storage database with the source data table structure to obtain the target storage data table structure includes:
and judging whether the structure of the storage data table is the same as that of the source data table.
And if the storage data table structure is different from the source data table structure, updating the storage data table structure by taking the source data table structure as a standard table structure to obtain a target storage data table structure. If the storage data table structure is the same as the source data table structure, keeping the storage data table structure unchanged, wherein the target storage data table structure is the same as the source data table structure.
In this embodiment, optionally, the determining whether the structure of the storage data table is the same as the structure of the source data table includes:
and obtaining the latest cache data table structure from the cache, wherein the latest cache data table structure is the same as the storage data table structure.
And judging whether the structure of the storage data table is the same as that of the cache data table.
And if the structure of the storage data table is the same as that of the cache data table, judging that the structure of the storage data table is the same as that of the source data table, otherwise, judging that the structure of the storage data table is different from that of the source data table.
In this embodiment, optionally, the writing the target warehousing data and the target storage data table structure into the target storage database after the second processing includes:
and writing the target warehousing data and the target storage data table structure into a cache, and writing the target warehousing data and the target storage data table structure which are written into the cache into the target storage database.
In this embodiment, optionally, the present invention implements multithreading real-time warehousing, which has high warehousing efficiency, and specifically includes: and establishing a thread pool, reading the thread number in the thread pool through configuration, adjusting the size of the thread pool according to the size of the data volume, and distributing one thread to each data table to synchronize data, wherein the maximum of the thread number in the same time is the size of the thread pool.
In this embodiment, optionally, the present invention may implement automatic compression of various files, and merge small files, where the compression format is: the data synchronized in real time is stored in a queue format (a storage format), the bottom layer of the data storage is stored in files, and the files are small files when the files are smaller than HDFS (Hadoop distributed file system) file blocks (default 128 m). When the number of the small files exceeds 1000, the query performance of the data table is greatly reduced, the corresponding program of the invention incrementally combines the files of all the tables in the previous day every day, and the number of the combined files can be set according to the size of the tables.
In this embodiment, optionally, before the obtaining, by the data configuration center, the target data and the source data table structure corresponding to the target database, the method includes:
firstly, acquiring a target data code corresponding to a new service every preset time, wherein the target data code is used for connecting the data configuration center.
Secondly, according to the target data code, the target data and the source data table structure corresponding to the target database are obtained through the data configuration center.
In this embodiment, optionally, the big data collecting method of the present invention may automatically identify new service data, and add the new service data to the real-time collection, specifically: and acquiring the code of the new service, acquiring the configuration of a corresponding database, a related table structure and the like from a configuration center through the code of the new service, and connecting the configuration with the corresponding database to extract data.
The embodiment of the invention obtains target data and a source data table structure corresponding to a target database through a data configuration center, analyzes and processes a designated field of the target data, cleans and processes non-standard data of the target data to obtain target warehousing data, synchronizes a storage data table structure corresponding to the target storage database with the source data table structure to obtain a target storage data table structure, writes the target warehousing data and the target storage data table structure into a cache, and finally writes the target warehousing data in the cache and the target storage data table structure into the target storage database.
Example two
Referring to fig. 2, fig. 2 is a partial structural framework diagram of a real-time big data acquisition apparatus according to an embodiment of the present invention, and as can be obtained by combining fig. 2, the real-time big data acquisition apparatus 100 according to the present invention includes:
the data obtaining module 110 is configured to obtain target data and a source data table structure corresponding to a target database through a data configuration center, where the target database includes the target data.
The first processing module 120 is configured to perform first processing on the target data to obtain target warehousing data, and synchronize a storage data table structure corresponding to a target storage database with the source data table structure to obtain a target storage data table structure.
And the second processing module 130 is configured to perform second processing on the target warehousing data and the target storage data table structure, and then write the target warehousing data and the target storage data table structure into a target storage database.
The data configuration center is configured before the target data is acquired, and the data stored by the data configuration center comprises related parameter data, a server address and a database address which are involved in the process of acquiring the big data in real time.
In this embodiment, optionally, the performing a first process on the target data to obtain target warehousing data includes:
analyzing the designated field of the target data, and cleaning the non-standard data of the target data to obtain the target warehousing data.
In this embodiment, optionally, the writing the target warehousing data and the target storage data table structure into the target storage database after the second processing includes:
and writing the target warehousing data and the target storage data table structure into a cache, and writing the target warehousing data and the target storage data table structure which are written into the cache into the target storage database.
The real-time acquisition method of the big data, which is realized by the embodiment of the invention, comprises the steps of acquiring target data and a source data table structure corresponding to a target database through a data configuration center, analyzing and processing a specified field of the target data, cleaning and processing non-standard data of the target data to obtain target warehousing data, synchronizing a storage data table structure corresponding to a target storage database with the source data table structure to obtain a target storage data table structure, writing the target warehousing data and the target storage data table structure into a cache, and finally writing the target warehousing data in the cache and the target storage data table structure into the target storage database.
EXAMPLE III
Referring to fig. 3, a computer-readable storage medium 10 according to an embodiment of the present invention can be seen, where the computer-readable storage medium 10 includes: ROM/RAM, magnetic disk, optical disk, etc., on which a computer program 11 is stored, the computer program 11, when executed, implementing a real-time acquisition method of big data as described in the first embodiment. Since the real-time acquisition method of the big data has been described in detail in the first embodiment, the description is not repeated here.
The real-time acquisition method of the big data, which is realized by the embodiment of the invention, comprises the steps of acquiring target data and a source data table structure corresponding to a target database through a data configuration center, analyzing and processing a specified field of the target data, cleaning and processing non-standard data of the target data to obtain target warehousing data, synchronizing a storage data table structure corresponding to a target storage database with the source data table structure to obtain a target storage data table structure, writing the target warehousing data and the target storage data table structure into a cache, and finally writing the target warehousing data in the cache and the target storage data table structure into the target storage database.
Example four
Referring to fig. 4, a computer device 20 according to an embodiment of the present invention includes a processor 21, a memory 22, and a computer program 221 stored in the memory 22 and executable on the processor 21, wherein the processor 21 implements the real-time obtaining method of big data according to an embodiment when executing the computer program 221. Since the real-time acquisition method of the big data has been described in detail in the first embodiment, the description is not repeated here.
The real-time acquisition method of the big data, which is realized by the embodiment of the invention, comprises the steps of acquiring target data and a source data table structure corresponding to a target database through a data configuration center, analyzing and processing a specified field of the target data, cleaning and processing non-standard data of the target data to obtain target warehousing data, synchronizing a storage data table structure corresponding to a target storage database with the source data table structure to obtain a target storage data table structure, writing the target warehousing data and the target storage data table structure into a cache, and finally writing the target warehousing data in the cache and the target storage data table structure into the target storage database.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes performed by the present specification and drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A real-time big data acquisition method is characterized by comprising the following steps:
acquiring target data and a source data table structure corresponding to a target database through a data configuration center, wherein the target database comprises the target data;
performing first processing on the target data to obtain target warehousing data, and synchronizing a storage data table structure corresponding to a target storage database with the source data table structure to obtain a target storage data table structure;
performing second processing on the target warehousing data and the target storage data table structure and writing the target warehousing data and the target storage data table structure into a target storage database;
the data configuration center is configured before the target data is acquired, and the data stored by the data configuration center comprises related parameter data, a server address and a database address which are involved in the process of acquiring the big data in real time.
2. The method for acquiring big data in real time according to claim 1, wherein the acquiring, by the data configuration center, the target data and the source data table structure corresponding to the target database includes:
acquiring a target database address corresponding to the target data through a data configuration center;
and accessing the target database address to enter a target database, and acquiring the target data and the source data table structure corresponding to the target database from the target database.
3. The method for acquiring big data in real time according to claim 1, wherein the first processing of the target data to obtain target warehousing data comprises:
analyzing the designated field of the target data, and cleaning the non-standard data of the target data to obtain the target warehousing data.
4. The method according to claim 1, wherein the step of synchronizing the storage data table structure corresponding to the target storage database with the source data table structure to obtain the target storage data table structure comprises:
judging whether the structure of the storage data table is the same as that of the source data table;
and if not, updating the storage data table structure by taking the source data table structure as a standard table structure to obtain a target storage data table structure, otherwise, keeping the storage data table structure unchanged, wherein the target storage data table structure is the same as the source data table structure.
5. The method according to claim 4, wherein the determining whether the structure of the storage data table is the same as the structure of the source data table comprises:
obtaining a latest cache data table structure from a cache, wherein the latest cache data table structure is the same as the storage data table structure;
judging whether the structure of the storage data table is the same as that of the cache data table or not;
and if so, judging that the structure of the storage data table is the same as that of the source data table, otherwise, judging that the structure of the storage data table is different from that of the source data table.
6. The method for acquiring big data in real time according to claim 1, wherein the writing the target warehousing data and the target storage data table structure into a target storage database after the second processing comprises:
and writing the target warehousing data and the target storage data table structure into a cache, and writing the target warehousing data and the target storage data table structure which are written into the cache into the target storage database.
7. The method for acquiring big data in real time according to claim 1, wherein before the acquiring, by the data configuration center, the target data and the source data table structure corresponding to the target database, the method comprises:
acquiring a target data code corresponding to a new service every preset time, wherein the target data code is used for connecting the data configuration center;
and acquiring the target data and the source data table structure corresponding to the target database through the data configuration center according to the target data code.
8. A real-time big data acquisition device is characterized by comprising:
the data acquisition module is used for acquiring target data and a source data table structure corresponding to a target database through a data configuration center, wherein the target database comprises the target data;
the first processing module is used for carrying out first processing on the target data to obtain target warehousing data, and synchronizing a storage data table structure corresponding to a target storage database with the source data table structure to obtain a target storage data table structure;
the second processing module is used for writing the target warehousing data and the target storage data table structure into a target storage database after second processing;
the data configuration center is configured before the target data is acquired, and the data stored by the data configuration center comprises related parameter data, a server address and a database address which are involved in the process of acquiring the big data in real time.
9. A computer-readable storage medium, having a computer program stored thereon, wherein the computer program is executed to implement the real-time big data acquisition method of any one of claims 1 to 7.
10. A computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the real-time big data acquisition method according to any one of claims 1 to 7 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910933843.XA CN110704442A (en) | 2019-09-29 | 2019-09-29 | Real-time acquisition method and device for big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910933843.XA CN110704442A (en) | 2019-09-29 | 2019-09-29 | Real-time acquisition method and device for big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110704442A true CN110704442A (en) | 2020-01-17 |
Family
ID=69196557
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910933843.XA Pending CN110704442A (en) | 2019-09-29 | 2019-09-29 | Real-time acquisition method and device for big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110704442A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112905706A (en) * | 2021-03-19 | 2021-06-04 | 平安消费金融有限公司 | Database synchronization method and device, storage medium and computer equipment |
| CN113986927A (en) * | 2021-10-29 | 2022-01-28 | 中国联合网络通信集团有限公司 | Data storage method, device and equipment and storage medium |
| CN115022890A (en) * | 2022-06-02 | 2022-09-06 | 西安电子科技大学 | Method for generating resource cell coverage structure facing capacity coverage |
| CN116893987A (en) * | 2023-09-11 | 2023-10-17 | 归芯科技(深圳)有限公司 | Hardware acceleration method, hardware accelerator and hardware acceleration system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107908481A (en) * | 2017-10-17 | 2018-04-13 | 链家网（北京）科技有限公司 | A kind of method of data synchronization, device and system |
| CN109828889A (en) * | 2019-01-31 | 2019-05-31 | 平安科技（深圳）有限公司 | Method, apparatus, computer equipment and the storage medium in monitoring data library |
| CN110209730A (en) * | 2019-04-25 | 2019-09-06 | 深圳壹账通智能科技有限公司 | Change synchronous method, device, computer equipment and the computer storage medium of data |
-
2019
- 2019-09-29 CN CN201910933843.XA patent/CN110704442A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107908481A (en) * | 2017-10-17 | 2018-04-13 | 链家网（北京）科技有限公司 | A kind of method of data synchronization, device and system |
| CN109828889A (en) * | 2019-01-31 | 2019-05-31 | 平安科技（深圳）有限公司 | Method, apparatus, computer equipment and the storage medium in monitoring data library |
| CN110209730A (en) * | 2019-04-25 | 2019-09-06 | 深圳壹账通智能科技有限公司 | Change synchronous method, device, computer equipment and the computer storage medium of data |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112905706A (en) * | 2021-03-19 | 2021-06-04 | 平安消费金融有限公司 | Database synchronization method and device, storage medium and computer equipment |
| CN113986927A (en) * | 2021-10-29 | 2022-01-28 | 中国联合网络通信集团有限公司 | Data storage method, device and equipment and storage medium |
| CN115022890A (en) * | 2022-06-02 | 2022-09-06 | 西安电子科技大学 | Method for generating resource cell coverage structure facing capacity coverage |
| CN115022890B (en) * | 2022-06-02 | 2023-06-30 | 西安电子科技大学 | Method for generating resource cell coverage structure facing capacity coverage |
| CN116893987A (en) * | 2023-09-11 | 2023-10-17 | 归芯科技(深圳)有限公司 | Hardware acceleration method, hardware accelerator and hardware acceleration system |
| CN116893987B (en) * | 2023-09-11 | 2024-01-12 | 归芯科技(深圳)有限公司 | Hardware acceleration method, hardware accelerator and hardware acceleration system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11941017B2 (en) | Event driven extract, transform, load (ETL) processing | |
| CN108536761B (en) | Report data query method and server | |
| US10180946B2 (en) | Consistent execution of partial queries in hybrid DBMS | |
| US10216770B1 (en) | Scaling stateful clusters while maintaining access | |
| CN110704442A (en) | Real-time acquisition method and device for big data | |
| CN104899295B (en) | A kind of heterogeneous data source data relation analysis method | |
| CN110674154A (en) | Spark-based method for inserting, updating and deleting data in Hive | |
| US20140358844A1 (en) | Workflow controller compatibility | |
| CN111651519B (en) | Data synchronization method, data synchronization device, electronic equipment and storage medium | |
| CN105740295B (en) | A method and device for processing distributed data | |
| CN104111996A (en) | Health insurance outpatient clinic big data extraction system and method based on hadoop platform | |
| CN105512336A (en) | Method and device for mass data processing based on Hadoop | |
| CN112269802B (en) | Method and system for optimizing based on Clickhouse frequent censoring | |
| AU2015316450A1 (en) | Method for updating data table of KeyValue database and apparatus for updating table data | |
| CN115374102A (en) | Data processing method and system | |
| CN105608126A (en) | Method and apparatus for establishing secondary indexes for massive databases | |
| GB2534374A (en) | Distributed System with accelerator-created containers | |
| CN112527801A (en) | Data synchronization method and system between relational database and big data system | |
| Ding et al. | ComMapReduce: An improvement of MapReduce with lightweight communication mechanisms | |
| CN111143468B (en) | Multi-database data management method based on MPP distributed technology | |
| US11354304B1 (en) | Stored procedures for incremental updates to internal tables for materialized views | |
| CN108595552B (en) | Data cube publishing method and device, electronic equipment and storage medium | |
| CN114265896A (en) | Data synchronization method, system, device and medium for multiple data sources | |
| US11836125B1 (en) | Scalable database dependency monitoring and visualization system | |
| CN114625743A (en) | Data updating method and device for personnel master data and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200117 |