CN112860776A - Method and system for extracting and scheduling various data - Google Patents
Method and system for extracting and scheduling various data Download PDFInfo
- Publication number
- CN112860776A CN112860776A CN202110076282.3A CN202110076282A CN112860776A CN 112860776 A CN112860776 A CN 112860776A CN 202110076282 A CN202110076282 A CN 202110076282A CN 112860776 A CN112860776 A CN 112860776A
- Authority
- CN
- China
- Prior art keywords
- data extraction
- information
- node
- storage process
- workflow
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a method and a system for extracting and scheduling multiple data, which comprises the steps of configuring basic information comprising different data extraction nodes, wherein each data extraction node supports at least one of a JAVA interface, a storage process and an information workflow data extraction mode, and each data extraction node can support all the data extraction modes; determining the number of selected data extraction nodes, and forming a logic of data extraction according to the extraction flow of each data extraction node; according to the logic of data extraction, each data extraction node operates, and data extraction is carried out by utilizing different data extraction modes; and judging whether the data extraction is successful, and if the data extraction is failed, repeatedly executing the setting times. The invention solves the problem that the traditional single data extraction mode can not complete the task.
Description
Technical Field
The invention belongs to the technical field of data extraction and scheduling, and particularly relates to a method and a system for extracting and scheduling various data.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
The ETL logic is realized by combining an Informatica and a database storage process at present, the scheduling in the Informatica is used for managing the overall task at present, the Informatica can flexibly schedule and manage the logic of the Informatica, the storage process cannot be flexibly scheduled and monitored, and due to the limitation of the product of the Informatica, the abnormal logs in the extraction process cannot be intuitively displayed, so that the monitoring is difficult. And there is also part of the scheduling logic in the JAVA program that is not temporarily added to the informational schedule.
Disclosure of Invention
The invention provides a method and a system for extracting and scheduling various data, which can customize data extraction logic according to actual conditions, and can be used in a serial and parallel combination mode, thereby supporting the simultaneous use of various data extraction modes.
According to some embodiments, the invention adopts the following technical scheme:
a multi-data extraction scheduling method comprises the following steps:
configuring basic information containing different data extraction nodes, wherein each data extraction node supports at least one of a JAVA interface, a storage process and an Informatica workflow data extraction mode, and each data extraction node can support all the data extraction modes;
determining the number of selected data extraction nodes, and forming a logic of data extraction according to the extraction flow of each data extraction node;
according to the logic of data extraction, each data extraction node operates, and data extraction is carried out by utilizing different data extraction modes;
and judging whether the data extraction is successful, and if the data extraction is failed, repeatedly executing the setting times.
As an alternative embodiment, the basic information of different data extraction nodes includes:
the basic information of the JAVA interface comprises an interface address and interface parameters;
the basic information of the stored process comprises a database address, a port, an instance name, a user name and a password;
the Informatica workflow contains database information for the IP address, computer name, integrated service name, port, username, password, and metadata of the Informatica service.
As an alternative embodiment, the logic for data extraction includes at least one of serial, parallel, and branch execution. The branch execution means that whether the following node is executed or not is determined according to the result of the previous node.
As an alternative embodiment, the extraction process of the JAVA interface includes:
and judging whether the interface is available or not according to whether the method list can be returned or not by remotely calling the Java interface address.
Maintaining basic information of a target interface method into an operating basic element, and maintaining a judgment basis and a subsequent processing mode together, wherein the set of the group of information forms a JAVA interface node;
calling a Java interface, receiving a return value, and judging the return result according to a previously maintained result judgment standard;
if the data extraction needs to be suspended or interrupted in the calling process, calling a data extraction control interface, inputting the data extraction interface needing to be operated and operation type information in a parameter input mode, finding the interface by the data extraction control interface, interrupting, suspending or continuing the interface, and controlling the data extraction process.
As an alternative embodiment, the data extraction process of the storage process includes:
driving through the database according to the maintained basic information of the database address, the port, the instance name, the user name and the password, and judging the connectivity of the database;
inquiring the information of the database storage process, selecting the required storage process from the information, maintaining the parameter entry and parameter exit information of the storage process, including parameter names, parameter types and parameter positions, and collecting the information to form a storage process node;
and calling a storage process, driving the calling storage process through a database, storing detailed information in the execution process of the storage process into a special log table, finally returning an execution result, and judging the returned result according to a judgment standard.
As a further limitation, in the process of the storage process, the process may be controlled, and according to the name of the storage process, the process of the storage process is found, and a stop command is sent to the data to stop the storage process;
and checking the storage process operation log table through the database drive, and acquiring detailed information of the storage process operation, including the number of data lines to be extracted, the number of extracted lines, error information and error position.
As an alternative embodiment, the process of the informational workflow includes:
judging whether the information service is started or not by communicating port numbers of the information service;
acquiring a workflow list by inquiring information metadata information, selecting a required workflow, and collecting the information to form an information workflow node;
connecting a remote information service, sending an information command through a network, starting an information workflow, acquiring return information after the workflow is executed, acquiring key information in result information by using word segmentation analysis, and judging success and failure of data extraction results according to the key information;
and sending stop, pause and continue commands to the Informatic server according to the workflow name to control the Informatic workflow extraction process.
As a further limitation, in the data extraction process, the execution logs of all the sessions in the workflow are obtained by inquiring the information metadata information so as to realize error positioning and problem solving.
As an alternative implementation manner, in the operation process, the operation result of each data extraction node is judged, if the operation result is correct, the operation is continued according to the set logic, and if the operation result is wrong, the execution of the subsequent data extraction nodes is suspended according to the set logic.
As an alternative embodiment, the judging of the operation result of each data extraction node includes two modes: (1) according to the return value, the return value is compared with result judgment information maintained at the generating node, so that success and failure of operation are judged; (2) and according to the running time, the running time is preset when the nodes are generated, if the running time is overtime, the running is judged to be failed, otherwise, the logic for extracting the data is blocked, and other data extraction is influenced.
As an alternative embodiment, an exception is found during the extraction process, and execution is resumed by selecting to pause, stop or ignore the exception.
A multiple data extraction scheduling system, comprising:
the parameter setting module is configured to configure basic information containing different data extraction nodes, each data extraction node supports at least one of a JAVA interface, a storage process and an Informatica workflow data extraction mode, and each data extraction node can support all the data extraction modes;
the logic configuration module is configured to determine the number of the selected data extraction nodes and form data extraction logic according to the extraction flow of each data extraction node;
the execution module is configured to operate each data extraction node according to the logic of data extraction, and perform data extraction scheduling by using different data extraction modes;
and the judging module is configured to judge whether the data extraction is successful, and if the data extraction is failed, the data extraction is repeatedly executed for a set time.
A computer readable storage medium having stored therein a plurality of instructions adapted to be loaded by a processor of a terminal device and to perform the steps of said one or more data extraction scheduling methods.
A terminal device comprising a processor and a computer readable storage medium, the processor being configured to implement instructions; the computer readable storage medium is used for storing a plurality of instructions which are suitable for being loaded by a processor and executing the steps of the data extraction scheduling method.
Compared with the prior art, the invention has the beneficial effects that:
the invention solves the problem that the traditional extraction data can only be in parallel or serial extraction logic, and can form various data extraction logics by making a flow chart according to the actual condition of a user.
The invention realizes the organic combination of a plurality of data extraction modes, and the data extraction modes are mutually matched under the condition of certain logic, thereby solving the problem that the traditional single data extraction mode can not complete the task.
The invention overcomes the defect that the data correctness can only be checked after the data extraction process is finished in the prior art, and can monitor the data extraction process at any time according to the actual condition and perform intervention control on the data extraction process according to the monitoring result.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the invention and together with the description serve to explain the invention and not to limit the invention.
FIG. 1 is a schematic flow chart of the present embodiment;
fig. 2 is a flowchart of specific implementation of the node according to this embodiment.
The specific implementation mode is as follows:
the invention is further described with reference to the following figures and examples.
It is to be understood that the following detailed description is exemplary and is intended to provide further explanation of the invention as claimed. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
As shown in fig. 1, a method for extracting and scheduling multiple data includes the following steps:
1. and after the information maintenance of different data extraction modes is completed. The data extraction process needs to be arranged according to actual conditions.
2. When the flow of data extraction is executed, it is judged whether the data is executed in parallel or in series, and if the data is executed in parallel, the data is executed in multiple threads, and after all the data is executed in parallel, the next step is executed.
3. When data extraction is performed, the data extraction mode is judged and selected in different data extraction logics.
4. If the data extraction fails, the extraction failure due to the network failure is excluded after repeated execution several times (3 times in the present embodiment).
5. If the node has branches, the next node is judged to be executed according to the execution result of the node.
Specifically, the basic information configuring the data extraction mode is divided into three categories: the method comprises the following steps of 1, JAVA interface, 2, storage process, 3, information workflow, and the basic information of the JAVA interface comprises interface address, interface parameters and the like. The basic information of the stored process comprises a database address, a port, an instance name, a user name, a password and the like. The Informatica workflow contains the IP address of the Informatica service, the computer name, the integrated service name, the port, the username, the password and the database information of the metadata.
The calling of the Java interface comprises the following steps:
verifying the availability of the interface. And judging whether the interface is available or not according to whether the method list can be returned or not by remotely calling the Java interface address.
And secondly, maintaining information such as interface methods, input parameters, return values, judgment bases and the like. And maintaining the basic information of the target interface method as an operating basic element, and maintaining the judgment basis and the subsequent processing mode together, wherein the set of the information forms a JAVA interface node.
And calling the interface. And calling a Java interface, receiving a return value, and judging the return result according to the previously maintained result judgment standard.
And fourthly, controlling the process of the Java interface. The data extraction is suspended or interrupted if necessary during the invocation. The data extraction control interface can be called, and information such as the data extraction interface to be operated and the operation type (interruption, pause and continuation) is transmitted in a parameter entering mode, and the data extraction control interface can find the interface and interrupt, pause or continue the interface, so that the purpose of controlling the data extraction process is achieved.
The calling of the storage process comprises the following steps:
judging the connectivity of the database. And driving through the database according to basic information such as maintained database addresses, ports, instance names, user names, passwords and the like. Whether the SQL sentences are communicated or not is judged by executing a result of obtaining the SQL sentences at the current time.
Selecting a target storage process. Inquiring the information of the database storage process, selecting the required storage process from the information, and maintaining the parameter entering and parameter leaving information of the storage process, including parameter names, parameter types, parameter positions and the like, wherein a group of information is integrated to form a storage process node.
And calling a storage process, calling the storage process through the database drive, storing detailed information in the execution process of the storage process into a special log table, and finally returning the execution result. And judging a return result according to the judgment standard.
And fourthly, controlling the process of the storage process. Firstly, the process of the storage process is found according to the name of the storage process, and a stop command is sent to data to stop the storage process.
Checking the log list of the running of the storage process through the database driver, and acquiring the detail information of the running of the storage process, including the number of rows of data to be extracted, the number of rows already extracted, error information and error position.
Informatic workflow calling procedure
Verifying the Informatic service connectivity. And judging whether the information service is started or not by connecting port numbers of the information service.
Selecting a target information workflow. The information is inquired to obtain a workflow list, the needed workflow is selected, and the information set forms an information workflow node.
Invoking the Informatic workflow. Connecting a remote information service, sending an information command through a network, starting an information workflow, acquiring return information after the workflow is executed, then using word segmentation analysis to acquire key information in result information, and judging success and failure of data extraction results according to the key information.
Fourthly, controlling the progress of the information workflow. And sending commands of stopping, suspending and continuing to the Informatic server according to the workflow name. The Informatic workflow extraction process is controlled.
Inquiring the detail of the log. By querying the information metadata information, the execution logs of all sessions in the workflow are obtained, so as to facilitate error positioning and problem solving.
As shown in fig. 2, the edit data extraction logic may use one data extraction method alone or may organically combine three data extraction methods. Three different data extraction nodes (JAVA, storage process and information workflow) can make extraction flows according to the needs of the nodes to form various data extraction logics, such as serial, parallel or branch (the execution logic of the following node is determined according to the result of the previous node).
In this embodiment, there are two judgment ways for judging the node operation result: 1. and comparing the return value with result judgment information maintained at the generating node according to the return value so as to judge success and failure of operation. 2. And according to the running time, the running time is preset when the nodes are generated, if the running time is overtime, the running is judged to be failed, otherwise, the logic for extracting the data is blocked, and other data extraction is influenced.
When an abnormality is found in the extraction process, operations such as pause, stop, continuation and the like can be performed. The information log is optimized and sorted, the information log can be quickly positioned and is simple and clear, and the abnormal information is sent to a relevant responsible person through a mail to give an early warning in time.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, it is not intended to limit the scope of the present invention, and it should be understood by those skilled in the art that various modifications and variations can be made without inventive efforts by those skilled in the art based on the technical solution of the present invention.
Claims (10)
1. A multi-data extraction scheduling method is characterized in that: the method comprises the following steps:
configuring basic information containing different data extraction nodes, wherein each data extraction node supports at least one of a JAVA interface, a storage process and an Informatica workflow data extraction mode, and each data extraction node can support all the data extraction modes;
determining the number of selected data extraction nodes, and forming a logic of data extraction according to the extraction flow of each data extraction node;
according to the logic of data extraction, each data extraction node operates, and data extraction is carried out by utilizing different data extraction modes;
and judging whether the data extraction is successful, and if the data extraction is failed, repeatedly executing the setting times.
2. The method as claimed in claim 1, wherein the method comprises: the basic information of different data extraction nodes comprises:
the basic information of the JAVA interface comprises an interface address and interface parameters;
the basic information of the stored process comprises a database address, a port, an instance name, a user name and a password;
the Informatica workflow contains database information for the IP address, computer name, integrated service name, port, username, password, and metadata of the Informatica service.
3. The method as claimed in claim 1, wherein the method comprises: logic for data extraction, including at least one of serial, parallel, and branch execution.
4. The method as claimed in claim 1, wherein the method comprises: the extraction process of the JAVA interface comprises the following steps:
judging whether the interface is available or not according to whether a method list can be returned or not by remotely calling a Java interface address;
maintaining basic information of a target interface method into an operating basic element, and maintaining a judgment basis and a subsequent processing mode together, wherein the set of the group of information forms a JAVA interface node;
calling a Java interface, receiving a return value, and judging the return result according to a previously maintained result judgment standard;
if the data extraction needs to be suspended or interrupted in the calling process, calling a data extraction control interface, inputting the data extraction interface needing to be operated and operation type information in a parameter input mode, finding the interface by the data extraction control interface, interrupting, suspending or continuing the interface, and controlling the data extraction process.
5. The method as claimed in claim 1, wherein the method comprises: the data extraction process of the storage process comprises the following steps:
driving through the database according to the maintained basic information of the database address, the port, the instance name, the user name and the password, and judging the connectivity of the database;
inquiring the information of the database storage process, selecting the required storage process from the information, maintaining the parameter entry and parameter exit information of the storage process, including parameter names, parameter types and parameter positions, and collecting the information to form a storage process node;
and calling a storage process, driving the calling storage process through a database, storing detailed information in the execution process of the storage process into a special log table, finally returning an execution result, and judging the returned result according to a judgment standard.
6. The method as claimed in claim 5, wherein the scheduling method comprises: in the process of the storage process, the process can be controlled, the process of the storage process is searched according to the name of the storage process, and a stop command is sent to data to stop the storage process;
and checking the storage process operation log table through the database drive, and acquiring detailed information of the storage process operation, including the number of data lines to be extracted, the number of extracted lines, error information and error position.
7. The method as claimed in claim 1, wherein the method comprises: the process of the informational workflow includes:
judging whether the information service is started or not by communicating port numbers of the information service;
acquiring a workflow list by inquiring information metadata information, selecting a required workflow, and collecting the information to form an information workflow node;
connecting a remote information service, sending an information command through a network, starting an information workflow, acquiring return information after the workflow is executed, acquiring key information in result information by using word segmentation analysis, and judging success and failure of data extraction results according to the key information;
sending stop, pause and continue commands to the Informatic server according to the workflow name, and controlling the Informatic workflow extraction process;
in the data extraction process, the execution logs of all the sessions in the workflow are acquired by inquiring the information metadata information so as to realize error positioning and problem solving.
8. The method as claimed in claim 1, wherein the method comprises: and in the operation process, judging the operation result of each data extraction node, if the operation result is correct, continuing to operate according to the set logic, and if the operation result is wrong, suspending the execution of the subsequent data extraction nodes according to the set logic.
9. The method as claimed in claim 1, wherein the method comprises: the operation result of each data extraction node is judged in two modes: (1) according to the return value, the return value is compared with result judgment information maintained at the generating node, so that success and failure of operation are judged; (2) and according to the running time, the running time is preset when the nodes are generated, if the running time is overtime, the running is judged to be failed, otherwise, the logic for extracting the data is blocked, and other data extraction is influenced.
10. A multi-data extraction scheduling system is characterized in that: the method comprises the following steps:
the parameter setting module is configured to configure basic information containing different data extraction nodes, each data extraction node supports at least one of a JAVA interface, a storage process and an Informatica workflow data extraction mode, and each data extraction node can support all the data extraction modes;
the logic configuration module is configured to determine the number of the selected data extraction nodes and form data extraction logic according to the extraction flow of each data extraction node;
the execution module is configured to operate each data extraction node according to the logic of data extraction, and perform data extraction scheduling by using different data extraction modes;
and the judging module is configured to judge whether the data extraction is successful, and if the data extraction is failed, the data extraction is repeatedly executed for a set time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110076282.3A CN112860776B (en) | 2021-01-20 | 2021-01-20 | Method and system for extracting and scheduling various data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110076282.3A CN112860776B (en) | 2021-01-20 | 2021-01-20 | Method and system for extracting and scheduling various data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112860776A true CN112860776A (en) | 2021-05-28 |
| CN112860776B CN112860776B (en) | 2022-12-06 |
Family
ID=76007752
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110076282.3A Active CN112860776B (en) | 2021-01-20 | 2021-01-20 | Method and system for extracting and scheduling various data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112860776B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115858310A (en) * | 2023-03-01 | 2023-03-28 | 美云智数科技有限公司 | Abnormal task identification method and device, computer equipment and storage medium |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722355A (en) * | 2012-06-04 | 2012-10-10 | 南京中兴软创科技股份有限公司 | Workflow mechanism-based concurrent ETL (Extract, Transform and Load) conversion method |
| CN103412897A (en) * | 2013-07-25 | 2013-11-27 | 中国科学院软件研究所 | Parallel data processing method based on distributed structure |
| US20170060974A1 (en) * | 2015-08-31 | 2017-03-02 | Jade Global, Inc. | Automated conversion tool for facilitating migration between data integration products |
| CN107733986A (en) * | 2017-09-15 | 2018-02-23 | 中国南方电网有限责任公司 | Support the protection of integrated deployment and monitoring operation big data support platform |
| CN108446335A (en) * | 2018-02-27 | 2018-08-24 | 广州港数据科技有限公司 | Heterogeneous system data pick-up based on database and unified outbound data exchange method |
| CN108846076A (en) * | 2018-06-08 | 2018-11-20 | 山大地纬软件股份有限公司 | The massive multi-source ETL process method and system of supporting interface adaptation |
| CN108984549A (en) * | 2017-05-31 | 2018-12-11 | 北京京东尚科信息技术有限公司 | Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library |
| CN109271435A (en) * | 2018-09-14 | 2019-01-25 | 南威软件股份有限公司 | A kind of data pick-up method and system for supporting breakpoint transmission |
| US20190197767A1 (en) * | 2017-12-26 | 2019-06-27 | Htc Corporation | Surface extrction method, apparatus, and non-transitory computer readable storage medium thereof |
| CN110032594A (en) * | 2019-03-21 | 2019-07-19 | 厦门市美亚柏科信息股份有限公司 | The data pick-up method, apparatus and storage medium of the Various database of customizable |
| CN110362562A (en) * | 2019-07-16 | 2019-10-22 | 中国工商银行股份有限公司 | The method and system of big data sample drawn data |
| CN110555074A (en) * | 2019-09-11 | 2019-12-10 | 山东浪潮商用系统有限公司 | method for extracting and processing data at fixed time, terminal and computer readable storage medium |
| CN111427943A (en) * | 2020-03-27 | 2020-07-17 | 北京明略软件系统有限公司 | Task management method and device in ET L system |
-
2021
- 2021-01-20 CN CN202110076282.3A patent/CN112860776B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722355A (en) * | 2012-06-04 | 2012-10-10 | 南京中兴软创科技股份有限公司 | Workflow mechanism-based concurrent ETL (Extract, Transform and Load) conversion method |
| CN103412897A (en) * | 2013-07-25 | 2013-11-27 | 中国科学院软件研究所 | Parallel data processing method based on distributed structure |
| US20170060974A1 (en) * | 2015-08-31 | 2017-03-02 | Jade Global, Inc. | Automated conversion tool for facilitating migration between data integration products |
| CN108984549A (en) * | 2017-05-31 | 2018-12-11 | 北京京东尚科信息技术有限公司 | Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library |
| CN107733986A (en) * | 2017-09-15 | 2018-02-23 | 中国南方电网有限责任公司 | Support the protection of integrated deployment and monitoring operation big data support platform |
| US20190197767A1 (en) * | 2017-12-26 | 2019-06-27 | Htc Corporation | Surface extrction method, apparatus, and non-transitory computer readable storage medium thereof |
| CN108446335A (en) * | 2018-02-27 | 2018-08-24 | 广州港数据科技有限公司 | Heterogeneous system data pick-up based on database and unified outbound data exchange method |
| CN108846076A (en) * | 2018-06-08 | 2018-11-20 | 山大地纬软件股份有限公司 | The massive multi-source ETL process method and system of supporting interface adaptation |
| CN109271435A (en) * | 2018-09-14 | 2019-01-25 | 南威软件股份有限公司 | A kind of data pick-up method and system for supporting breakpoint transmission |
| CN110032594A (en) * | 2019-03-21 | 2019-07-19 | 厦门市美亚柏科信息股份有限公司 | The data pick-up method, apparatus and storage medium of the Various database of customizable |
| CN110362562A (en) * | 2019-07-16 | 2019-10-22 | 中国工商银行股份有限公司 | The method and system of big data sample drawn data |
| CN110555074A (en) * | 2019-09-11 | 2019-12-10 | 山东浪潮商用系统有限公司 | method for extracting and processing data at fixed time, terminal and computer readable storage medium |
| CN111427943A (en) * | 2020-03-27 | 2020-07-17 | 北京明略软件系统有限公司 | Task management method and device in ET L system |
Non-Patent Citations (1)
| Title |
|---|
| 刘如九等: "一种通用的多数据库间数据抽取方法及应用", 《北京交通大学学报》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115858310A (en) * | 2023-03-01 | 2023-03-28 | 美云智数科技有限公司 | Abnormal task identification method and device, computer equipment and storage medium |
| CN115858310B (en) * | 2023-03-01 | 2023-07-21 | 美云智数科技有限公司 | Abnormal task identification method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112860776B (en) | 2022-12-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111209131B (en) | Method and system for determining faults of heterogeneous system based on machine learning | |
| CN101477543B (en) | System and method for automating ETL application | |
| JP3370276B2 (en) | Method of managing semiconductor manufacturing equipment management system | |
| CN108038054B (en) | Automatic testing method and device and computer readable storage medium | |
| CN104425001A (en) | System for simultaneously burning multiple switches, method thereof and corresponding switches | |
| WO2014188502A1 (en) | Management system, management program, and management method | |
| CN112631919B (en) | Contrast test method, device, computer equipment and storage medium | |
| CN112115029B (en) | Performance test method and device, computer equipment and computer readable storage medium | |
| CN110275992B (en) | Emergency processing method, device, server and computer readable storage medium | |
| CN110659255A (en) | Data migration method, device and system and storage medium | |
| CN111462811A (en) | Automatic testing method and device, storage medium and electronic equipment | |
| CN109828886B (en) | CI/CD monitoring method and system under container cloud environment | |
| CN110941519A (en) | Chip testing method and device of network equipment | |
| CN112860776B (en) | Method and system for extracting and scheduling various data | |
| CN112799782B (en) | Model generation system, method, electronic device and storage medium | |
| CN111798083B (en) | Production scheduling monitoring system, method and storage medium | |
| CN115712623B (en) | Batch data fault-tolerant collection method based on captured metadata change | |
| CN106384283A (en) | Internet plus based service bus structure and service bus system | |
| CN114238474A (en) | Data processing method, device and equipment based on drainage system and storage medium | |
| CN107577802A (en) | A kind of data base management method and device | |
| KR101566355B1 (en) | Master pattern generation method and apparatus for checking normal operation of plc based manufacturing system | |
| JP2007140997A (en) | Apparatus for diagnosing fault | |
| CN114186219A (en) | Heterogeneous system integration method and system for password cracking | |
| US20220242021A1 (en) | Method for providing an operating system of a machine controller | |
| CN110297642A (en) | A kind of system and method for the concurrent programming in multiple terminals, configuration and test |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 12 / F, building 1, Aosheng building, 1166 Xinluo street, hi tech Zone, Jinan City, Shandong Province Applicant after: Zhongyang Health Technology Group Co.,Ltd. Address before: 12 / F, building 1, Aosheng building, 1166 Xinluo street, hi tech Zone, Jinan City, Shandong Province Applicant before: SHANDONG MSUNHEALTH TECHNOLOGY GROUP Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |