CN116303765A

CN116303765A - Data copying method, device, electronic equipment and storage medium

Info

Publication number: CN116303765A
Application number: CN202310124924.1A
Authority: CN
Inventors: 赵立国; 林晖; 李尚哲
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2023-02-07
Filing date: 2023-02-07
Publication date: 2023-06-23

Abstract

The present disclosure provides a data replication method, apparatus, electronic device, and storage medium, which may be applied to the big data technical field, the financial field, or other fields. The method comprises the following steps: creating a data playback process in response to triggering the data replication event; carrying out data copying consensus with the host side equipment by using a parameter interaction table of the detection tool and the host side equipment to obtain a consensus result; loading a data playback script into a data playback process if it is determined that the consensus result indicates consensus is achieved; running a data playback process so as to pull a target message from the message queue, and playing back target data obtained by analyzing the target message in a database of platform end equipment; the host device is configured to load the log analysis script into a log analysis process, and run the log analysis process so as to analyze the database log of the host device to obtain target data, and send a target message obtained by packaging the target data to the message queue.

Description

Data copying method, device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of big data technology and the field of finance, and more particularly, to a data copying method, apparatus, electronic device, and storage medium.

Background

In order to meet the requirements of service innovation development, meet the requirements of continuous increase of service volume, service continuity, processing performance and the like, relieve the host resource pressure caused by the increase of the service volume, an application architecture combining a host with an open platform is constructed in the related technology, and data copying from the host to the platform cannot be avoided under the application architecture.

In the related art, the data copying from the host computer to the platform is generally completed by manual operation of a technician, but with the rapid increase of the number of platform servers, the workload of manual operation of the technician is increased, so that the accuracy of data copying cannot be ensured.

Disclosure of Invention

In view of this, the present disclosure provides a data replication method, apparatus, electronic device, readable storage medium, and computer program product.

One aspect of the present disclosure provides a data replication method, including: creating a data playback process in response to triggering the data replication event; carrying out data copying consensus with the host equipment by using a parameter interaction table of the detection tool and the host equipment to obtain a consensus result; loading a data playback script into the data playback process if it is determined that the consensus result indicates consensus is reached; the data playback process is operated so as to draw the target message from the message queue, and the target data obtained by analyzing the target message is played back in the database of the platform end equipment; the host device is configured to load a log analysis script into a log analysis process, and run the log analysis process so as to analyze a database log of the host device to obtain the target data, and send the target message obtained by encapsulating the target data to the message queue.

According to an embodiment of the present disclosure, the creating a data playback process in response to triggering a data copy event includes: stopping the current data playback process in response to triggering the data copying event; importing a table copy list of data to be copied from host equipment; and creating the data playback process based on the table copy manifest.

According to an embodiment of the present disclosure, the data playback process is executed to pull a target message from a message queue, and play back target data obtained by parsing the target message in a database of the platform end device, including: running the data playback process, and sequentially acquiring messages from the message queue; screening the message by using the table copy list to obtain a screening result; analyzing the target message to obtain the target data when the screening result indicates that the message is the target message; and playing back the target data in the database of the platform end device by utilizing the data playback script.

According to an embodiment of the present disclosure, the above method further includes: and returning the message to the message queue when the screening result indicates that the message is not the target message.

According to an embodiment of the present disclosure, the above method further includes: determining the time information of the data to be copied from a task log; transmitting the time information to the host device; wherein the host-side device is configured to update the parameter value of the first field in the parameter interaction table based on the time information.

According to an embodiment of the present disclosure, the above-described host-side device is configured to: loading the log analysis script into the log analysis process; running the log analysis process to analyze the database log of the host equipment based on the time information to obtain the data to be copied; and encapsulating the data to be copied into a plurality of messages, and sequentially sending the messages to the message queue.

According to an embodiment of the present disclosure, the foregoing parameter interaction table using a detection tool and a host device performs data replication consensus with the host device to obtain a consensus result, including: under the condition that the data playback process is successfully established, updating the parameter value of the second field of the parameter interaction table to a first preset value, wherein the host side device is configured to establish the log analysis process and update the parameter value of the third field of the parameter interaction table to the second preset value under the condition that the parameter value of the second field of the parameter interaction table obtained by query is the first preset value; when the detection tool detects that the parameter value of the third field of the parameter interaction table is the second preset value, updating the parameter value of the fourth field of the parameter interaction table to the third preset value, wherein the host device is configured to configure the log analysis process based on the parameter value of the first field of the parameter interaction table and update the parameter value of the fifth field of the parameter interaction table to the fourth preset value when the parameter value of the fourth field of the parameter interaction table is the third preset value; and obtaining the consensus result for achieving the data replication consensus with the host device when the detection tool detects that the parameter value of the fifth field of the parameter interaction table is the fourth preset value.

According to an embodiment of the present disclosure, the above method further includes: under the condition that the data playback process is in an operation state, updating a parameter value of a sixth field of the parameter interaction table to a fifth preset value; the host device is configured to load the log analysis script into the log analysis process and run the log analysis process when the parameter value of the sixth field of the parameter interaction table obtained by query is the fifth preset value.

According to an embodiment of the present disclosure, the data replication event includes that a parameter value of a seventh field of the parameter interaction table is a sixth preset value, and the parameter value of the seventh field of the parameter interaction table includes that the parameter interaction table is detected by using the detection tool; the host device is configured to stop the current log parsing process in response to the trigger table access event, and update the parameter value of the seventh field to the sixth preset value.

According to an embodiment of the present disclosure, the platform end device includes a platform server; the method further comprises the following steps: responding to a trigger server online event, and determining a target platform server to be accessed; transmitting the detection tool and the data playback script to the target platform server through a server management system; and sending server online information to the host side device, wherein the host side device is configured to establish a parameter interaction table associated with the target platform server based on the server online information.

Another aspect of the present disclosure provides a data copying apparatus, including: the creation module is used for responding to the triggering data copying event and creating a data playback process; the consensus module is used for carrying out data copying consensus with the host equipment by utilizing the detection tool and the parameter interaction table of the host equipment to obtain a consensus result; the loading module is used for loading the data playback script into the data playback process under the condition that the consensus result is determined to indicate that consensus is achieved; the running module is used for running the data playback process so as to pull the target message from the message queue and play back the target data obtained by analyzing the target message in the database of the platform end equipment; the host device is configured to load a log analysis script into a log analysis process, and run the log analysis process so as to analyze a database log of the host device to obtain the target data, and send the target message obtained by encapsulating the target data to the message queue.

Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more instructions that, when executed by the one or more processors, cause the one or more processors to implement the method as described above.

Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are configured to implement a method as described above.

Another aspect of the present disclosure provides a computer program product comprising computer executable instructions which, when executed, are adapted to implement the method as described above.

According to the embodiment of the disclosure, when implementing data copying from the host side device to the platform side device, the host side device and the platform side device can be subjected to consensus by using the parameter interaction table, and after the consensus is achieved, the platform side device can run a data playback process, the host side device can run a log analysis process, and the data copying from the host side device to the platform side device is completed by means of the message queue. By the technical means, the manual operation of the technicians can be replaced by the consensus based on the parameter interaction table, so that the technical problem of low accuracy of data copying caused by large manual operation of the technicians in the related technology is at least partially solved, and the efficiency and accuracy of data copying are effectively improved.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments thereof with reference to the accompanying drawings in which:

fig. 1 schematically illustrates an exemplary system architecture to which data replication methods and apparatuses may be applied according to embodiments of the present disclosure.

Fig. 2 schematically illustrates a flow chart of a data replication method according to an embodiment of the present disclosure.

Fig. 3 schematically illustrates a flow chart of a data replication consensus method according to an embodiment of the present disclosure.

Fig. 4 schematically shows a timing diagram of a data replication method according to an embodiment of the present disclosure.

Fig. 5 schematically illustrates a block diagram of a data replication apparatus according to an embodiment of the present disclosure.

Fig. 6 schematically illustrates a block diagram of an electronic device adapted to implement a data replication method according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a formulation similar to at least one of "A, B or C, etc." is used, in general such a formulation should be interpreted in accordance with the ordinary understanding of one skilled in the art (e.g. "a system with at least one of A, B or C" would include but not be limited to systems with a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

In the technical scheme of the disclosure, the related data (such as including but not limited to personal information of a user) are collected, stored, used, processed, transmitted, provided, disclosed, applied and the like, all conform to the regulations of related laws and regulations, necessary security measures are adopted, and the public welcome is not violated.

In order to meet the requirements of service innovation development, meet the requirements of continuous increase of service volume, service continuity, processing performance and the like, relieve the host resource pressure caused by the increase of the service volume, an application architecture combining a host with an open platform is constructed in the related technology, and data copying from the host to the platform cannot be avoided under the application architecture. With more and more application data platforms, platform servers are rapidly growing, so that the manual operation workload of technicians is huge, and the accuracy of data copying cannot be ensured.

In view of this, embodiments of the present disclosure provide a data replication method, apparatus, electronic device, storage medium, and computer program product to implement interactive automatic commissioning between a host and a platform. Specifically, the data replication method is applied to platform end equipment, and comprises the following steps: creating a data playback process in response to triggering the data replication event; carrying out data copying consensus with the host side equipment by using a parameter interaction table of the detection tool and the host side equipment to obtain a consensus result; loading a data playback script into a data playback process if it is determined that the consensus result indicates consensus is achieved; running a data playback process so as to pull the target message from the message queue, and playing back target data obtained by analyzing the target message in a database of the platform end equipment; the host device is configured to load a log analysis script into a log analysis process, and run the log analysis process so as to analyze a database log of the host device to obtain target data, and send a target message obtained by packaging the target data to the message queue.

It should be noted that, the data replication method and apparatus determined by the embodiments of the present disclosure may be used in the big data technical field or the financial field, and may also be used in any field other than the big data technical field and the financial field, and the application field of the data replication method and apparatus determined by the embodiments of the present disclosure is not limited.

Fig. 1 schematically illustrates an exemplary system architecture to which data replication methods and apparatuses may be applied according to embodiments of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which embodiments of the present disclosure may be applied to assist those skilled in the art in understanding the technical content of the present disclosure, but does not mean that embodiments of the present disclosure may not be used in other devices, systems, environments, or scenarios.

As shown in fig. 1, a system architecture 100 according to this embodiment may include a host-side device 101, a network 102, and platform-

side devices

103, 104, 105.

The host device 101 may be a server cluster formed by various servers, and the various servers may include an entity server provided at a local end, or may include a cloud server provided at a cloud end, which is not limited herein.

The network 102 is the medium used to provide communication links between the host-side device 101 and the platform-

side devices

103, 104, 105. Network 102 may include various connection types, such as wired and/or wireless communication links, and the like.

The

platform end devices

103, 104, 105 may include various electronic devices such as smartphones, tablets, laptop portable computers, desktop computers, and the like. Alternatively, the

platform end devices

103, 104, 105 may be various servers.

It should be noted that, the data replication method provided by the embodiments of the present disclosure may be generally performed by the

platform end devices

103, 104, 105. Accordingly, the data replication apparatus provided by the embodiments of the present disclosure may be generally disposed in the

platform end devices

103, 104, 105.

It should be understood that the number of host-side devices, networks, and platform-side devices in fig. 1 are merely illustrative. There may be any number of host-side devices, networks, and platform-side devices, as desired for implementation.

As shown in fig. 2, the method includes operations S201 to S204.

In operation S201, a data playback process is created in response to triggering a data copy event.

In operation S202, the data replication consensus is performed with the host device by using the detection tool and the parameter interaction table of the host device, so as to obtain a consensus result.

In operation S203, in the case where it is determined that the consensus result indicates consensus is reached, a data playback script is loaded into the data playback process.

In operation S204, a data playback process is performed to pull the target message from the message queue, and playback the target data obtained by parsing the target message in the database of the platform end device.

According to an embodiment of the present disclosure, the methods of operations S201 to S204 may be performed by a platform end device, where the platform end device may refer to any platform server, and the platform server may be an entity server disposed in a machine room, or may be a cloud server leased or purchased at a cloud end, which is not limited herein.

According to embodiments of the present disclosure, the data replication event may be a periodically triggered event, that is, the data replication event is triggered every time a period of time, so as to perform data replication from the host device to the platform device. Alternatively, the data copy event may be generated by the host device and sent to the platform device when a certain condition is satisfied, for example, the amount of incremental data of the host device reaches a threshold value, or a parameter value of one field in the parameter interaction table is a specific value. Still alternatively, the data replication event may be generated by the platform end device when a certain condition is satisfied, for example, the platform end device is in an idle state, or an application running in the platform end device reports errors due to lack of necessary data.

According to embodiments of the present disclosure, the data playback process may be an instance of a data playback program in a platform-side device. Accordingly, creating a data playback process may be a cold start of a data playback program, which may include loading, starting, rendering a view, etc. the data playback program.

According to embodiments of the present disclosure, the detection tool may be an application program provided in the platform end device. The detection tool can realize the loop detection of the specific field in the parameter interaction table of the host equipment so as to periodically acquire the parameter value of the specific field in the parameter interaction table. The detection tool may be compiled from a variety of programming languages, such as c++, JAVA, etc., and is not limited in this regard.

According to an embodiment of the present disclosure, the parameter interaction table may be a data table stored in the host-side device and having an association relationship with the platform-side device. A plurality of fields may be included in the parameter interaction table, each field may have a parameter value.

According to an embodiment of the present disclosure, the data replication consensus may be a process of performing data replication preparation work validation between the platform side device and the host side device based on the parameter values of the fields in the parameter interaction table. For example, the data replication consensus process may include the platform-side device and the host-side device writing data into the parameter interaction table, respectively, to indicate that the platform-side device and the host-side device are ready.

According to embodiments of the present disclosure, the data playback script may be a script file containing message retrieval, message parsing, and data playback logic that the data playback script includes when the data playback process runs after loading the data playback script into the data playback process.

According to the embodiments of the present disclosure, the message queue may be provided by a message middleware, which may be disposed on the host device, or may be disposed on the platform device, or may be disposed on another device capable of communicating with the host device and the platform device, which is not limited herein.

According to an embodiment of the present disclosure, playback of the target data in the database of the platform end device may be that the target data is written into the database of the platform end device in the original order.

According to the embodiment of the disclosure, after the platform end device runs the data playback process, the host end device may be configured to load the log analysis script into the log analysis process and run the log analysis process so as to analyze the database log of the host end device to obtain target data, and send a target message obtained by encapsulating the target data to the message queue.

The method shown in fig. 2 is further described below with reference to fig. 3-4 in conjunction with the exemplary embodiment.

According to an embodiment of the present disclosure, the data replication event may include the parameter value of the seventh field of the parameter interaction table being a sixth preset value. That is, the host device or the platform device may generate the data copy event when detecting that the parameter value of the seventh field of the parameter interaction table is a sixth preset value.

According to embodiments of the present disclosure, the parameter values of the seventh field of the parameter interaction table may be actively modified by the host-side device, e.g., the host-side device may be configured to stop the current log parsing process and update the parameter values of the seventh field to a sixth preset value in response to a trigger table access event. The table access event may be the creation of a new data table in the database of the host-side device or the importation of a new data table from an external data source. Stopping the current log analysis process can avoid the interference of the existing process to the newly added data replication task so as to ensure the data consistency of the host end equipment and the platform end equipment. The sixth preset value may be set according to a specific application scenario, for example, may be set to 1, Y, etc., which is not limited herein.

It should be noted that descriptions such as "first field", "seventh field", "first preset value", "sixth preset value", and the like are used herein only for distinguishing between different "fields" or "preset values", and are not intended to limit the order of "fields" or "preset values".

According to an embodiment of the present disclosure, operation S201 may specifically include the following operations:

stopping the current data playback process in response to triggering the data replication event; importing a table copy list of data to be copied from host equipment; and creating a data playback process based on the table copy manifest.

According to embodiments of the present disclosure, stopping the current data playback process may be the platform end device shutting down the current data playback process using a task manager or the like. By stopping the current data playback process, the interference of the current process to the newly added data replication task can be avoided, so that the data consistency of the host end equipment and the platform end equipment is ensured.

According to an embodiment of the disclosure, the table replication list of the data to be replicated may include a plurality of data table identifiers, and the data table corresponding to each data table identifier may include incremental data, where the incremental data may refer to newly added data from the last data replication task to the host device between the last data replication task and the first data replication task.

According to embodiments of the present disclosure, creating a data playback process based on a table replication manifest may be cold starting the data playback process and importing the table replication manifest into the data playback process.

According to embodiments of the present disclosure, data replication consensus may be implemented using a parameter interaction table of host-side devices.

As shown in fig. 3, the method includes operations S301 to S305, wherein operations S301, S303, and S305 may be performed by a platform-side device, and operations S302 and S304 may be performed by a host-side device.

In operation S301, in case it is determined that the creation of the data playback process is successful, the parameter value of the second field of the parameter interaction table is updated to the first preset value.

In operation S302, when the query results in that the parameter value of the second field of the parameter interaction table is the first preset value, a log parsing process is created, and the parameter value of the third field of the parameter interaction table is updated to the second preset value.

In operation S303, in case that the parameter value of the third field of the parameter interaction table is detected as the second preset value by the detection tool, the parameter value of the fourth field of the parameter interaction table is updated to the third preset value.

In operation S304, when the query results in the parameter value of the fourth field of the parameter interaction table being the third preset value, the log parsing process is configured based on the parameter value of the first field of the parameter interaction table, and the parameter value of the fifth field of the parameter interaction table is updated to the fourth preset value.

In operation S305, if the detection tool detects that the parameter value of the fifth field of the parameter interaction table is the fourth preset value, a consensus result for achieving data replication consensus with the host device is obtained.

According to the embodiment of the present disclosure, the first preset value, the second preset value, the third preset value, and the fourth preset value may be set according to a specific application scenario, for example, may be set to 1, Y, etc., which is not limited herein.

According to embodiments of the present disclosure, creating the log parsing process may be the host device cold starting the log parsing process.

According to an embodiment of the present disclosure, the platform end device updating the parameter value of the second field of the parameter interaction table to the first preset value may be notifying the host end device that the cold start of the data playback process has been completed. Correspondingly, the host device updating the parameter value of the third field of the parameter interaction table to the second preset value may be notifying the platform device that the log parsing process has been completed. And then, the platform end equipment and the host end equipment can finish one handshake operation by modifying the parameter value of the fourth field and the parameter value of the fifth field in sequence so as to give information of preparation work before completing data replication. By the adoption of the consensus mode, automatic data copying tasks can be guaranteed, and accuracy of data copying is improved.

According to the embodiments of the present disclosure, in the case where the parameter value of the second field is not the first preset value, or the parameter value of the fourth field is not the third preset value, the host device may continue the query of the second field or the fourth field after a period of time is spaced, where the period of time may be set according to a specific application scenario, for example, may be set to 1s, or the like. Correspondingly, if the platform end device detects that the parameter value of the third field is not the second preset value or the parameter value of the fifth field is not the fourth preset value, the platform end device can perform cyclic detection by using the detection tool.

According to an embodiment of the present disclosure, operation S204 may include the following operations:

running a data playback process, and sequentially acquiring messages from a message queue; screening the message by using the table replication list to obtain a screening result; analyzing the target message to obtain target data under the condition that the screening result indicates that the message is the target message; and playing back the target data in the database of the platform end device by utilizing the data playback script.

According to the embodiment of the disclosure, before the log parsing process of the host device runs, the message queue may not include a message, that is, the data playback process fails to acquire the message, and at this time, the data playback process may periodically perform a retry operation of acquiring the message.

According to embodiments of the present disclosure, in a single data replication task, a host-side device may transmit data to a plurality of platform-side devices using a single message queue, and the data required may be different between the plurality of platform-side devices. The table copy list of each platform terminal device can be recorded with the data required by the platform terminal device, and the accuracy of data copy can be ensured by screening the message by using the table copy list.

According to the embodiment of the disclosure, in the case that the screening result indicates that the message is not the target message, the message can be returned to the message queue, so that other platform end devices acquire the message and screen the message.

According to an embodiment of the present disclosure, when data replication is performed using a method such as operations S201 to S204, the platform side device may determine time information of data to be replicated from the task log and transmit the time information to the host side device, and the host side device may be configured to update a parameter value of a first field in the parameter interaction table based on the time information.

According to an embodiment of the present disclosure, data in a database of the host-side device may be stored in the form of records, and each record may be configured with time information formed by the record. After completing the data copying task, the platform end device may write the last recorded time information of the playback into the task log, where the time information may be the time information of the data to be copied in the next data copying.

According to the embodiment of the disclosure, the host device may restart the operation of the log parsing process after the platform device completes the operation of the data playback process. Specifically, the platform end device may update the parameter value of the sixth field of the parameter interaction table to a fifth preset value when the data playback process is in an operation state. The host device may be configured to load the log parsing script into the log parsing process and run the log parsing process when the parameter value of the sixth field of the parameter interaction table obtained by the query is a fifth preset value.

According to the embodiment of the present disclosure, the fifth preset value may be set according to a specific application scenario, for example, may be set to 1, Y, etc., which is not limited herein.

According to an embodiment of the present disclosure, in the process of executing the operation of loading the log parsing script into the log parsing process and running the log parsing process, the host device may be specifically configured to perform the following operations:

loading a log analysis script into a log analysis process; running a log analysis process to analyze a database log of the host device based on the time information to obtain data to be copied; and encapsulating the data to be copied into a plurality of messages, and sequentially sending the plurality of messages to a message queue.

In the embodiment disclosed at the root, the database log of the host device is parsed based on the time information, so that the data to be copied can be obtained by determining all data records located after the time information in the database of the host device from the database log based on the time information, and pulling all the data records from the host device.

According to an embodiment of the present disclosure, the data to be copied may include data corresponding to different data tables, and the encapsulating the data to be copied into the plurality of messages may be encapsulating the data corresponding to each data table in the data to be copied into one or more messages according to a size of a data amount, that is, all data included in a single message belong to the same data table. Or when the data to be copied is packaged into a plurality of messages, the data to be copied can be classified into sub-data related to each platform end device according to the table copy list of each platform end device, and each sub-data is packaged into one or a plurality of messages according to the size of the data volume.

According to the embodiment of the disclosure, the platform end device may include a platform server, and after the platform server is accessed to the platform end device, the platform end device and the host end device may configure the newly accessed platform server, so that the platform server may complete a data replication task. Specifically, the platform end device may determine a target platform server to be accessed in response to a trigger server online event; sending a detection tool and a data playback script to a target platform server through a server management system; and sending server online information to the host side device, wherein the host side device is configured to establish a parameter interaction table associated with the target platform server based on the server online information.

According to embodiments of the present disclosure, the platform end device may be another platform server different from the newly accessed platform server, or another terminal device, which is not limited herein.

According to an embodiment of the present disclosure, taking the newly accessed platform server as an example, a timing sequence flow of data copying from the host device to the platform server is described, where in the following embodiment, the first preset value, the second preset value, the third preset value, the fourth preset value, the fifth preset value and the sixth preset value are all set to Y.

As shown in fig. 4, for a newly accessed platform server 402, the flow of data replication may include an online flow and a replication flow.

According to an embodiment of the present disclosure, after the platform server 402 is online in the online process, a server online event may be sent to the platform end device 403, the platform end device 403 may upload a detection tool and a data playback script to the platform server through the server management system, and at the same time, the platform end device 403 may send a server online message to the host end device 401, and the host end device 401 may establish and initialize a parameter interaction table. As an optional implementation manner, in the online process, the platform server 402 may also send a server online event to the host device 401 and the platform device 403 respectively, or the platform server 402 may also send the server online event to the host device 401 first, and after the establishment of the parameter interaction table is completed, the host device 401 notifies the platform device 403. That is, the execution timing of the operation of uploading the detection tool and the data playback script by the platform-side device 403 and the operation of creating the parameter interaction table by the host-side device 401 may not be limited.

In accordance with an embodiment of the present disclosure, in the replication process, the platform server 402 may loop detect whether the parameter value of the seventh field in the parameter interaction table is Y. After triggering the table access event, the host device 401 may stop the current log parsing process and update the parameter value of the seventh field to Y. After detecting that the parameter value of the seventh field is Y, the platform server 402 may close the current data playback process and import a table copy list of the data to be copied from the host device 401, then, the platform server 402 may query the task log for time information of the latest data record, update the parameter value of the first field in the parameter interaction table with the time information, and then perform cold start on the data playback process, and after the cold start is completed, update the parameter value of the second field in the parameter interaction table to be Y. The host device 401 may implement the cyclic query on the parameter interaction table by using various tools, and after the query obtains the parameter value of the second field as Y, may cold start the log parsing process, and update the parameter value of the third field in the parameter interaction table as Y. After detecting that the parameter value of the third field is Y, the platform server 402 may update the parameter value of the fourth field in the parameter interaction table to Y. After the host device 401 queries that the parameter value of the fourth field is Y, it may suspend the log parsing process, configure the log parsing process based on the parameter value of the first field, and update the parameter value of the fifth field to be Y. After detecting that the parameter value of the fifth field is Y, the platform server 402 may run a data playback process and update the parameter value of the sixth field to be Y. After the host device 401 queries that the parameter value of the sixth field is Y, the log parsing process may be run. The log parsing process of the host device 401 may send the data to be copied to the message queue, and the data playback process of the platform server 402 may play back the message in the message pair queue in the database of the platform server 402, so as to implement data copying from the host device 401 to the platform server 402. After completing the data replication, the host device 401 may also initialize the parameter values of the fields in the parameter interaction table for performing the next data replication task.

As shown in fig. 5, the data replication device 500 includes a creation module 510, a consensus module 520, a loading module 530, and a running module 540. The data replication device 500 may be disposed on a platform end device.

A creation module 510 for creating a data playback process in response to triggering a data replication event.

And the consensus module 520 is configured to perform data replication consensus with the host device by using the detection tool and the parameter interaction table of the host device, so as to obtain a consensus result.

And a loading module 530, configured to load the data playback script into the data playback process if it is determined that the consensus result indicates consensus is reached.

And the operation module 540 is used for operating a data playback process so as to pull the target message from the message queue and playing back the target data obtained by analyzing the target message in the database of the platform end device.

According to the embodiment of the disclosure, the host device is configured to load a log analysis script into a log analysis process, run the log analysis process so as to analyze a database log of the host device to obtain target data, and send a target message obtained by packaging the target data to a message queue.

According to an embodiment of the present disclosure, the creation module 510 includes a first creation unit, a second creation unit, and a third creation unit.

And the first creating unit is used for responding to the triggering data copying event and stopping the current data playback process.

And the second creation unit is used for importing a table copy list of the data to be copied from the host device.

And a third creating unit for creating a data playback process based on the table copy list.

According to an embodiment of the present disclosure, the operation module 540 includes a first operation unit, a second operation unit, a third operation unit, and a fourth operation unit.

The first operation unit is used for operating the data playback process and sequentially acquiring the messages from the message queue.

And the second operation unit is used for screening the messages by using the table copy list to obtain screening results.

And the third operation unit is used for analyzing the target message to obtain target data under the condition that the screening result indicates that the message is the target message.

And the fourth running unit is used for playing back the target data in the database of the platform end equipment by utilizing the data playback script.

According to an embodiment of the present disclosure, the run module 540 further includes a fifth run unit.

And a fifth operation unit, configured to return the message to the message queue if the screening result indicates that the message is not the target message.

According to an embodiment of the present disclosure, the data replication device 500 further includes a first determination module and a first transmission module.

And the first determining module is used for determining the time information of the data to be copied from the task log.

And the first sending module is used for sending the time information to the host equipment.

According to an embodiment of the present disclosure, the host-side device is configured to update the parameter value of the first field in the parameter interaction table based on the time of day information.

According to an embodiment of the present disclosure, a host-side device is configured to: loading a log analysis script into a log analysis process; running a log analysis process to analyze a database log of the host device based on the time information to obtain data to be copied; and encapsulating the data to be copied into a plurality of messages, and sequentially sending the plurality of messages to a message queue.

According to an embodiment of the present disclosure, the consensus module 520 includes a first consensus unit, a second consensus unit, and a third consensus unit.

The first consensus unit is configured to update a parameter value of a second field of the parameter interaction table to a first preset value when it is determined that the data playback process is successfully created, where the host device is configured to create a log parsing process and update a parameter value of a third field of the parameter interaction table to the second preset value when the parameter value of the second field of the parameter interaction table obtained by querying is the first preset value.

And the second consensus unit is used for updating the parameter value of the fourth field of the parameter interaction table to a third preset value when the parameter value of the third field of the parameter interaction table is detected to be the second preset value by the detection tool, wherein the host device is configured to configure a log analysis process based on the parameter value of the first field of the parameter interaction table and update the parameter value of the fifth field of the parameter interaction table to the fourth preset value when the parameter value of the fourth field of the parameter interaction table is obtained by inquiry.

And the third consensus unit is used for obtaining a consensus result for achieving data replication consensus with the host side equipment under the condition that the detection tool detects that the parameter value of the fifth field of the parameter interaction table is a fourth preset value.

According to an embodiment of the present disclosure, the data replication device 500 further includes an update module.

And the updating module is used for updating the parameter value of the sixth field of the parameter interaction table to a fifth preset value under the condition that the data playback process is in an operation state.

According to an embodiment of the disclosure, the host device is configured to load the log parsing script into the log parsing process and run the log parsing process when the parameter value of the sixth field of the parameter interaction table obtained by the query is a fifth preset value.

According to an embodiment of the present disclosure, the data replication event includes that the parameter value of the seventh field of the parameter interaction table is a sixth preset value, and the parameter value of the seventh field of the parameter interaction table includes that the parameter interaction table is detected by using a detection tool.

According to an embodiment of the present disclosure, the host-side device is configured to stop the current log parsing process and update the parameter value of the seventh field to a sixth preset value in response to the trigger table access event.

According to an embodiment of the present disclosure, a platform end device includes a platform server.

According to an embodiment of the present disclosure, the data replication device 500 further includes a second determination module, a second transmission module, and a third transmission module.

And the second determining module is used for determining a target platform server to be accessed in response to the trigger server online event.

And the second sending module is used for sending the detection tool and the data playback script to the target platform server through the server management system.

And the third sending module is used for sending the server online information to the host equipment, wherein the host equipment is configured to establish a parameter interaction table associated with the target platform server based on the server online information.

Any number of modules, sub-modules, units, sub-units, or at least some of the functionality of any number of the sub-units according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented as split into multiple modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an Application Specific Integrated Circuit (ASIC), or in any other reasonable manner of hardware or firmware that integrates or encapsulates the circuit, or in any one of or a suitable combination of three of software, hardware, and firmware. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be at least partially implemented as computer program modules, which when executed, may perform the corresponding functions.

For example, any of the creation module 510, consensus module 520, loading module 530, and execution module 540 may be combined in one module/unit/sub-unit or any of the modules/units/sub-units may be split into multiple modules/units/sub-units. Alternatively, at least some of the functionality of one or more of these modules/units/sub-units may be combined with at least some of the functionality of other modules/units/sub-units and implemented in one module/unit/sub-unit. At least one of the creation module 510, the consensus module 520, the loading module 530, and the execution module 540 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable way of integrating or packaging circuitry, or in any one of or in any suitable combination of three implementations of software, hardware, and firmware, according to embodiments of the present disclosure. Alternatively, at least one of the creation module 510, the consensus module 520, the loading module 530, and the running module 540 may be at least partially implemented as a computer program module, which when run, may perform the corresponding functions.

It should be noted that, in the embodiment of the present disclosure, the data copying apparatus portion corresponds to the data copying method portion in the embodiment of the present disclosure, and the description of the data copying apparatus portion specifically refers to the data copying method portion and is not described herein again.

Fig. 6 schematically illustrates a block diagram of an electronic device adapted to implement a data replication method according to an embodiment of the disclosure. The electronic device shown in fig. 6 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.

As shown in fig. 6, a computer electronic device 600 according to an embodiment of the present disclosure includes a processor 601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. The processor 601 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. Processor 601 may also include on-board memory for caching purposes. The processor 601 may comprise a single processing unit or a plurality of processing units for performing different actions of the method flows according to embodiments of the disclosure.

In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The processor 601 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 602 and/or the RAM 603. Note that the program may be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the present disclosure, the electronic device 600 may also include an input/output (I/O) interface 605, the input/output (I/O) interface 605 also being connected to the bus 604. The electronic device 600 may also include one or more of the following components connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.

According to embodiments of the present disclosure, the method flow according to embodiments of the present disclosure may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 601. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples may include, but are not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 602 and/or RAM 603 and/or one or more memories other than ROM 602 and RAM 603 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program comprising program code for performing the methods provided by the embodiments of the present disclosure, the program code for causing an electronic device to implement the data replication methods provided by the embodiments of the present disclosure when the computer program product is run on the electronic device.

The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 601. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and/or installed from the removable medium 611. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be combined in various combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims

1. A data replication method applied to a platform end device, the method comprising:

creating a data playback process in response to triggering the data replication event;

carrying out data copying consensus with the host side equipment by using a parameter interaction table of the detection tool and the host side equipment to obtain a consensus result;

loading a data playback script into the data playback process if it is determined that the consensus result indicates consensus is achieved; and

running the data playback process so as to pull the target message from the message queue, and playing back the target data obtained by analyzing the target message in a database of the platform end equipment;

The host device is configured to load a log analysis script into a log analysis process, and run the log analysis process so as to analyze a database log of the host device to obtain the target data, and send the target message obtained by packaging the target data to the message queue.

2. The method of claim 1, wherein the creating a data playback process in response to triggering a data replication event comprises:

stopping the current data playback process in response to triggering the data replication event;

importing a table copy list of data to be copied from host equipment; and

the data playback process is created based on the table copy manifest.

3. The method of claim 2, wherein running the data playback process to pull a target message from a message queue and play back target data obtained by parsing the target message in a database of the platform end device comprises:

running the data playback process, and sequentially acquiring messages from the message queue;

screening the message by using the table replication list to obtain a screening result;

Analyzing the target message to obtain the target data under the condition that the screening result indicates that the message is the target message; and

and playing back the target data in a database of the platform end equipment by utilizing the data playback script.

4. A method according to claim 3, further comprising:

and returning the message to the message queue in the case that the screening result indicates that the message is not the target message.

5. The method of claim 2, further comprising:

determining the time information of the data to be copied from a task log; and

transmitting the time information to the host device;

wherein the host-side device is configured to update the parameter value of the first field in the parameter interaction table based on the time information.

6. The method of claim 5, wherein the host-side device is configured to:

loading the log analysis script into the log analysis process;

the log analysis process is operated to analyze the database log of the host equipment based on the time information to obtain the data to be copied; and

And packaging the data to be copied into a plurality of messages, and sequentially sending the messages to the message queue.

7. The method of claim 1, wherein the performing data replication consensus with the host device using the detection tool and the parameter interaction table of the host device to obtain a consensus result comprises:

under the condition that the data playback process is successfully established, updating the parameter value of the second field of the parameter interaction table to a first preset value, wherein the host side device is configured to establish the log analysis process and update the parameter value of the third field of the parameter interaction table to a second preset value under the condition that the parameter value of the second field of the parameter interaction table is obtained by query to be the first preset value;

when the detection tool detects that the parameter value of the third field of the parameter interaction table is the second preset value, updating the parameter value of the fourth field of the parameter interaction table to the third preset value, wherein the host device is configured to configure the log analysis process based on the parameter value of the first field of the parameter interaction table and update the parameter value of the fifth field of the parameter interaction table to the fourth preset value when the parameter value of the fourth field of the parameter interaction table is the third preset value; and

And under the condition that the detection tool detects that the parameter value of the fifth field of the parameter interaction table is the fourth preset value, obtaining the consensus result which achieves the data replication consensus with the host side equipment.

8. The method of claim 1, further comprising:

updating the parameter value of the sixth field of the parameter interaction table to a fifth preset value under the condition that the data playback process is in an operation state;

and the host device is configured to load the log analysis script into the log analysis process and run the log analysis process when the parameter value of the sixth field of the parameter interaction table obtained by query is the fifth preset value.

9. The method of claim 1, wherein the data replication event includes a sixth preset value for a parameter value of a seventh field of the parameter interaction table, the parameter value of the seventh field of the parameter interaction table including detection of the parameter interaction table with the detection tool;

the host device is configured to respond to a trigger table access event, stop the current log analysis process, and update the parameter value of the seventh field to the sixth preset value.

10. The method of claim 1, wherein the platform end device comprises a platform server;

the method further comprises the steps of:

responding to a trigger server online event, and determining a target platform server to be accessed;

transmitting the detection tool and the data playback script to the target platform server through a server management system; and

and sending server online information to the host side equipment, wherein the host side equipment is configured to establish a parameter interaction table associated with the target platform server based on the server online information.

11. A data replication apparatus disposed on a platform end device, the apparatus comprising:

the creation module is used for responding to the triggering data copying event and creating a data playback process;

the consensus module is used for carrying out data copying consensus with the host equipment by utilizing the detection tool and the parameter interaction table of the host equipment to obtain a consensus result;

the loading module is used for loading a data playback script into the data playback process under the condition that the consensus result is determined to indicate that consensus is achieved; and

the operation module is used for operating the data playback process so as to pull the target message from the message queue and playing back the target data obtained by analyzing the target message in the database of the platform end equipment;

12. An electronic device, comprising:

one or more processors;

a memory for storing one or more instructions,

wherein the one or more instructions, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1 to 10.

13. A computer readable storage medium having stored thereon executable instructions which when executed by a processor cause the processor to implement the method of any of claims 1 to 10.

14. A computer program product comprising computer executable instructions for implementing the method of any one of claims 1 to 10 when executed.