CN117992467A

CN117992467A - Data processing system, method and device and related equipment

Info

Publication number: CN117992467A
Application number: CN202211608424.7A
Authority: CN
Inventors: 王伟; 任仁; 曹宇
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2022-11-02
Filing date: 2022-12-14
Publication date: 2024-05-07
Also published as: WO2024093263A1

Abstract

There is provided a data processing system comprising a computing cluster, a storage cluster, wherein the computing cluster comprises a master computing node, a slave computing node, the storage cluster comprising at least one storage node; the main computing node is used for responding to the data updating request to generate binlog (binary log) and sending the binlog to the storage cluster for storage; and the slave computing node is used for reading the binlog stored in the storage cluster and updating the data stored in the storage cluster in a lasting way through playing back the binlog. Therefore, the binlog is transmitted between the master computing node and the slave computing nodes through the storage cluster, so that the fact that the slave computing nodes do not acquire the binlog can be avoided, and the slave computing nodes play back the acquired binlog, so that data synchronization between the master computing node and the slave computing nodes is realized, or data recovery is realized when the master computing node fails. In this way, the RPO of the data processing system can be implemented as 0, and the reliability of the data processing system can be improved.

Description

Data processing system, method and device and related equipment

The present application claims priority from the chinese patent application filed on month 2 of 2022 at 11 of the chinese national intellectual property agency, application number 202211363509.3, application name "a method of managing binlog", the entire contents of which are incorporated herein by reference.

Technical Field

The present application relates to the field of database technologies, and in particular, to a data processing system, method, apparatus, and related device.

Background

With the development of informatization technology, data processing systems, such as MySQL, postgreSQL and openGauss, are widely used in fields of finance, communication, medical treatment, logistics, electronic commerce and the like, and are used for performing persistent storage on business data in various fields.

Currently, data processing systems are typically deployed with a main center (or production center) and at least one disaster recovery center. The disaster recovery center comprises a main computing node and a main storage node. In normal operation, the main center provides data read-write service to the outside by using the main computing node and the main storage node; the backup center is responsible for backing up the data stored in the main center, and when the main center fails, the disaster recovery center can utilize the backed-up data to continue to provide data read-write service to the outside, so that the data is prevented from being lost, and the reliability of data storage is ensured.

In general, when a master computing node updates data stored in a master storage node in a persistent manner, a binary log (binlog) file is sent to a slave computing node, so that the slave computing node completes data updating in the slave storage node by playing back the binlog file, thereby realizing data synchronization between the master center and the disaster recovery center. However, in the actual application scenario, after the main center fails, the data stored in the disaster recovery center and the data of the main center before the failure often have inconsistent conditions, so that the recovery point target (recovery point object, RPO) of the data processing system cannot reach 0, and the reliability of the data processing system is affected. The RPO may be used, among other things, to measure the maximum amount of data loss that occurs during disaster recovery of the data processing system.

Disclosure of Invention

The data processing system is provided to keep the data stored in the disaster recovery center consistent with the data before the failure of the main center, so that the reliability of the data processing system is improved, and the RPO of the data processing system is 0. Furthermore, corresponding data processing methods, apparatuses, clusters of computing devices, chips, computer readable storage media and computer program products are provided.

In a first aspect, an embodiment of the present application provides a data processing system, where the data processing system includes a computing cluster and a storage cluster, where the computing cluster and the storage cluster are connected by a network, for example, by a wired network or a wireless network, and the computing cluster includes a master computing node and a slave computing node, where in a normal case, the slave computing node is a disaster recovery node of the master computing node, and the storage cluster includes at least one storage node; a master computing node for generating a binlog (binary log) in response to a data update request, which may be sent by a user to the master computing node or the like through a client, for example, and for sending the binlog to a storage cluster for storage; the slave computing node is used for reading the binlog stored in the storage cluster and updating the data stored in the storage cluster in a lasting way by playing back the binlog (specifically playing back the database statement recorded in the binlog), for example, the slave computing node can realize the data synchronization with the master computing node by playing back the binlog generated by the master computing node in time, or the slave computing node can realize the data recovery by playing back the binlog when the master computing node fails.

The binlog is transmitted between the master computing node and the slave computing nodes through the storage cluster, and the fact that the binlog is directly transmitted to the slave computing nodes by the master computing node is not needed is avoided, so that the situation that the load of the master computing node is overlarge, or a data transmission link between the master computing node and the slave computing nodes is unstable, the slave computing nodes do not acquire the binlog generated by the master computing node, the slave computing nodes play back the acquired binlog, and data synchronization between the master computing node and the slave computing nodes is achieved, or data recovery is achieved when the master computing node fails. Therefore, when the main computing node fails, the slave computing node can take over the service on the main computing node based on the data of the main computing node in the failure, so that the RPO of the data processing system is 0, and the reliability of the data processing system is improved.

In one possible implementation, the storage cluster includes a log storage area that is accessed by the master computing node and the slave computing nodes, i.e., the master computing node and the slave computing nodes may share the log storage area, such that the master computing node sends the binlog to the log storage area for storage when sending the binlog to the storage cluster; accordingly, binlog is read from the compute node, in particular from the log storage area; the storage cluster not only comprises a log storage area, but also comprises a data storage area, wherein the data storage area is used for storing service data, such as service data processed in the normal operation process of a master computing node, and the data storage area can be accessed by the master computing node and a slave computing node or only the master computing node (the slave computing node can access the data storage area only when the master computing node fails). Therefore, the transmission of the binlog can be realized through a shared storage area in the storage cluster between the master computing node and the slave computing nodes, so that the slave computing nodes can acquire the binlog generated by the master computing node as much as possible, and the reliability of the data processing system is improved.

In one possible implementation, the slave computing node is specifically configured to synchronize data with the master computing node by playing back binlog, that is, during normal operation of the master computing node, the slave computing node may continuously perform playback of binlog generated by the master computing node to achieve data synchronization with the master computing node; or the slave computing node is configured to recover the data lost when the master computing node fails by replaying the binlog, i.e., before the master computing node fails, the slave computing node may not perform the operation of replaying the binlog, and after the master computing node fails, the slave computing node recovers the data lost when the master computing node fails by performing the process of replaying the binlog so that the RPO of the data processing system is 0. Thus, the reliability of the data processing system can be effectively improved.

In one possible implementation manner, the storage nodes included in the storage cluster are a master storage node and a slave storage node, wherein the slave storage node is used as a disaster recovery device of the master storage node, and the master storage node and the slave storage node are deployed in the same data center or the same available area, and in general, the master storage node is used for providing a service of reading and writing data for the master computing node, and the slave storage node is used for providing a service of reading and writing data for the slave computing node; in this way, the slave computing node can read the binlog generated and issued by the master computing node from the log storage area of the storage cluster in the normal operation process of the master computing node, and update the data stored in the slave computing node in a lasting manner by playing back the binlog. Therefore, the slave computing node can realize data synchronization with the master computing node by continuously playing back the binlog generated by the master computing node, so that when the master computing node fails, the slave computing node can take over the service on the master computing node based on the synchronized data, RPO of the data processing system is 0, and the reliability of the data processing system is improved.

In one possible implementation, the storage cluster includes a storage node having a target storage node for persisting data written by the host computing node; the slave computing node may not perform the operation of replaying binlog in the normal operation of the master computing node, but may read binlog from the log storage area when the master computing node fails, and update the data stored in the target storage node in a persistent manner by replaying binlo, so as to recover the data lost by the master computing node when the master computing node fails, thereby improving the reliability of the data processing system.

In one possible implementation manner, the storage cluster includes storage nodes including a master storage node and a slave storage node, where the slave storage node is used as a disaster recovery device of the master storage node, and the master storage node and the slave storage node are disposed in different data centers or different available areas, and in general, the master storage node is used to provide a service of reading and writing data for the master computing node, and the slave storage node is used to provide a service of reading and writing data for the slave computing node; in the process of transmitting the binlog, the master computing node specifically sends the binlog to the master storage node for storage, and then the master storage node sends the binlog to the slave storage node, so that the slave computing node can read the binlog stored in the slave storage node. In this way, between the master computing node and the slave computing nodes, the reliability of the data processing system can be improved by transmitting the binlog between the master storage node and the slave storage nodes and ensuring that the slave computing nodes can acquire the binlog generated by the master computing node as much as possible.

In one possible implementation, the master storage node is further configured to, before sending the binlog to the slave storage node, also send baseline data to the slave storage node, where the baseline data typically includes data that the master storage node persists at a certain time (i.e., data stored on a data page) and binlog that the master computing node generated before the time; storing the baseline data from the storage node after it is received; thus, the slave computing node can update the baseline data stored in the slave storage node by playing back the binlog to realize data synchronization with the master center.

In one possible implementation, a target application is running on a host computing node, the binlog transmitted through the storage cluster being generated during the running of the target application, the target application comprising a relational database management system RDBMS comprising at least one of MySQL, postgreSQL, openGauss, oracle. Thus, aiming at a main computing node running any type of application, the reliability of the data processing system can be ensured by utilizing a mode of transmitting binlog by using a storage cluster, so that the compatibility and expansibility of the data processing system to database application can be improved.

In one possible implementation, the storage nodes in the storage cluster are in particular storage arrays for persisting the storage data. Because the storage array is generally configured with techniques based on redundant array of independent disks technology, erasure coding technology, erasure compression, data backup and the like, the reliability of persistent storage of data in the storage cluster can be further improved.

In a second aspect, an embodiment of the present application provides a data processing method, where the method is applied to a data processing system, where the data processing system includes a computing cluster and a storage cluster, where the computing cluster and the storage cluster are connected by a network, the computing cluster includes a master computing node and a slave computing node, and the storage cluster includes at least one storage node; the method comprises the following steps: the master computing node generates a binary log binlog in response to the data update request; the main computing node sends binlog to a storage cluster for storage; reading binlog stored in the storage cluster from the computing node; the data persistently stored in the storage cluster is updated from the computing node by playing back binlog.

In one possible implementation, a storage cluster includes a log storage area that is accessed by a master computing node and slave computing nodes; the master computing node sends binlog to the storage cluster for storage, including: the main computing node sends binlog to a log storage area for storage; reading binlog stored in a storage cluster from a computing node, comprising: reading binlog from the log storage area from the computing node; the storage cluster further comprises a data storage area, wherein the data storage area is used for storing service data, and the data storage area is accessed by the master computing node and the slave computing nodes or is accessed only by the master computing node.

In one possible implementation, the slave computing node, by playing back binlog, comprises: the slave computing node synchronizes data with the master computing node by replaying binlog or is used to recover data lost when the master computing node fails by replaying binlog.

In one possible implementation, the at least one storage node includes a master storage node and a slave storage node, where the slave storage node is used as a disaster recovery device for the master storage node, and the master storage node and the slave storage node are deployed in the same data center or the same available area; reading binlog stored in a storage cluster from a computing node, comprising: reading binlog from a log storage area by a slave computing node in the normal operation process of a master computing node; updating data persistently stored in a storage cluster from a computing node by playing back binlog, comprising: the slave computing node updates the data persisted from the storage node by playing back binlog.

In one possible implementation, the at least one storage node comprises a target storage node for persisting data written by the master computing node; reading binlog stored in a storage cluster from a computing node, comprising: the slave computing node reads binlog from the log storage area when the master computing node fails; updating data persistently stored in a storage cluster from a computing node by playing back binlog, comprising: the data persistently stored in the target storage node is updated from the computing node by playing back binlog.

In one possible implementation, the at least one storage node includes a master storage node and a slave storage node, where the slave storage node is used as a disaster recovery device for the master storage node, and the master storage node and the slave storage node are disposed in different data centers, or the master storage node and the slave storage node are disposed in different available areas; the master computing node sends binlog to the storage cluster for storage, including: the main computing node sends binlog to a main storage node for storage; reading binlog stored in a storage cluster from a computing node, comprising: the slave computing node is specifically configured to read the binlog stored in the slave storage node, where the binlog is sent by the master storage node.

In one possible embodiment, the method further comprises: the master storage node sends the baseline data to the slave storage node for storage before sending the binlog to the slave storage node; updating data persistently stored in a storage cluster from a computing node by playing back binlog, comprising: the slave computing node updates the baseline data stored in the slave storage node by playing back binlog.

In one possible implementation, the target application is running on the host computing node, binlog is generated during the running of the target application, the target application comprises a relational database management system RDBMS, and the RDBMS comprises at least one of MySQL, postgreSQL, openGauss, oracle.

In one possible implementation, the storage nodes are storage arrays that are used to persist the data.

In a third aspect, an embodiment of the present application provides a data processing apparatus, where the data processing apparatus is applied to a data processing system, where the data processing system includes a computing cluster and a storage cluster, where the computing cluster and the storage cluster are connected by a network, the computing cluster includes a master computing node and a slave computing node, and the storage cluster includes at least one storage node; the data processing device includes: the storage module is used for indicating the main computing node to send the binary log binlog generated in response to the data updating request to the storage cluster for storage; a reading module, configured to instruct a slave computing node to read the binlog stored in the storage cluster to the slave computing node; a playback module for instructing the slave computing node to update the data persistently stored in the storage cluster by playing back binlog.

In one possible implementation, a storage cluster includes a log storage area that is accessed by a master computing node and slave computing nodes; the storage module is specifically used for indicating the main computing node to send the binlog to the log storage area for storage; the reading module is specifically used for indicating the slave computing node to read binlog from the log storage area to the slave storage node; the storage cluster further comprises a data storage area, wherein the data storage area is used for storing service data, and the data storage area is accessed by the master computing node and the slave computing nodes or is accessed only by the master computing node.

In one possible implementation, the revisit module is specifically configured to instruct the slave computing node to synchronize data with the master computing node by replaying binlog, or specifically configured to instruct the slave computing node to recover data lost when the master computing node fails by replaying binlog.

In one possible implementation, the at least one storage node includes a master storage node and a slave storage node, where the slave storage node is used as a disaster recovery device for the master storage node, and the master storage node and the slave storage node are deployed in the same data center or the same available area; the reading module is specifically used for indicating the slave computing node to read binlog from the log storage area to the slave computing node in the normal operation process of the master computing node; and the playback module is specifically used for indicating the slave computing node to update the data stored in the slave storage node in a lasting way through playing back the binlog.

In one possible implementation, the at least one storage node comprises a target storage node for persisting data written by the master computing node; the reading module is specifically used for indicating the slave computing node to read binlog from the log storage area when the master computing node fails; and the playback module is specifically used for indicating the slave computing node to update the data stored in the target storage node in a persistent mode through playing back binlog.

In one possible implementation, the at least one storage node includes a master storage node and a slave storage node, where the slave storage node is used as a disaster recovery device for the master storage node, and the master storage node and the slave storage node are disposed in different data centers, or the master storage node and the slave storage node are disposed in different available areas; the storage module is specifically used for indicating the main computing node to send the binlog to the main storage node for storage; the reading module is specifically configured to instruct the slave computing node to read the binlog stored in the slave storage node, where the binlog is sent by the master storage node.

In a possible implementation manner, the storage module is further configured to: instruct the master storage node to send the baseline data to the slave storage node for storage before sending the binlog to the slave storage node; and the playback module is specifically used for indicating the slave computing node to update the baseline data stored in the slave storage node by playing back binlog.

In a fourth aspect, an embodiment of the present application provides a computing device cluster including at least one computing device, each of the at least one computing device comprising: a processor and a memory; the memory is configured to store instructions that the processor executes to cause the cluster of computing devices to perform the data processing method described in the second aspect or any implementation of the second aspect, or to implement the data processing apparatus described in the third aspect or any implementation of the third aspect. It should be noted that the memory may be integrated into the processor or may be independent of the processor. The computing device may also include a bus. The processor is connected with the memory through a bus. The memory may include a readable memory and a random access memory, among others.

In a fifth aspect, an embodiment of the present application provides a chip, including a power supply circuit and a processing circuit, where the power supply circuit is configured to supply power to the processing circuit, and the processing circuit implements the data processing apparatus described in the third aspect or any implementation manner of the third aspect.

In a sixth aspect, an embodiment of the present application further provides a computer readable storage medium, where a program or an instruction is stored, which when executed on a computer, causes the data processing method described by the first device in the second aspect or any implementation manner of the second aspect to be performed.

In a seventh aspect, embodiments of the present application also provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the data processing method as described in any of the implementations of the second or third aspects above.

In addition, the technical effects caused by any implementation manner of the second aspect to the seventh aspect may refer to the technical effects caused by different implementation manners of the first aspect, which are not described herein.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments described in the present application, and other drawings may be obtained according to these drawings for those of ordinary skill in the art.

FIG. 1 is a schematic diagram of an exemplary data processing system according to an embodiment of the present application;

FIG. 2 is a schematic diagram of another exemplary data processing system provided in accordance with an embodiment of the present application;

FIG. 3 is a schematic diagram of a further exemplary data processing system provided in accordance with an embodiment of the present application;

FIG. 4 is a schematic diagram of a further exemplary data processing system provided in accordance with an embodiment of the present application;

FIG. 5 is a schematic diagram of a further exemplary data processing system provided in accordance with an embodiment of the present application;

FIG. 6 is a schematic diagram of a further exemplary data processing system provided in accordance with an embodiment of the present application;

FIG. 7 is a flow chart of an exemplary data processing method according to an embodiment of the present application;

FIG. 8 is a schematic diagram of an exemplary data processing apparatus according to an embodiment of the present application;

FIG. 9 is a schematic diagram of an exemplary computing device provided in accordance with an embodiment of the present application.

Detailed Description

In order that the above objects, features and advantages of the present application will be more readily understood, a more particular description of various non-limiting embodiments of the application will be rendered by reference to the appended drawings. It will be apparent that the described embodiments are some, but not all, embodiments of the application. All other embodiments obtained based on the above are within the scope of protection of the present application based on the embodiments of the present application.

With reference to FIG. 1, a schematic diagram of an exemplary data processing system 100 is shown, in which data processing system 100 may employ a memory-split architecture. As shown in fig. 1, a data processing system 100 includes a computing cluster 101 and a storage cluster 102, and the computing cluster 101 and the storage cluster 102 may communicate with each other via a network, for example, a wired network or a wireless network.

Wherein the computing cluster 101 comprises a plurality of computing nodes, the different computing nodes may communicate with each other, and each computing node may be a computing device including a processor, such as a server, a desktop computer, etc. In the computing cluster 101, a portion of the computing nodes may act as disaster recovery for another portion of the computing nodes. For ease of illustration, computing cluster 101 is illustrated in FIG. 1 as including a master computing node 1011 and a slave computing node 1012, with slave computing node 1012 acting as a disaster recovery device for master computing node 1011. Illustratively, slave computing node 1012 may act as either a hot or a cold standby for master computing node 1011. Wherein, when the slave computing node 1012 is used as a hot standby, the slave computing node 1012 and the master computing node 1011 are continuously in an operation state; thus, when a failure occurs in the master computing node 1011, the slave computing node 1012 can immediately take over traffic on the master computing node 1011 using the backed up data, specifically to process requests that the master computing node 1011 did not process for completion at the time of the failure. When the slave computing node 1012 is operating as a backup, the slave computing node 1012 may not be operating (e.g., in a dormant state, etc.) during normal operation of the master computing node 1011, or the slave computing node 1012 may free computing resources thereon and utilize the freed computing resources to process other traffic, such as offline computing traffic, etc. When the primary computing node 1011 fails, the secondary computing node 1012 initiates the running/reclaiming of computing resources and utilizes the backed up data to effect takeover of traffic on the primary computing node 1011. In practice, the master computing node 1011 may have a plurality of slave computing nodes acting as disaster recovery nodes, so that some of the slave computing nodes may act as cold-standby for the master computing node 1011, while another of the slave computing nodes may act as hot-standby for the master computing node 1011, etc.

Storage cluster 102 may include one or more storage nodes, each of which may be a device including a persistent storage medium, such as network attached storage (network attached storage, NAS), storage servers, etc., that may be used to persistent store data. The persistent storage medium in the storage node may be, for example, a hard disk, such as a solid state hard disk or a shingled magnetic recording hard disk. In practice, each storage node may be built up from one or more devices for persisting stored data. When storage cluster 102 includes multiple storage nodes, a portion of the storage nodes may act as disaster recovery for another portion of the storage nodes. For ease of illustration, storage cluster 102 is illustrated in fig. 1 as including a master storage node 1021 and a slave storage node 1022, with master storage node 1021 being a disaster recovery for slave storage node 1022, data storage areas on master storage node 1021 and slave storage node 1022 being used to store traffic data, respectively, and data storage areas on master storage node 1021 being accessed by master computing node 1011 and data storage areas on slave storage node 1022 being accessed by slave computing node 1012. The master storage node 1021 and the slave storage node 1022 may be disposed in the same data center, or disposed in the same available area (availability zones, AZ), or the like. At this time, by creating multiple copies for the data stored persistently in the same data center or in the same AZ, the reliability of the data stored locally can be improved. Or the master storage node 1021 and the slave storage node 1022 may be deployed in different data centers, e.g., master storage node 1021 deployed in data center a and slave storage node 1022 deployed in data center B; or the master storage node 1021 and the slave storage node 1022 may be disposed in different AZs, e.g., the master storage node 1021 is disposed in AZ ₁, the slave storage node 1022 is disposed in AZ ₂, etc. Therefore, data disaster recovery can be realized across data centers or across AZ, so that the reliability of data storage in different places is improved.

The master computing node 1011 provides data read-write services using the master storage node 1021, and, after a failure of the master computing node 1011, the slave computing node 1012 takes over traffic on the master computing node 1011 using the data backed up on the slave storage node 1022. In an actual application scenario, the master computing node 1011 and the master storage node 1021 may form a master center (typically belonging to a production site), and the slave computing node 1012 and the slave storage node 1022 may form a disaster recovery center (typically belonging to a disaster recovery site). In other possible data processing systems, storage cluster 102 may also include a storage node, where master computing node 1011 and slave computing node 1012 may share the storage node, i.e., traffic data stored in a data storage area on the storage node may be accessed by master computing node 1011 and slave computing node 1012, so that after a failure of master computing node 1011, slave computing node 1012 may continue to provide data read and write services using the data stored in the storage node.

The primary computing node 1011 may have one or more applications deployed thereon (not shown in fig. 1), which may be, for example, database applications or other applications, etc. By way of example, the database application may be a relational database management system (relational database MANAGEMENT SYSTEM, RDBMS) or the like, which may include at least one of MySQL, postgreSQL, openGauss, oracle, or may be other types of database systems or the like. During the running of an application, the primary computing node 1011 typically receives a data update request sent by a client or other device on the user side, such as a data update request sent by a client on the user side to read or modify data in the primary storage node 1021. At this point, an application on the host computing node 1011 may provide corresponding data read-write services for the client or other device in response to the data update request. Wherein, when a data update request received by master computing node 1011 is used to request a new data write to data processing system 100, or to request a modification to data already stored persistently in data processing system 100, or to request a deletion of data already stored persistently in data processing system 100, an application on master computing node 1011 generates a binary log (binlog) and saves the binlog in a local storage area. The binlog is a logical log, and is used for recording and updating database sentences, such as SQL sentences, of data stored in the main storage node 1021 in a persistent manner. In an actual scenario, the application may include a service layer and a storage engine layer, and binlog may be generated and saved by the service layer. The master computing node 1011 then sends the generated binlog to the slave computing node 1012 and the slave computing node 1012 updates the data in the slave storage node 1022 by executing the database statement in the binlog, so that the data in the slave storage node 1022 is consistent with the data in the master storage node 1021, i.e., copying the data in the master storage node 1021 into the slave storage node 1022 is accomplished.

After a failure of the master computing node 1011, the slave computing node 1012 needs to run/reclaim the computing resources and use the computing resources to launch applications running on the slave computing node 1012. The application on the slave computing node 1012 then executes the database statement recorded in binlog sent by the master computing node 1011 before the failure so that the data in the slave storage node 1022 remains consistent with the data of the storage node 1021 before the failure. In this way, slave computing node 1012 is able to take over outstanding requests at master computing node 1011 based on data stored in slave storage node 1022.

However, in an actual application scenario, there may be some binlog in the master computing node 1011 that is not successfully transmitted to the slave computing node 1012. For example, when the data transmission link between the master computing node 1011 and the slave computing node 1012 is unstable (such as high data transmission jitter or high communication network transmission pressure, etc.), the binlog transmitted by the master computing node 1011 may be lost during transmission of the communication network, resulting in difficulty in receiving the binlog from the computing node 1012. For another example, when the traffic load on the master computing node 1011 is large, it may be difficult for the master computing node 1011 to send locally stored binlogs to the slave computing node 1012 in time while continuing to generate new binlogs, and the storage space of the local storage area in the master computing node 1011 for storing the binlogs is limited, which causes the master computing node 1011 to eliminate the first stored part of the binlogs in the local storage area so as to store the new binlogs. At this time, since a part of binlog on the master computing node 1011 is not transmitted to the slave computing node 1012, it is difficult for the slave computing node 1012 to keep the data in the master storage node 1021 and the data between the slave storage nodes 1022 in synchronization by playing back the part of binlog, that is, data inconsistency between the master center and the disaster recovery center. Thus, when the primary computing node 1011 of the primary center fails, the disaster recovery center cannot restore the data to the state when the primary computing node 1011 fails, which results in the RPO of the data processing system 100 not reaching 0, i.e., there is a loss of part of the data, which affects the reliability of the data processing system 100.

Based on this, in the data processing system 100 provided by the present application, after generating the binlog, the master computing node 1011 sends the binlog to the storage cluster 102, and the storage cluster 102 stores the binlog. The slave computing node 1012 then reads the binlog from the storage cluster 102 and updates the data persistently stored in the storage cluster 102 by playing back the binlog (i.e., playing back the database statement recorded in the binlog), specifically, the data in the slave storage node 1021, to achieve that the data in the slave storage node 1022 is kept consistent with the data in the master storage node 1021, or with the data in the master storage node 1021+the data cached by the master computing node 1011. Since the binlog generated by the master computing node 1011 is transmitted to the slave computing node 1012 through the storage side, it is possible to avoid the problem of data asynchronization caused by that the load of the master computing node 1011 is excessive or the data transmission link between the master computing node 1011 and the slave computing node 1012 is unstable, resulting in that the slave computing node 1012 does not acquire the binlog generated by the master computing node 1011. Thus, when the master computing node 1011 fails, the data in the slave storage node 1022 remains consistent with the data in the master storage node 1021, which enables the slave computing node 1012 to take over traffic on the master computing node 1011 based on the data in the slave storage node 1022, thereby achieving an RPO of 0 for the data processing system 100 and improving the reliability of the data processing system 100.

In addition, when the master computing node 1011 and the slave computing node 1012 simultaneously include a plurality of database applications MySQL, postgreSQL, openGauss, oracle, unified processing logic can be adopted between the master computing node 1011 and the slave computing node 1012 to copy the binlog generated by the master computing node 1011 to the slave computing node 1012, so that the compatibility of the data processing system 100 to the database applications can be improved, and the deployment difficulty of the database applications on the computing cluster 101 can be reduced.

It is noted that data processing system 100 shown in FIG. 1 is only illustrative and that data processing system 100 may be implemented in other ways during actual use. To facilitate understanding, the present embodiment provides the following several implementation examples.

In a first implementation example, only one storage node may be included in storage cluster 102 for master computing node 1011 and slave computing node 1012, such that master computing node 1011 and slave computing node 1012 may share access to pages of data in the storage nodes.

In a second implementation example, a metadata management cluster may also be included between the computing cluster 101 and the storage cluster 102, where the metadata management cluster is responsible for managing metadata stored in the storage cluster 102; accordingly, a computing node in the computing cluster 101 may first access metadata from the metadata management cluster to access data stored in the storage cluster 102 based on the metadata.

In a third implementation example, 3 or more nodes may be included in both the compute cluster and the storage cluster in data processing system 200, as shown in FIG. 2. Specifically, the computing cluster includes a plurality of computing nodes 410, each of the computing nodes 410 may communicate with each other, and a portion of the computing nodes 410 may act as a disaster recovery for another portion of the computing nodes 410. Each computing node 410 is a computing device, such as a server, desktop computer, or the like, that includes a processor. In hardware, as shown in fig. 2, computing node 410 includes at least a processor 412, a memory 413, a network card 414, and a storage medium 415. Wherein the processor 412 is a central processing unit (central processing unit, CPU) for processing data access requests from outside the computing node 410 or requests generated internally to the computing node 410. Processor 412 reads data from memory 413 or when the total amount of data in memory 413 reaches a threshold, processor 412 sends the data stored in memory 413 to storage node 400 for persistent storage. Only one CPU 412 is shown in fig. 2, and in practical applications, there are often a plurality of CPUs 412, where one CPU 412 has one or more CPU cores. The present embodiment does not limit the number of CPUs and the number of CPU cores. Also, the processor 412 in the compute node 410 may also be used to implement the above described functionality of writing binlog to and/or reading and playing back binlog from a storage cluster to achieve data synchronization between different storage nodes 400 in the storage cluster.

Memory 413 is an internal memory that exchanges data directly with the processor, and can read and write data at any time and at a high speed, and is used as a temporary data memory for an operating system or other running programs. The Memory includes at least two types of memories, for example, the Memory may be a random access Memory (ram) or a Read Only Memory (ROM). In practice, multiple memories 413 may be configured in the compute node 410, as well as different types of memories 413. The number and type of the memories 413 are not limited in this embodiment.

The network card 414 is used to communicate with the storage node 400. For example, when the total amount of data in memory 413 reaches a threshold, computing node 410 may send a request to storage node 400 via network card 414 to persist the data. In addition, the computing node 410 may also include a bus for communication among the components within the computing node 410. In practical implementation, the computing node 410 may also have a small number of hard disks built in, or be externally connected with a small number of hard disks.

Each computing node 410 may access storage nodes 400 in a storage cluster over a network. The storage cluster includes a plurality of storage nodes 400, and a portion of the storage nodes 400 may act as disaster recovery for another portion of the storage nodes 400. A storage node 400 includes one or more controllers 401, network cards 404, and a plurality of hard disks 405. The network card 404 is used to communicate with a computing node 410. The hard disk 405 is used to persist data and may be a magnetic disk or other type of storage medium, such as a solid state disk or shingled magnetic recording hard disk, etc. The controller 401 is configured to write data to the hard disk 405 or read data from the hard disk 405 according to a read/write data request sent by the computing node 410. In the process of reading and writing data, the controller 401 needs to convert an address carried in a read/write data request into an address identifiable by the hard disk.

For ease of understanding and explanation, the process of implementing data synchronization between a master hub (including master computing node 1011 and master storage node 1021) and a disaster recovery hub (including slave computing node 1012 and slave storage node 1022) based on data processing system 100 shown in FIG. 1 is described in detail below.

Typically, one or more applications (such as MySQL) are running on the host computing node 1011, and for ease of understanding, the following description will be given by taking as an example a target application running on the host computing node 1011, where the target application can support the host computing node 1011 to provide data read-write services for users during running. Taking the example of modifying data by a user request, after the host computing node 1011 receives a data update request for modifying data sent by a user through a client, the target application may first read a data page where the data requested to be modified by the data update request is located from the host storage node 1021 to a buffer pool (buffer pool) in the host computing node 1011, and complete modification of the data page in the buffer pool according to the data update request, specifically, modify the data on the data page into new data (the new data may be empty, and delete the data on the data page at this time). At this point, the target application will generate a binlog for the data modification content, which is used to record database statements indicating modification to the data, and for convenience of distinction and description, the new data will be referred to as target data hereinafter.

After completing the modification to the data in the buffer pool and generating binlog, the master computing node 1011 may feed back to the client that the data writing/modification was successful. Since the speed of writing data into the buffer pool is typically higher than the speed of persistent storage of data, the host computing node 1011 can be accelerated in response to data update requests. In actual application, when the amount of data accumulated in the buffer pool reaches a threshold, the master computing node 1011 sends the data in the buffer pool to the master storage node 1021 for persistent storage, and may delete the binlog corresponding to the data. Alternatively, the master computing node 1011 may send the binlog to the storage cluster 102 successfully for storage, and then feed back the success of data writing/modification to the client.

The data may be stored in a file format in the master storage node 1021 (and the slave storage node 1022), and at this time, the master storage node 1021 (and the slave storage node 1022) may be deployed with a corresponding file system (FILE SYSTEM, FS) for managing the file stored in the persistent manner. Alternatively, the data may be stored in the master storage node 1021 (and the slave storage node 1022) in a data block format (data block format), that is, when the master storage node 1021 (and the slave storage node 1022) store the data, the data may be partitioned into blocks of a fixed size, and the data amount of each partition may be 512 bytes or 4 Kilobytes (KB), for example. Or the data may be stored in the master storage node 1021 (and the slave storage node 1022) in an object (object) format, where an object may be a basic unit of storage node storing data, and each object may include a complex of data and an attribute of the data, where the attribute of the data may be set according to a requirement of an application in the computing node, including data distribution, quality of service, and so on. The storage format of the data in this embodiment is not limited.

In this embodiment, the following several exemplary implementations are provided for the process of the master computing node 1011 sending the generated binlog to the storage cluster 102 and storing the binlog in the storage cluster 102.

In a first possible implementation, the master computing node 1011 may send the binlog to the master storage node 1021 in the storage cluster 102 so that the master storage node 1021 maintains the binlog. In the storage cluster 102, since the slave storage node 1022 is a disaster recovery device for the master storage node 1021, after the master computing node 1011 writes the binlog to the master storage node 1021, the master storage node 1021 may back up the written binlog, and send the backed-up binlog to the slave storage node 1022 through a wired or wireless manner. When determining that the slave storage node 1022 successfully writes the backup binlog, the master storage node 1021 may feed back to the master computing node 1011 that the binlog was successfully written.

Illustratively, as shown in fig. 3, the master storage node 1021 and the slave storage node 1022 may be disposed in the same data center or the same AZ, at this time, a wired or wireless connection may be established between the master storage node 1021 and the slave storage node 1022, and the backup binlog is sent to the slave storage node 1022 for storage through the wired or wireless connection.

Alternatively, as shown in fig. 4, the master storage node 1021 and the slave storage node 1022 may be disposed in different AZ, e.g., the master storage node 1021 is disposed in AZ ₁, the slave storage node 1022 is disposed in AZ ₂, etc. At this time, the master storage node 1021 may send the backed up binlog to the slave storage node 1022 through the network card or the network interface for storage.

The implementations shown in fig. 3 and 4 described above are merely illustrative. For example, in other possible embodiments, there may be multiple slave storage nodes serving as disaster recovery nodes in the master storage node 1021, where some of the multiple slave storage nodes may be disposed in the same physical area as the master storage node 1021, such as in the same data center or the same AZ, and another of the multiple slave storage nodes may be disposed in a different physical area as the master storage node 1021, such as in a different data center/AZ, so as to improve the reliability of data storage locally and storage in different places.

In this way, the slave computing node 1012 may read the binlog stored in the slave storage node 1022 and update the data stored in the slave storage node 1022 (in particular, the data on the data page) by playing back the database statement recorded in the binlog, thereby achieving data synchronization between the data in the slave storage node 1022 and the master storage node 1021, or referred to as data synchronization between the disaster recovery center and the master center.

As an example, where an input/output (IO) thread and a database thread (e.g., an SQL thread) may be created in the slave computing node 1012, the slave computing node 1012 may access the slave storage node 1022 with the IO thread to read the binlog in the slave storage node 1022 and store the read binlog to the local storage area. The slave computing node 1012 may then sequentially perform playback operations on each binlog in the local storage area using the database thread. For example, each binlog in the local storage area may have a log number (log sequence number, LSN) so that the database thread may sequentially replay each binlog in order of LSN from small to large. Specifically, when each binlog is played back, the database thread may parse a database statement to be executed, such as an SQL statement, from the binlog, and perform semantic analysis and syntax analysis on the database statement to determine validity of the database statement. The grammar analysis is to check whether grammar errors exist in the database statement according to grammar rules of database language; semantic analysis refers to analyzing whether the semantics of the database statement are legal. After passing the validity check, the database thread may generate a plan tree for the database statement, the plan tree indicating an execution plan for processing the data. Finally, the database thread may perform data updates from the storage node 1022 based on the optimized plan tree after optimization of the plan tree is completed.

In practice, the master computing node 1011 and the slave computing node 1012 may have the same configuration. For example, when creating the disaster recovery center, configuration files in the master computing node 1011 and the master storage node 1021 may be backed up, and the backed up configuration files may be sent to the slave computing node 1012 and the slave storage node 1022 in the disaster recovery center, respectively, so that the slave computing node 1012 may have the same configuration as the master computing node 1012 based on the received configuration files, and the slave storage node 1022 may have the same configuration as the master storage node 1021 based on the received configuration files. In this way, when the master computing node 1011 sends the binlog to the master storage node 1021, the binlog in the master storage node 1021 may be mounted to a designated directory, so that after receiving the binlog sent by the master storage node 1021, the slave storage node 1022 may store the binlog in a storage location corresponding to the directory. In this way, the slave computing node 1012 can read the binlog belonging to the directory stored in the slave storage node 1022 based on the directory uniformly configured with the master computing node 1011.

In a second possible implementation, as shown in FIG. 5, a log storage area 501 may be configured in storage cluster 102, where log storage area 501 is commonly accessible by master computing node 1011 and slave computing nodes 1012. Illustratively, as shown in fig. 5, the log storage area 501 may be a partial storage area on the master storage node 1021 or the slave storage node 1022, or may be a storage area on another storage node independent of the master storage node 1021 and the slave storage node 1022, or the like, which is not limited in this embodiment. At this time, the master computing node 1011 may send the binlog to the log storage area 501 in the storage cluster 102 for storage, for example, may send the binlog to the log storage area 501 under a specified directory for storage. The slave computing node 1012 may access the log storage area 501 to obtain the binlog generated by the master computing node 1011, for example, by accessing the corresponding log storage area 501 according to a specified directory to obtain the binlog, etc. Further, the master computing node 1012 may implement data synchronization between the master storage node 1021 and the slave storage node 1022 by playing back the binlog. The specific implementation of the playback binlog by the master computing node 1012 may be referred to in the foregoing description, and will not be repeated here.

It should be noted that the above two implementations are only exemplary, and that the slave computing node 1012 may obtain the binlog generated by the master computing node 1011 from the storage cluster 102 in other manners when actually applied.

In a further possible implementation, a baseline (baseline) copy may also be pre-completed between the master storage node 1021 and the slave storage node 1022 before the master computing node 1011 synchronizes the currently generated binlog to the slave computing node 1012 through the storage cluster 102. The baseline replication refers to sending all the data that the primary storage node 1021 has been stored permanently at a certain point in time (e.g., at the current time) and the binlog that the primary computing node 1011 has generated to the disaster recovery center. For example, when the slave storage node 1022 is created, the data synchronization process is performed between the master storage node 1021 and the slave storage node 1022 for the first time, and then the data synchronization may be implemented between the master storage node 1021 and the slave storage node 1022 in a baseline copy manner.

In particular, the primary storage node 1021 may determine a first time (e.g., a current time) as the time corresponding to the baseline, and determine baseline data based on the time. The baseline data includes data that the primary storage node 1021 is persistently storing at a first time (i.e., data stored on a data page), and binlog generated by the primary computing node 1011 before the first time, i.e., the LSN is less than or equal to the binlog of the LSN corresponding to the first time. Typically, the updated data indicated by binlog in the baseline data is stored in a buffer pool and has not yet been sunk into the primary storage node 1021 for persistent storage. The master storage node 1021 may then send this baseline data to the slave storage node 1022, either by wire or wirelessly. After the slave storage node 1022 successfully stores the baseline data, the slave computing node 1012 may sequentially perform the process of playing back the binlog of each of the baseline data according to the sequence from small to large LSNs, so as to update the data belonging to the data page in the baseline data, thereby implementing synchronization of the data stored by the master center at the first moment to the disaster recovery center. Thus, when the master computing node 1011 generates a binlog based on the data newly written by the user, the master computing node 1011 sends the binlog to the slave computing node 1012 through the storage cluster 102, and then the slave computing node 1012 updates the baseline data stored in the slave storage node 1022 by playing back the binlog, so that the data synchronization between the master center and the disaster recovery center is realized in time.

In actual application, when a part of binlog exists in the master storage node 1021 and is deleted in the master storage node 1021 without being timely sent to the slave storage node 1022, such as the life cycle of the part of binlog in the master storage node 1021 is deleted beyond a preset time length, at this time, the slave storage node 1022 cannot obtain the part of binlog deleted to keep data synchronization with the master storage node 1021, and therefore, data synchronization between the master storage node 1021 and the slave storage node 1022 can be achieved by performing baseline copy again.

During operation of data processing system 100, slave computing node 1012 may detect whether a failure has occurred in master computing node 1011 in real time or periodically, e.g., slave computing node 1012 may determine whether a failure has occurred in master computing node 1011 when a heartbeat message sent by master computing node 1011 has not been received, or determine whether a failure has occurred in master computing node 1011 when a failure notification sent by an arbitration server of a third party has been received, etc. When it is determined that the primary computing node 1011 is down, the secondary computing node 1012 upgrades to the primary computing node and instructs the storage node 1022 to upgrade to the primary storage node. At this time, the slave computing node 1012 may first check whether the slave storage node 1022 or the log storage area 501 stores binlog that has not yet been played back, and if so, the slave computing node 1012 first reads and plays back the binlog to complete updating of the data stored in the slave storage node 1022 in a persistent manner, where the updated data is the data stored in the master center when the master computing node 101 fails; the slave computing node 1012 then continues to take over traffic on the master computing node 1011, again based on the data in the storage node 1022.

In taking over traffic from the computing node 1012, when a user requests a modification to data persisted in the data processing system 100, the corresponding binlog may be generated by the computing node 1012 and written to the secondary storage node 1022 or log storage area 501 in the storage cluster 102 so that the subsequent primary computing node 1011 performs data synchronization according to the binlog stored in the storage cluster 102 after failure recovery. After the failure recovery, the master computing node 1011 may act as a disaster recovery slave to the slave computing node 1012; alternatively, the master computing node 1011 may restore the master computing node 1011 to the master node by master-slave switching, and the present embodiment is not limited to this.

In this embodiment, since the binlog generated by the master computing node 1011 is transmitted to the slave computing node 1012 through the storage side, it is possible to avoid the problem of data asynchronization caused by that the load of the master computing node 1011 is excessive or the data transmission link between the master computing node 1011 and the slave computing node 1012 is unstable, and the binlog generated by the master computing node 1011 is not acquired by the slave computing node 1012. Thus, when the master computing node 1011 fails, the data in the slave storage node 1022 remains consistent with the data in the master storage node 1021, which enables the slave computing node 1012 to take over traffic on the master computing node 1011 based on the data in the slave storage node 1022, thereby achieving an RPO of 0 for the data processing system 100 and improving the reliability of the data processing system 100.

In addition, when the master computing node 1011 and the slave computing node 1012 simultaneously include multiple database applications, unified processing logic can be adopted between the master computing node 1011 and the slave computing node 1012 to copy the binlog generated by the master computing node 1011 to the slave computing node 1012, so that the compatibility of the data processing system 100 for the database applications can be improved, and the deployment difficulty of the database applications on the computing cluster 101 can be reduced.

In practical application, the master storage node 1021 and the slave storage node 1022 may be implemented by a storage array (memory array), or may be implemented by a device including the storage array, so that the master storage node 1021 and the slave storage node 1022 may store data based on the storage array in a persistent manner, and may further improve the reliability of the data stored in the master storage node 1021 and the slave storage node 1022 in a persistent manner based on techniques such as an independent disk redundancy array (Redundant Arrays of INDEPENDENT DISKS, RAID) technique, an Erasure Coding (EC) technique, a deduplication compression technique, and a data backup technique.

In the data processing system 100 shown in fig. 1 to 5 described above, the master computing node 1011 and the slave computing node 1012 are respectively provided with respective storage nodes, and the master storage node 1021 is accessible only to the master computing node 1011, and the slave storage node 1022 is accessible only to the slave computing node 1012. In other possible data processing systems, the same storage node may also be shared between the master computing node 1011 and the slave computing nodes 1012, i.e., data on a data page in the storage node is allowed to be accessed by the master computing node 1011 and the slave computing nodes 1012, which is described in detail below in connection with FIG. 6.

Referring to FIG. 6, a schematic diagram of another data processing system provided by the present application is shown. As shown in FIG. 6, data processing system 600 still employs a split-memory architecture, comprising a computing cluster 601 and a storage cluster 602, wherein the computing cluster 601 and the storage cluster 602 may communicate over a network.

The computing cluster 601 includes a plurality of computing nodes, and for convenience of description, the computing cluster 601 in fig. 6 is exemplified by a master computing node 6011 and a slave computing node 6012, and the slave computing node 1012 is a disaster recovery device of the master computing node 1011, which may be specifically a hot standby or a cold standby.

The storage cluster 602 includes at least one storage node, which in fig. 6 is exemplified as including a storage node 6021, and the storage cluster 602 further includes a log storage area 6022. As shown in fig. 6, the log storage area 6022 may be disposed in the storage node 6021, or may be disposed independently of the storage node 6021, such as other storage nodes disposed in the storage cluster 602, which is not limited in this embodiment. Also included on storage node 6021 is a data storage area (not shown in fig. 6) that is accessible to both master computing node 6011 and slave computing node 6012 as well as log storage area 6022. And, the storage node 6021 is configured to persist data using the data storage area, such as data generated when the master computing node 6011 processes traffic. The log storage area 6022 is used to store binlog generated by the master computing node 6011 during operation.

After receiving a data update request for requesting to update (including adding, deleting, changing, etc. of data) data stored persistently in storage node 6021, master computing node 6011 may read a data page where the data requested to be modified by the data update request is located from storage node 6021 to a buffer pool in master computing node 6011, complete modification of the data page in the buffer pool according to the data update request, and generate binlog for the data modification content. Then, the master computing node 6011 may feed back to the user side that the data update was successful, and send the generated binlog to the log storage area 6022 in the storage cluster 602 for storage (and send the updated data page to the storage node 6021 for persistent storage); or the master computing node 6011 may send the generated binlog to the log storage area 6022 in the storage cluster 602 for storage, and then feed back to the user side that the data update is successful, which is not limited in this embodiment.

It is noted that since the master computing node 6011 shares the data page in the storage node 6021 with the slave computing node 6012, after the master computing node 6011 successfully writes the binlog into the log storage area 6022, the slave computing node 6012 may not need to read and play back the binlog in the log storage area 6022 if the master computing node 6011 operates normally. In practical applications, when the master computing node 6011 writes the data in the buffer pool into the storage node 6021, the binlog corresponding to the data in the buffer pool may be eliminated from the log storage area 6022.

The slave computing node 6012 may detect whether the master computing node 6011 has failed in real time or periodically, and when it is determined that the master computing node 6011 has failed, the slave computing node 6012 upgrades to the master node and detects whether binlog exists in the log storage area 6022. If so, it indicates that, when the master computing node 6011 fails, there may be new data in the buffer pool of the master computing node 6011 that is not stored in the storage node 6021 permanently, at this time, the slave computing node 6012 may read the binlog in the log storage area 6022, and update the data in the storage node 6021 by playing back the binlog, so as to restore the data cached in the buffer pool by the master computing node 6011 during the failure, thereby implementing that the data in the data processing system 600 is restored to the state when the master computing node 6011 fails, that is, implementing that the RPO of the data processing system 600 is 0.

Note that, in playing back the binlog in the log storage area 6022 from the computing node 6012, the binlog in the log storage area 6022 may be read to the local storage space of the computing node 6012 first, and then the playback operation may be sequentially performed on the binlogs of each of the local storage spaces in order of LSNs from small to large. Alternatively, the slave computing node 6012 may sequentially read the respective binlogs in the log storage area 6022 in order of LSN from small to large and directly perform the playback operation on the read binlogs. In this manner, the resource consumption required to playback binlog may be further reduced, reducing data recovery latency, thereby reducing the recovery point target (recovery point objective, RTO) of the data processing system 600. RTO refers to the time interval between the beginning of a data processing system 600 service pause and the ending of the data processing system 600 service recovery after a disaster.

In the data processing systems shown in fig. 1 to 6, the operations performed by the master computing node and the slave computing nodes may be implemented by an application deployed on the master computing node, and the application may be, for example, a database application such as MySQL, postgreSQL, openGauss or Oracle, or may be other applications.

In this way, the version update of the existing application deployed on the master computing node and the slave computing node can be implemented, so that the binlog is transmitted from the master computing node to the slave computing node through the storage side, and the data synchronization between the master center and the disaster recovery center is implemented.

Or the operations performed by the master computing node and the slave computing nodes may also be performed by data processing apparatuses separately deployed in the computing cluster, i.e. the master computing node may write the generated binlog into the storage cluster under the control of the data processing apparatus, and the slave computing node may read the binlog from the storage cluster and play back the binlog under the control of the data processing apparatus.

The data processing means may be implemented by software or hardware, for example.

When implemented in software, the data processing apparatus may be, for example, program code or the like disposed on a hardware device. In actual use, the data processing apparatus may be, for example, deployed as software in a plug-in, component, or application to a master computing node and/or a slave computing node (e.g., in a controller of the master computing node and/or the slave computing node). At this time, by deploying the data processing device on the master computing node and/or the slave computing node, the transmission of the binlog between the master computing node and the slave computing node through the storage side can be achieved, so that the difficulty of implementing the scheme can be reduced or no modification is required to database applications deployed on the master computing node and the slave computing node.

Alternatively, the data processing apparatus may be implemented by a physical device, where the physical device may be, for example, a CPU, or may be any one of or any combination of an application-specific integrated circuit (ASIC), a programmable logic device (programmable logic device, PLD), a complex program logic device (complex programmable logical device, CPLD), a field-programmable GATE ARRAY (FPGA), a generic array logic (GENERIC ARRAY logic, GAL), a system on chip (SoC), a software defined architecture (software-defined infrastructure, SDI) chip, an artificial intelligence (ARTIFICIAL INTELLIGENCE, AI) chip, a data processing unit (data processing unit, DPU), or the like, which is not limited in this embodiment.

The above description, in connection with fig. 1 to 6, describes a process for implementing data synchronization between a primary center and a disaster recovery center in a data processing system by transmitting binlog on a storage side. In the following, with reference to fig. 7, a flow of implementing data synchronization between a primary center and a disaster recovery center is exemplarily described from the viewpoint of a method flow. Referring to fig. 7, a flowchart of a data processing method according to an embodiment of the present application is shown. For ease of understanding, fig. 7 is an illustration of an example application to data processing system 100 shown in fig. 1, and as shown in fig. 7, the method may specifically include:

s701: master computing node 1011 receives a data update request requesting data processing system 100 to update the persistently stored data.

In practice, master computing node 1011 may receive data update requests sent by clients or other devices on the user side, which may be used to request modifications to data persisted in data processing system 100, or may be used to request writing new data to data processing system 100, etc.

S702: in response to the data update request, the master computing node 1011 completes the data update in the buffer pool and generates a corresponding binlog for the data update request.

S703: master computing node 1011 sends binlog to storage cluster 102 for storage.

In a first implementation example, the storage cluster 102 includes a master storage node 1021 and a slave storage node 1022, where the master storage node 1021 supports data read and write by the master computing node 1011 and the slave storage node 1022 supports data read and write by the slave computing node 1012. In this way, the master computing node 1011 may write the binlog to the master storage node 1021, back up the binlog by the master storage node 1021, and then send the backed up binlog to the slave storage node 1022 for storage. The master storage node 1021 and the slave storage node 1022 may be disposed in the same physical area (e.g., a data center/AZ), or may be disposed in different physical areas.

In a second implementation example, a log storage area, such as log storage area 501 described above, may be deployed in storage cluster 102 such that master computing node 1011 may write the generated binlog to the log storage area, which is accessible to master computing node 1011 as well as slave computing node 1012.

S704: the slave computing node 1012 reads binlog from the storage cluster 102.

In particular implementations, the slave computing node 1012 may read binlog, etc., from a slave storage node 1022 or log storage area in the storage cluster 102.

Wherein the slave computing node 1012 may have the same configuration as the master computing node 1011. For example, in creating the slave computing node 1012, the master computing node 1011 may back up its own configuration files and send the backed up data to the slave computing node 1012, so that the slave computing node 1012 completes the corresponding configuration according to the received backup data, such as configuring the logic of the processing service, the application running on the slave computing node 1012, the directory on which the binlog in the storage cluster 102 is mounted, and so on.

S705: the read binlog is played back from the computing node 1012 to update the data in the storage cluster 102 that was persisted from the storage node 1022.

In the data processing system 100 shown in fig. 1, the slave computing node 1012 may specifically read the binlog from the storage node 1022 or log storage area, and by playing back the binlog, it is achieved that the data in the slave storage node 1022 is kept in a synchronized state with the data in the master center. In particular, the computing node 1012 may parse a database statement, such as an SQL statement, from the binlog, and perform semantic analysis and syntax analysis on the database statement to determine validity of the database statement. After passing the validity check, the slave computing node 1012 may generate a planning tree for the database statement, the planning tree indicating an execution plan for processing the data. Finally, the slave computing node 1012 may perform data updates in the slave storage node 1022 based on the optimized planning tree after optimization of the planning tree is completed.

Thus, after a failure of the master computing node 1011, the slave computing node 1012 may take over traffic on the master computing node 1011 by maintaining data in a synchronized state with the master hub from the slave storage node 1022, thereby achieving a failback of the data processing system 100.

Before the slave computing node 1012 plays back binlog, the master computing node 1011 may control the master storage node 1021 to send the baseline data to the slave storage node 1022 to complete the baseline replication, and the specific implementation process of the baseline replication may be referred to the description of the relevant point of the foregoing embodiment, which is not described herein.

In other possible data processing systems, such as data processing system 600 shown in FIG. 6, the master computing node 1011 and the slave computing nodes 1012 may share the same storage node, hereinafter referred to as the target storage node. Then, during normal operation of the master computing node 1011, the slave computing node 1012 may not perform the operations of reading binlog from the storage cluster 102 and playing back binlog for the binlog written to the storage cluster 102 by the master computing node 1011. At this time, the binlog stored in the storage cluster 102 is the binlog corresponding to the updated data stored in the buffer pool of the master computing node 1011; when data in a buffer pool is written to storage cluster 102, the binlog corresponding to the data in the buffer pool may be eliminated from storage cluster 102. When the primary computing node 1011 fails, the data cached in the buffer pool of the primary computing node 1011 may be lost due to the failure of the primary computing node 1011, and at this time, the secondary computing node 1012 updates the data stored in the target storage node by reading binlog from the storage cluster 102 and performing the operation of playing back binlog, thereby recovering the data stored in the target storage node 1011 in the buffer pool which has not been stored in a persistent manner.

It should be noted that, the steps S701 to S705 shown in fig. 7 correspond to the system embodiments shown in fig. 1 to 6, and therefore, the specific implementation procedure of the steps S701 to S705 may be referred to the relevant points of the foregoing embodiments, which are not repeated here.

The data processing system provided by the present application is described in detail above with reference to fig. 1 to 7, and the data processing apparatus and the data processing device provided according to the present application will be described below with reference to fig. 8 and 9.

The application concept similar to the method is that the embodiment of the application also provides a data processing device. Referring to fig. 8, a schematic diagram of a data processing apparatus according to an embodiment of the present application is shown. The data processing apparatus 800 shown in fig. 8 is located in a data processing system, such as the data processing system 100 shown in fig. 1, the data processing system 600 shown in fig. 6, and the like, where the data processing system includes a computing cluster, and a storage cluster, where the computing cluster is connected to the storage cluster through a network, and the computing cluster includes a master computing node and a slave computing node, and the storage cluster includes at least one storage node, and in general, the slave computing node is a disaster recovery device of the master computing node.

As shown in fig. 8, the data processing apparatus 800 includes:

A storage module 801, configured to instruct a master computing node to send binary log binlog generated in response to a data update request to a storage cluster for storage;

a reading module 802, configured to instruct a slave computing node to read the binlog stored in the storage cluster;

A playback module 803 for instructing the slave computing node to update the data persistently stored in the storage cluster by playing back binlog.

In one possible implementation, a storage cluster includes a log storage area that is accessed by a master computing node and slave computing nodes;

The storage module 801 is specifically configured to instruct the main computing node to send the binlog to the log storage area for storage;

A reading module 802, specifically configured to instruct the slave computing node to read binlog from the log storage area to the slave storage node; the storage cluster further comprises a data storage area, wherein the data storage area is used for storing service data, and the data storage area is accessed by the master computing node and the slave computing nodes or is accessed only by the master computing node.

In one possible implementation, the revisit module 803 is specifically configured to instruct the slave computing node to synchronize data with the master computing node by replaying binlog, or specifically configured to instruct the slave computing node to recover data lost when the master computing node fails by replaying binlog.

In one possible implementation, the at least one storage node includes a master storage node and a slave storage node, where the slave storage node is used as a disaster recovery device for the master storage node, and the master storage node and the slave storage node are deployed in the same data center or the same available area;

The reading module 802 is specifically configured to instruct the slave computing node to read binlog from the log storage area to the slave computing node during normal operation of the master computing node;

A playback module 803 is specifically configured to instruct the slave computing node to update the data persistently stored in the slave storage node by playing back binlog.

In one possible implementation, the at least one storage node comprises a target storage node for persisting data written by the master computing node;

A reading module 802, specifically configured to instruct the slave computing node to read binlog from the log storage area when the master computing node fails;

a playback module 803 is specifically configured to instruct the slave computing node to update the data stored persistently in the target storage node by playing back binlog.

In one possible implementation, the at least one storage node includes a master storage node and a slave storage node, where the slave storage node is used as a disaster recovery device for the master storage node, and the master storage node and the slave storage node are disposed in different data centers, or the master storage node and the slave storage node are disposed in different available areas;

the storage module 801 is specifically configured to instruct the master computing node to send binlog to the master storage node for storage;

The reading module 802 is specifically configured to instruct the slave computing node to read the binlog stored in the slave storage node, where the binlog is sent by the master storage node.

In one possible implementation, the storage module 801 is further configured to: instruct the master storage node to send the baseline data to the slave storage node for storage before sending the binlog to the slave storage node;

a playback module 803 is specifically configured to instruct the slave computing node to update the baseline data stored in the slave storage node by playing back binlog.

The data processing apparatus 800 provided in this embodiment corresponds to the data processing system in each embodiment, and is configured to implement the data processing process performed in each embodiment, so the functions of each module and the technical effects thereof in this embodiment can be described in the related parts in the foregoing embodiments, and are not described herein.

In addition, an embodiment of the present application further provides a computing device, as shown in fig. 9, where a computing device 900 may include a communication interface 910 and a processor 920. Optionally, a memory 930 may also be included in computing device 900. The memory 930 may be disposed within the computing device 900 or external to the computing device 900. Illustratively, the various actions performed by the data processing apparatus in the above embodiments, which instruct the master computing node, the slave computing node (and the master storage node), may all be implemented by the processor 920. In implementation, each step of the process flow may implement the method in the foregoing embodiment through an integrated logic circuit of hardware in the processor 920 or an instruction in a software form. For brevity, the description is omitted here. Program code executed by processor 920 for implementing the above-described methods may be stored in memory 930. The memory 930 is coupled to the processor 920 such as by a coupling connection or the like.

Some features of embodiments of the present application may be implemented/supported by processor 920 executing program instructions or software code in memory 930. The software components loaded on the memory 930 may be summarized functionally or logically, for example, the storage module 801, the playback module 803 shown in fig. 8, and the functions of the reading module 802 shown in fig. 8 may be implemented by the communication interface 910.

Any communication interface referred to in the embodiments of the present application may be a circuit, a bus, a transceiver, or any other device that may be used to perform information interaction. Such as communication interface 910 in computing device 900, which may be, for example, a device connected to computing device 900, etc.

Embodiments of the present application further provide a computing device cluster, where the computing device cluster may include one or more computing devices, each of which may have a hardware structure of the computing device 900 shown in fig. 9, and where the computing device cluster, in operation, may be used to implement the data processing method in the embodiment shown in fig. 7.

Based on the above embodiments, the present application further provides a chip, which includes a power supply circuit and a processing circuit, where the power supply circuit is configured to supply power to the processing circuit, and the processing circuit is configured to implement the functions of the data processing apparatus 800 shown in fig. 8.

The processors referred to in the embodiments of the present application may be general purpose processors, digital signal processors, application specific integrated circuits, field programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and may implement or perform the methods, steps, and logic blocks disclosed in the embodiments of the present application. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in the processor for execution.

The coupling in the embodiments of the present application is an indirect coupling or communication connection between devices, modules or modules, which may be in electrical, mechanical or other form for information interaction between the devices, modules or modules.

The processor may operate in conjunction with the memory. The memory may be a nonvolatile memory such as a hard disk or a solid state disk, or may be a volatile memory such as a random access memory. The memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such.

The embodiment of the application is not limited to the specific connection medium among the communication interface, the processor and the memory. Such as memory, processor, and communication interfaces, may be connected by a bus. The buses may be classified as address buses, data buses, control buses, etc.

Based on the above embodiments, the present application also provides a computer storage medium, where a software program is stored, where the software program may implement the method performed by the data processing apparatus 102 provided in any one or more of the above embodiments when read and executed by one or more computing devices. The computer storage medium may include: various media capable of storing program codes, such as a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The terms first, second and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and are merely illustrative of the manner in which embodiments of the application have been described in connection with the description of the objects having the same attributes.

It will be apparent to those skilled in the art that various modifications and variations can be made to the embodiments of the present application without departing from the scope of the embodiments of the application. Thus, if such modifications and variations of the embodiments of the present application fall within the scope of the claims and the equivalents thereof, the present application is also intended to include such modifications and variations.

Claims

1. A data processing system, wherein the data processing system comprises a computing cluster and a storage cluster, the computing cluster is connected with the storage cluster through a network, the computing cluster comprises a master computing node and a slave computing node, and the storage cluster comprises at least one storage node;

The main computing node is used for responding to a data updating request to generate a binary log binlog and sending the binlog to the storage cluster for storage;

the slave computing node is configured to read the binlog stored in the storage cluster, and update data stored persistently in the storage cluster by playing back the binlog.

2. The data processing system of claim 1, wherein the storage cluster comprises a log storage area, the log storage area being accessed by the master computing node and the slave computing nodes;

The main computing node is specifically configured to send the binlog to the log storage area for storage;

The slave computing node is specifically configured to read the binlog from the log storage area;

The storage cluster further comprises a data storage area, wherein the data storage area is used for storing service data, and the data storage area is accessed by the master computing node and the slave computing nodes or is only accessed by the master computing node.

3. The data processing system according to claim 2, wherein the slave computing node is in particular configured to synchronize data with a master computing node by playing back the binlog or to recover data lost upon failure of the master computing node by playing back the binlog.

4. A data processing system according to claim 3, wherein the at least one storage node comprises a master storage node and a slave storage node, the slave storage node acting as a disaster recovery device for the master storage node, the master storage node and the slave storage node being deployed in the same data center or in the same availability zone;

The slave computing node is specifically configured to read the binlog from the log storage area during a normal operation process of the master computing node, and update data stored in the slave storage node in a persistent manner by playing back the binlog.

5. A data processing system according to claim 3, wherein the at least one storage node comprises a target storage node for persisting data written by the master computing node;

The slave computing node is specifically configured to, when the master computing node fails, read the binlog from the log storage area, and update the data stored in the target storage node in a persistent manner by playing back the binlo.

6. The data processing system of claim 1, wherein the at least one storage node comprises a master storage node and a slave storage node, the slave storage node acting as a disaster recovery device for the master storage node, the master storage node being disposed in a different data center than the slave storage node or the master storage node being disposed in a different availability zone than the slave storage node;

The main computing node is specifically configured to send the binlog to the main storage node for storage;

the master storage node is configured to send the binlog to the slave storage node;

the slave computing node is specifically configured to read the binlog stored by the slave storage node.

7. The data processing system of claim 6, wherein the data processing system further comprises a data processing system,

The master storage node is further configured to send baseline data to the slave storage node before sending the binlog to the slave storage node;

The slave storage node is used for storing the baseline data;

the slave computing node is specifically configured to update the baseline data stored in the slave storage node by playing back the binlog.

8. The data processing system of any of claims 1 to 7, wherein the host computing node has a target application running thereon, the binlog being generated during the running of the target application, the target application comprising a relational database management system, RDBMS, the RDBMS comprising at least one of MySQL, postgreSQL, openGauss, oracle.

9. A data processing system according to any one of claims 1 to 8, wherein the storage nodes are storage arrays for persisting data.

10. A data processing method, characterized in that the method is applied to a data processing system, the data processing system comprises a computing cluster and a storage cluster, the computing cluster is connected with the storage cluster through a network, the computing cluster comprises a master computing node and a slave computing node, and the storage cluster comprises at least one storage node; the method comprises the following steps:

The master computing node generating a binary log binlog in response to a data update request;

the main computing node sends the binlog to the storage cluster for storage;

Reading the binlog stored in the storage cluster from a computing node;

The slave computing node updates the persistently stored data in the storage cluster by playing back the binlog.

11. The data processing method of claim 10, wherein the storage cluster includes a log storage area, the log storage area being accessed by the master computing node and the slave computing nodes;

the master computing node sends the binlog to the storage cluster for storage, including:

The main computing node sends the binlog to the log storage area for storage;

the reading the binlog stored in the storage cluster from a computing node includes:

The slave computing node reading the binlog from the log storage area;

12. The data processing method of claim 11, wherein the slave computing node, by playing back the binlog, comprises:

the slave computing node synchronizes data with the master computing node by playing back the binlog, or is used to recover data lost when the master computing node fails by playing back the binlog.

13. The data processing method according to claim 12, wherein the at least one storage node includes a master storage node and a slave storage node, the slave storage node being a disaster recovery device for the master storage node, the master storage node and the slave storage node being disposed in a same data center or a same availability area;

The slave computing node reads the binlog from the log storage area in the normal operation process of the master computing node;

The slave computing node updating the data persistently stored in the storage cluster by playing back the binlog, comprising:

The slave computing node updates the data persistently stored in the slave storage node by playing back the binlog.

14. The data processing method of claim 12, wherein the at least one storage node comprises a target storage node for persisting data written by the master computing node;

The slave computing node reads the binlog from the log storage area when the master computing node fails;

the slave computing node updates the data persistently stored in the target storage node by playing back the binlog.

15. The data processing method according to claim 10, wherein the at least one storage node includes a master storage node and a slave storage node, the slave storage node being a disaster recovery device for the master storage node, the master storage node and the slave storage node being disposed in different data centers or the master storage node and the slave storage node being disposed in different available areas;

The main computing node sends the binlog to the main storage node for storage;

The slave computing node is specifically configured to read the binlog stored by the slave storage node, where the binlog is sent by the master storage node.

16. The data processing method of claim 15, wherein the method further comprises:

The master storage node sends baseline data to the slave storage node for storage before sending the binlog to the slave storage node;

the slave computing node updates the baseline data stored in the slave storage node by playing back the binlog.

17. A data processing method according to any of claims 10 to 16, wherein a target application is running on the host computing node, the binlog being generated during the running of the target application, the target application comprising a relational database management system, RDBMS, the RDBMS comprising at least one of MySQL, postgreSQL, openGauss, oracle.

18. A data processing method according to any one of claims 10 to 17, wherein the storage nodes are storage arrays for persisting data.

19. A data processing device, wherein the data processing device is applied to a data processing system, the data processing system comprises a computing cluster and a storage cluster, the computing cluster is connected with the storage cluster through a network, the computing cluster comprises a master computing node and a slave computing node, and the storage cluster comprises at least one storage node; the data processing apparatus includes:

The storage module is used for indicating the main computing node to send binary log binlog generated in response to a data update request to the storage cluster for storage;

A reading module, configured to instruct the slave computing node to read the binlog stored in the storage cluster to the slave computing node;

And a playback module for instructing the slave computing node to update the data persistently stored in the storage cluster by playing back the binlog.

20. The data processing apparatus of claim 19, wherein the storage cluster comprises a log storage area, the log storage area being accessed by the master computing node and the slave computing nodes;

The storage module is specifically configured to instruct the master computing node to send the binlog to the log storage area for storage;

the reading module is specifically configured to instruct the slave computing node to read the binlog from the log storage area to the slave storage node;

21. The data processing apparatus according to claim 20, wherein the return module is specifically configured to instruct the slave computing node to synchronize data with a master computing node by playing back the binlog, or specifically configured to instruct the slave computing node to recover data lost when the master computing node fails by playing back the binlog.

22. The data processing apparatus of claim 21, wherein the at least one storage node comprises a master storage node and a slave storage node, the slave storage node acting as a disaster recovery device for the master storage node, the master storage node and the slave storage node being deployed in a same data center or a same availability zone;

The reading module is specifically configured to instruct the slave computing node to read the binlog from the log storage area to the slave computing node in a normal operation process of the master computing node;

The playback module is specifically configured to instruct the slave computing node to update the data stored in the slave storage node in a persistent manner by playing back the binlog.

23. The data processing apparatus of claim 21, wherein the at least one storage node comprises a target storage node for persisting data written by the master computing node;

The reading module is specifically configured to instruct the slave computing node to read the binlog from the log storage area when the master computing node fails;

The playback module is specifically configured to instruct the slave computing node to update the data stored in the target storage node in a persistent manner by playing back the binlog.

24. The data processing apparatus of claim 19, wherein the at least one storage node comprises a master storage node and a slave storage node, the slave storage node acting as a disaster recovery device for the master storage node, the master storage node being disposed in a different data center than the slave storage node or the master storage node being disposed in a different availability zone than the slave storage node;

The storage module is specifically configured to instruct the master computing node to send the binlog to the master storage node for storage;

The reading module is specifically configured to instruct the slave computing node to read the binlog stored by the slave storage node, where the binlog is sent by the master storage node.

25. The data processing apparatus of claim 24, wherein the memory module is further configured to:

Instruct the master storage node to send baseline data to the slave storage node for storage before sending the binlog to the slave storage node;

the playback module is specifically configured to instruct the slave computing node to update the baseline data stored in the slave storage node by playing back the binlog.

26. The data processing apparatus of any of claims 19 to 25, wherein a target application is running on the master computing node, the binlog being generated during the running of the target application, the target application comprising a relational database management system, RDBMS, the RDBMS comprising at least one of MySQL, postgreSQL, openGauss, oracle.

27. A data processing apparatus according to any one of claims 19 to 26, wherein the storage nodes are storage arrays for persisting data.

28. A cluster of computing devices, wherein the cluster of computing devices comprises at least one computing device, each of the at least one computing device comprising a processor and a memory;

The processor is configured to execute instructions stored in the memory to cause the cluster of computing devices to perform the method of any one of claims 10 to 18.

29. A computer readable storage medium comprising instructions for implementing the method of any one of claims 10 to 18.

30. A computer program product containing instructions which, when run on a computer, cause the computer to perform the method of any of claims 10 to 18.