WO2021174537A1

WO2021174537A1 - Data transmission method and apparatus

Info

Publication number: WO2021174537A1
Application number: PCT/CN2020/078243
Authority: WO
Inventors: 劳志毅
Original assignee: 深圳市欢太科技有限公司; Oppo广东移动通信有限公司
Priority date: 2020-03-06
Filing date: 2020-03-06
Publication date: 2021-09-10
Also published as: CN115210694A

Abstract

Disclosed are a data transmission method and apparatus, which are applied to a subscriber device in a data transmission system. The data transmission system further comprises a source device, a consumer device and a target device. The method comprises: acquiring, from a source device, a data set to be synchronized; and transmitting said data set to a consumer device by means of two-phase commit, and instructing the consumer device to hash said data set into multiple files of a WAL and transmitting the multiple files to a target device. Therefore, a data set to be synchronized is transmitted to a consumer device by means of two-phase commit, thereby ensuring the successful transmission of the data set to be synchronized. Moreover, a data set to be synchronized is transmitted to multiple files of a WAL, thereby improving the synchronous transmission efficiency of data.

Description

Data transmission method and device

Technical field

This application relates to the field of electronic technology, in particular to a data transmission method and device.

Background technique

The existing data synchronization technology relies on storage services, such as the structured query language (my structured query language, mysql) itself to ensure the orderliness of the data synchronization process. MySQL is a relational database management system. The database stores data in different tables. The structured query language (SQL) used by mysql is the most commonly used standardized language for accessing the database. In the synchronization process, the order of data synchronization is achieved by giving a globally unique order mark.

However, due to the excessive reliance on the characteristics of the storage service itself, the data transmission process relies on the globally unique order mark, so that the input and output of the data are synchronized in a single thread, which affects the efficiency of the data transmission process.

Summary of the invention

The embodiments of the present application provide a data transmission method and device, which can ensure that data is not lost during the synchronization transmission of data and improve the efficiency of data transmission.

In the first aspect, an embodiment of the present application provides a data transmission method, which is applied to a subscriber device in a data transmission system, the data transmission system further includes a source device, a consumer device, and a target device, and the method includes:

Acquiring a data set to be synchronized from the source device;

The data set to be synchronized is transmitted to the consumer device in a two-phase commit method, and the consumer device is instructed to hash the data set to be synchronized into multiple files in the WAL, and the multiple A file is transferred to the target device.

In a second aspect, an embodiment of the present application provides a data transmission method, which is applied to a consumer device in a data transmission system. The data transmission system further includes a source device, a subscriber device, and a target device. The method includes:

Receiving the to-be-synchronized data set transmitted by the subscribing end device in a two-phase commit method, where the to-be-synchronized data set is data obtained by the subscribing end device from the source end device;

Hash the to-be-synchronized data set into multiple files of WAL;

Transferring the multiple files to the target device.

In a third aspect, an embodiment of the present application provides a data transmission device, which is applied to a subscribing device in a data transmission system. The data transmission system further includes a source device, a consumer device, and a target device. The data transmission The device includes:

The first obtaining unit is configured to obtain a data set to be synchronized from the source device;

The first transmission unit is configured to transmit the to-be-synchronized data set to the consumer device in a two-phase commit method, and instruct the consumer-end device to hash the to-be-synchronized data set to multiple files of WAL And transfer the multiple files to the target device.

In a fourth aspect, an embodiment of the present application provides a data transmission device, which is applied to a consumer device in a data transmission system. The data transmission system further includes a source device, a subscriber device, and a target device. The data transmission The device includes:

A receiving unit, configured to receive a data set to be synchronized transmitted by the subscribing end device in a two-phase commit manner, where the data set to be synchronized is data obtained by the subscribing end device from the source device;

The hash unit is used to hash the data set to be synchronized into multiple files of the WAL;

The second transmission unit is configured to transmit the multiple files to the target device.

In a fifth aspect, an embodiment of the present application provides a subscription terminal device, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured by The foregoing processor executes, and the foregoing program includes instructions for executing the steps in the first aspect of the embodiments of the present application.

In a sixth aspect, embodiments of the present application provide a consumer device, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured by The foregoing processor executes, and the foregoing program includes instructions for executing the steps in the first aspect of the embodiments of the present application.

In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the above-mentioned computer-readable storage medium stores a computer program for electronic data exchange, wherein the above-mentioned computer program enables a computer to execute Some or all of the steps described in one aspect.

In an eighth aspect, the embodiments of the present application provide a computer program product, wherein the above-mentioned computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the above-mentioned computer program is operable to cause a computer to execute as implemented in this application. Example part or all of the steps described in the first aspect. The computer program product may be a software installation package.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1A is a schematic structural diagram of a data transmission system for implementing a data transmission method according to an embodiment of the present application;

FIG. 1B is a schematic flowchart of a data transmission method disclosed in an embodiment of the present application;

FIG. 2 is a schematic flowchart of another data transmission method disclosed in an embodiment of the present application;

FIG. 3 is a schematic flowchart of another data transmission method disclosed in an embodiment of the present application;

Fig. 4A is a schematic structural diagram of another subscriber device disclosed in an embodiment of the present application;

4B is a schematic structural diagram of another consumer device disclosed in an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a data transmission device disclosed in an embodiment of the present application;

6A is a schematic structural diagram of another data transmission device disclosed in an embodiment of the present application;

FIG. 6B is a modified structure of the data transmission device shown in FIG. 6A disclosed in an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the application, the technical solutions in the embodiments of the application will be clearly and completely described below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by a person of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific sequence. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes unlisted steps or units, or optionally also includes Other steps or units inherent to these processes, methods, products or equipment.

The reference to "embodiments" herein means that a specific feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art clearly and implicitly understand that the embodiments described herein can be combined with other embodiments.

In order to facilitate a better understanding of the technical solutions described in the present invention, the technical terms involved in the embodiments of the present invention are explained below:

mysql: is a relational database management system. Relational databases store data in different tables.

Two-phase commit (2PC): An algorithm for maintaining consistency when committing transactions to all nodes based on a distributed system architecture.

Write-Ahead Logging (WAL): An efficient logging algorithm. For non-memory databases, disk I/O operations are a major bottleneck for database efficiency. Under the same amount of data, when a transaction is committed in a database system using WAL, disk write operations are only about half of the traditional rollback log, which greatly improves the efficiency of database disk I/O operations, thereby improving database performance.

The following describes the embodiments of the application in detail.

The data transmission method provided in the embodiment of the present application is suitable for the scenario of synchronous data transmission across computer rooms. Please refer to FIG. 1A, which is a schematic structural diagram of a data transmission system for implementing a data transmission method according to an embodiment of the present invention. The data transmission system may include a source device 1001, a subscriber device 1002, a consumer device 1003, and a target device 1004. The source device is the device of the data publisher, the subscriber device is the device of the data subscriber, and the consumer device is The device is the device of the data consumer, the target device is the device of the data synchronization storage side, the source device is connected to the subscribing device, the subscribing device is also connected to the consumer device, and the consumer device is also connected to the target device. The end device 1001, the subscribing end device 1002, the consumer end device 1003, and the target end device 1004 respectively correspond to different computer rooms. When the source device has a set of data to be synchronized that needs to be transmitted synchronously, the remote device can transmit the data to be synchronized to the subscribing device, and the subscribing device will transmit the data to be synchronized to the consumer device in a two-phase commit method. , And then, the consumer device hashes the set of data to be synchronized into multiple files in the WAL, and transmits the multiple files to the target device, thereby realizing synchronized data transmission across computer rooms and ensuring multiple locations Data can be synchronized between different computer rooms. Wherein, the to-be-synchronized data set may include at least one to-be-synchronized data.

Please refer to FIG. 1B. FIG. 1B is a schematic flowchart of a data transmission method disclosed in an embodiment of the present application. Terminal device, the data transmission method includes the following steps:

101. Obtain a data set to be synchronized from the source device.

In the embodiment of the present application, when there is data to be synchronized that needs to be synchronously transmitted in the remote device, the remote device can transmit the to-be-synchronized data set including at least one data to be synchronized to the subscribing device, so that the subscribing device can download from the source The device obtains the data collection to be synchronized.

Optionally, in the above step 101, the obtaining the data set to be synchronized from the source device may include the following steps:

Acquire the to-be-synchronized data set transmitted by the source device through a single thread.

Wherein, during data synchronization transmission, each to-be-synchronized data in the to-be-synchronized data set carries a sequence identifier used to characterize the transmission sequence, so that the sequence identifier corresponding to each to-be-synchronized data in the to-be-synchronized data set can be identified Obtain each data to be synchronized transmitted by the source device in an orderly manner through a single thread. Specifically, the data set to be synchronized transmitted by the source device can be obtained through a single thread, and the data to be synchronized can be obtained in a time sequence represented by the sequence identifier.

102. Transmit the to-be-synchronized data set to the consumer device in a two-phase commit method, and instruct the consumer-end device to hash the to-be-synchronized data set into multiple files in the WAL, and combine all The multiple files are transferred to the target device.

Among them, both the subscribing device and the consumer device can determine whether their operations are successful, but they cannot determine whether the operations of other devices are successful. Through the two-phase submission method, the subscribing device can send to the consumer device in the first stage whether it is OK Execute the query message for sending the data set to be synchronized. If the subscribing device receives the feedback message from the consumer device within the preset time, it enters the second stage. In the second stage, the subscribing device transmits the data set to be synchronized to the consumer If the subscribing device does not receive a feedback message from the consumer device within a preset time, the subscribing device sends a rollback message to the consumer device, thereby ensuring that the data set to be synchronized is successfully transmitted to Consumer equipment. The subscribing device can instruct the consumer device to hash the data set to be synchronized into multiple files in the WAL, and transfer the multiple files to the target device, so that multiple files can be transferred to the WAL The way of data collection to be synchronized improves the efficiency of synchronized data transmission.

It can be seen that the data transmission method described in the embodiments of this application obtains the data set to be synchronized from the source device; transmits the data set to be synchronized to the consumer device in a two-phase commit method, and instructs the consumer device to The data set to be synchronized is hashed to multiple files in the WAL, and the multiple files are transferred to the target device. In this way, the data set to be synchronized is transmitted to the consumer device through a two-phase commit method to ensure the data to be synchronized The collection is successfully transmitted, and the data collection to be synchronized is transmitted through multiple files of WAL, thereby improving the efficiency of data synchronization transmission.

Consistent with the above, FIG. 2 is a schematic flowchart of a data transmission method disclosed in an embodiment of the present application. The data transmission method is applied to a consumer device in a data transmission system. The data transmission system further includes a source device, a subscriber device, and a target device. The data transmission method includes the following steps:

201. Receive a data set to be synchronized transmitted by the subscribing end device in a two-phase commit method, where the data set to be synchronized is data obtained by the subscribing end device from the source device.

Among them, both the subscribing device and the consumer device can determine whether their operations are successful, but they cannot determine whether the operations of other devices are successful. Through the two-phase submission method, the subscribing device can send to the consumer device in the first stage whether it is OK Execute the query message for sending the data set to be synchronized. If the subscribing device receives the feedback message from the consumer device within the preset time, it enters the second stage. In the second stage, the subscribing device transmits the data set to be synchronized to the consumer If the subscribing device does not receive the feedback message from the consumer device within a preset time, the subscribing device sends a rollback message to the consumer device, thereby ensuring that the consumer device successfully receives the pending synchronization Data collection.

202. Hash the to-be-synchronized data set into multiple files in the WAL.

Among them, WAL can improve the efficiency of non-memory data and ensure data integrity. Therefore, by using WAL to store the to-be-synchronized data in the to-be-synchronized data set, hash the to-be-synchronized data set into multiple files in the WAL, and then transfer the multiple files in the WAL to the target device, the synchronization data transmission can be improved efficient. Specifically, the data in the data set to be synchronized can be written into multiple files through multiple data transmission channels, so that the disk input and output usage rate can be increased, and the file writing rate can be increased.

Optionally, the to-be-synchronized data set includes multiple to-be-synchronized data, and each of the to-be-synchronized data corresponds to one piece of routing information. In the above step 202, the to-be-synchronized data set is hashed into multiple files of WAL , Can include the following steps:

21. Divide the multiple data to be synchronized into multiple groups according to multiple routing information corresponding to the multiple data to be synchronized, and the data to be synchronized in each of the groups corresponds to the same routing information;

22. Write the to-be-synchronized data of each group in the plurality of groups into the file corresponding to the group in the WAL to obtain a plurality of files.

Wherein, the above-mentioned routing information may be the device identification of the routing device corresponding to the data to be synchronized. According to the routing information of the data to be synchronized, the data to be synchronized with the same routing information can be divided into one group, and then the to-be-synchronized data of each group The data is written into a file corresponding to the group, and further, the to-be-synchronized data of multiple groups can be written into multiple files, and the multiple groups correspond to the multiple files one-to-one. In this way, the number of multiple files can be dynamically adjusted according to the routing information, and the number of files is different, the transmission rate of the data to be synchronized is also different, so that the transmission rate of the data set to be synchronized can be dynamically adjusted.

Optionally, in the above step 22, writing the to-be-synchronized data of each of the multiple groups into the file corresponding to the group in the WAL to obtain multiple files may include the following steps:

2201. If the number of data types of all to-be-synchronized data in group i is greater than a preset value, divide all the to-be-synchronized data in group i into multiple sub-groups, and the to-be-synchronized data in each sub-group The data corresponds to the same data type, where the group i is any group of the plurality of groups;

2202. Write the to-be-synchronized data of each sub-group of the multiple sub-groups of the group i into the file corresponding to the sub-group in the WAL to obtain multiple files.

Among them, for any group i among multiple groups, if the number of data types of all the data to be synchronized in the group i is greater than the preset value, it indicates that the number of data types in group i is larger, and the number of data in group i Furthermore, all the data to be synchronized in the group i can be divided into multiple sub-groups according to the data type. Specifically, the data to be synchronized with the same data type in the group i are divided into the same sub-group, and further, each The to-be-synchronized data of a sub-group is written into the file corresponding to the sub-group, so that the to-be-synchronized data of the same sub-group can be written into one file. In this way, the number of multiple files can be dynamically adjusted according to the data type. When there are more data types in the group i, the more the number of corresponding files, the different transmission rate of the data to be synchronized. Therefore, the synchronization can be adjusted dynamically. The transmission rate of the data collection.

Optionally, the group i includes m pieces of data to be synchronized. In the embodiment of the present application, the following steps may be further included:

Acquiring the type identifiers of the m to-be-synchronized data in the group i to obtain m type identifiers;

The m type identifiers are counted to obtain the number n of data types of all data to be synchronized in the group i, where n is less than or equal to m.

Wherein, before step 2201, the number n of data types of all data to be synchronized in group i can also be determined. Specifically, the type identifiers of m data to be synchronized in group i can be obtained, and m type identifiers can be obtained. Synchronized data corresponds to a type identifier, and different data to be synchronized may correspond to the same type identifier, or may correspond to different type identifiers, and further, the number n of type identifiers can be counted.

Optionally, in the above step 2202, the data to be synchronized of each sub-group of the multiple sub-groups of the group i is written into the file corresponding to the sub-group in the WAL to obtain multiple files, which may include the following step:

In the manner of writing files in sequence, the to-be-synchronized data of each sub-group of the multiple sub-groups of the group i is written into the file in the WAL to obtain multiple files.

In the embodiment of the present application, when the data to be synchronized is written into the file, the file is written sequentially, which can avoid a large amount of disk addressing time, thereby increasing the file writing rate.

203. Transmit the multiple files to the target device.

The consumer device can transfer multiple files to the target device, and through a two-phase submission, it is ensured that the data set to be synchronized at the data source end is consistent with the data transferred to the target device, thereby ensuring that the data set to be synchronized is not lost during the transmission process.

Optionally, in the above step 203, transferring the multiple files to the target device may include the following steps:

The multiple files are transferred to the target device through multiple asynchronous threads.

Among them, multiple asynchronous threads are used to transfer multiple files to the target device, which can increase the CPU usage of the consumer device and increase the transfer rate of multiple files.

It can be seen that the data transmission method described in the embodiment of the present application receives the data set to be synchronized transmitted by the subscribing device through a two-phase submission method, where the data set to be synchronized is obtained by the subscribing device from the source. Data acquired by the device; hash the data set to be synchronized into multiple files in WAL; transfer multiple files to the target device, so that the data set to be synchronized is transmitted to the consumer device through a two-phase commit method, Therefore, it is ensured that the data set to be synchronized is successfully transmitted, and the data set to be synchronized is transmitted through multiple files of the WAL, thereby improving the efficiency of data synchronization transmission.

Consistent with the above, FIG. 3 is a schematic flowchart of a data transmission method disclosed in an embodiment of the present application. Applied to the data transmission system shown in FIG. 1A, the data transmission system may include a source device, a subscription device, a consumer device, and a target device. The data transmission method includes the following steps:

301. A source device sends a data set to be synchronized to a subscriber device through a single thread, where the data set to be synchronized includes a plurality of data to be synchronized, and each data to be synchronized corresponds to one piece of routing information.

302. The subscribing device transmits the to-be-synchronized data set to the consumer device in a two-phase commit manner.

303. The consumer device divides the multiple data to be synchronized into multiple groups according to multiple routing information corresponding to the multiple data to be synchronized, and the data to be synchronized in each of the groups corresponds to the same routing information.

304. The consumer device writes the to-be-synchronized data of each group in the plurality of groups into the file corresponding to the group in the WAL in a manner of sequentially writing files to obtain a plurality of files.

305. The consumer device transmits the multiple files to the target device through multiple asynchronous threads.

For the specific description of the above step 301 to step 305, reference may be made to the corresponding description of the data transmission method described in FIG. 1B and FIG. 2, which will not be repeated here.

It can be seen that the data transmission method described in the embodiment of this application sends a data set to be synchronized to the subscriber device through a single thread through the source device. The data set to be synchronized includes multiple data to be synchronized, and each data to be synchronized corresponds to For a piece of routing information, the subscribing device transmits the data set to be synchronized to the consumer device in a two-phase commit method, and the consumer device divides the multiple data to be synchronized into multiple groups according to multiple routing information corresponding to the multiple data to be synchronized, The data to be synchronized in each group corresponds to the same routing information. The consumer device writes the data to be synchronized in each group in the multiple groups into the file corresponding to the group in the WAL by writing files in order to obtain multiple File, the consumer device transfers multiple files to the target device through multiple asynchronous threads. In this way, the data set to be synchronized is transmitted to the consumer device through a two-phase commit method to ensure the successful transmission of the data set to be synchronized. In addition, the data set to be synchronized is transmitted to multiple files of the WAL, thereby improving the efficiency of data synchronization transmission.

Please refer to FIG. 4A. FIG. 4A is a schematic structural diagram of a subscribing device disclosed in an embodiment of the present application. As shown in the figure, the subscribing device includes a processor, a memory, a communication interface, and one or more programs. The above-mentioned one or more programs are stored in the above-mentioned memory and are configured to be executed by the above-mentioned processor, and the above-mentioned programs include instructions for executing the following steps:

Acquiring a data set to be synchronized from the source device;

The data set to be synchronized is transmitted to the consumer device through a two-phase commit method, and the consumer device is instructed to hash the data set to be synchronized into multiple files of the pre-write log WAL, and Transferring the multiple files to the target device.

In a possible example, in the aspect of obtaining the data set to be synchronized from the source device, the above program is specifically used to execute the instructions of the following steps:

Please refer to FIG. 4B. FIG. 4B is a schematic structural diagram of a consumer device disclosed in an embodiment of the present application. As shown in the figure, the consumer device includes a processor, a memory, a communication interface, and one or more programs. The above-mentioned one or more programs are stored in the above-mentioned memory and are configured to be executed by the above-mentioned processor, and the above-mentioned programs include instructions for executing the following steps:

Hash the to-be-synchronized data set into multiple files of WAL;

Transferring the multiple files to the target device.

In a possible example, the to-be-synchronized data set includes a plurality of to-be-synchronized data, and each of the to-be-synchronized data corresponds to a piece of routing information, and in the step of hashing the to-be-synchronized data set to multiple files of WAL On the other hand, the above program includes instructions for performing the following steps:

Dividing the multiple data to be synchronized into multiple packets according to multiple routing information corresponding to the multiple data to be synchronized, and the data to be synchronized in each of the packets corresponds to the same routing information;

Write the to-be-synchronized data of each of the multiple groups into the file corresponding to the group in the write-ahead log WAL to obtain multiple files.

In a possible example, in the aspect of writing the to-be-synchronized data of each group in the plurality of groups into the file corresponding to the group in the write-ahead log WAL to obtain a plurality of files, the above-mentioned program includes a method for executing Instructions for the following steps:

If the number of data types of all to-be-synchronized data in group i is greater than the preset value, divide all the to-be-synchronized data in said group i into multiple sub-groups, each of said sub-groups corresponding to said data to be synchronized The same data type, wherein the group i is any group of the plurality of groups;

Write the to-be-synchronized data of each sub-group in the multiple sub-groups of the group i into the file corresponding to the sub-group in the WAL to obtain multiple files.

In a possible example, the packet i includes m data to be synchronized, and the above program further includes instructions for executing the following steps:

In a possible example, in the aspect of writing the to-be-synchronized data of each sub-group of the multiple sub-groups of the group i into the file in the WAL to obtain multiple files, the above-mentioned program includes: Step instructions:

The foregoing mainly introduces the solution of the embodiment of the present application from the perspective of the execution process on the method side. It can be understood that, in order to implement the above-mentioned functions, an electronic device includes hardware structures and/or software modules corresponding to each function. Those skilled in the art should easily realize that in combination with the units and algorithm steps of the examples described in the embodiments provided herein, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

The embodiment of the present application may divide the electronic device into functional units according to the foregoing method examples. For example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The above-mentioned integrated unit can be implemented in the form of hardware or software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.

Please refer to FIG. 5. FIG. 5 is a schematic structural diagram of a data transmission device disclosed in an embodiment of the present application. Terminal equipment, the data transmission apparatus includes a first acquisition unit 501 and a first transmission unit 502, wherein,

The first obtaining unit 501 is configured to obtain a data set to be synchronized from the source device;

The first transmission unit 502 is configured to transmit the data set to be synchronized to the consumer device in a two-phase commit method, and instruct the consumer device to hash the data set to be synchronized to the WAL Multiple files, and transfer the multiple files to the target device.

Optionally, in the aspect of acquiring the data set to be synchronized from the source device, the first acquiring unit 501 is specifically configured to:

It can be seen that the data transmission device described in the embodiment of this application obtains the data set to be synchronized from the source device; transmits the data set to be synchronized to the consumer device in a two-phase commit method, and instructs the consumer device to The data set to be synchronized is hashed to multiple files in the WAL, and the multiple files are transferred to the target device. In this way, the data set to be synchronized is transmitted to the consumer device through a two-phase commit method to ensure the data to be synchronized The collection is successfully transmitted, and the data collection to be synchronized is transmitted through multiple files of WAL, thereby improving the efficiency of data synchronization transmission.

It should be noted that the subscriber device described in the embodiment of the present application is presented in the form of a functional unit. The term "unit" used here should be understood as the broadest possible meaning. The object used to realize the functions described by each "unit" may be, for example, an integrated circuit ASIC, a single circuit, used to execute one or more software or firmware. Program processor (shared, dedicated or chipset) and memory, combinatorial logic circuit, and/or provide other suitable components for realizing the above-mentioned functions.

Wherein, the first acquisition unit 501 and the first transmission unit 502 may be a control circuit or a processor.

Please refer to FIG. 6A. FIG. 6A is a schematic structural diagram of a data transmission device disclosed in an embodiment of the present application, which is applied to a consumer device in a data transmission system. The data transmission system also includes a source device, a subscriber device, and a target. A terminal device, the data transmission apparatus includes a receiving unit 601, a hashing unit 602, and a second transmission unit 603, where:

The receiving unit 601 is configured to receive the to-be-synchronized data set transmitted by the subscribing end device in a two-phase commit method, where the to-be-synchronized data set is obtained by the subscribing end device from the source end device The data;

The hashing unit 602 is configured to hash the to-be-synchronized data set into multiple files of WAL;

The second transmission unit 603 is configured to transmit the multiple files to the target device.

Optionally, the to-be-synchronized data set includes a plurality of to-be-synchronized data, and each of the to-be-synchronized data corresponds to one piece of routing information. In the aspect of hashing the to-be-synchronized data set into multiple files of WAL, The hash unit 602 is specifically configured to:

Optionally, in the aspect of writing the to-be-synchronized data of each of the multiple packets into the file corresponding to the packet in the write-ahead log WAL to obtain multiple files, the hashing unit is specifically used for:

Optionally, the packet i includes m data to be synchronized, as shown in FIG. 6B. FIG. 6B is a modified structure of the data transmission device shown in FIG. 6A. Compared with FIG. 6A, it may further include: a second acquiring unit 604 and statistical unit 605, of which,

The second obtaining unit 604 is configured to obtain the type identifiers of the m to-be-synchronized data in the group i to obtain m type identifiers;

The statistics unit 605 is configured to count the m type identifiers to obtain the number n of data types of all data to be synchronized in the group i, where n is less than or equal to m.

Optionally, in the aspect of writing the to-be-synchronized data of each sub-group of the multiple sub-groups of the group i into the file in the WAL to obtain multiple files, the hash unit is specifically used for:

Optionally, in the aspect of transmitting the multiple files to the target device, the second transmission unit is specifically configured to:

It can be seen that the data transmission device described in the embodiment of the present application receives the data set to be synchronized transmitted by the subscribing end device in a two-phase submission manner, where the data set to be synchronized is obtained by the subscribing device from the source end. The data acquired by the device; hash the data set to be synchronized into multiple files in the WAL; transfer multiple files to the target device, so that the data set to be synchronized is transmitted to the consumer device through a two-phase commit method, Therefore, it is ensured that the data set to be synchronized is successfully transmitted, and the data set to be synchronized is transmitted through multiple files of the WAL, thereby improving the efficiency of data synchronization transmission.

It should be noted that the consumer devices described in the embodiments of the present application are presented in the form of functional units. The term "unit" used here should be understood as the broadest possible meaning. The object used to realize the functions described by each "unit" may be, for example, an integrated circuit ASIC, a single circuit, used to execute one or more software or firmware. Program processor (shared, dedicated or chipset) and memory, combinatorial logic circuit, and/or provide other suitable components for realizing the above-mentioned functions.

Among them, the receiving unit 601, the hashing unit 602, the second transmission unit 603, the second acquisition unit 604, and the statistics unit 605 may be a control circuit or a processor.

An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program causes a computer to execute a part of any data transmission method as described in the above method embodiment. Or all steps.

The embodiments of the present application also provide a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to cause a computer to execute the method described in the foregoing method embodiment. Part or all of the steps of any data transmission method.

It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited by the described sequence of actions. Because according to this application, some steps can be performed in other order or at the same time. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in an embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed device may be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or may be Integrate into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software program modules.

If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer readable memory. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory. A number of instructions are included to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), mobile hard disk, magnetic disk, or optical disk and other media that can store program codes.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable memory, and the memory can include: a flash disk , ROM, RAM, magnetic disk or CD, etc.

The embodiments of the application are described in detail above, and specific examples are used in this article to illustrate the principles and implementation of the application. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the application; at the same time, for Those of ordinary skill in the art, based on the idea of the application, will have changes in the specific implementation and the scope of application. In summary, the content of this specification should not be construed as a limitation to the application.

Claims

A data transmission method, characterized in that it is applied to a subscriber device in a data transmission system, the data transmission system further includes a source device, a consumer device, and a target device, and the method includes:

Acquiring a data set to be synchronized from the source device;

The data set to be synchronized is transmitted to the consumer device through a two-phase commit method, and the consumer device is instructed to hash the data set to be synchronized into multiple files of the pre-write log WAL, and Transferring the multiple files to the target device.
The method according to claim 1, wherein the obtaining a data set to be synchronized from the source device comprises:

Acquire the to-be-synchronized data set transmitted by the source device through a single thread.
A data transmission method, characterized in that it is applied to a consumer device in a data transmission system. The data transmission system further includes a source device, a subscriber device, and a target device. The method includes:

Receiving the to-be-synchronized data set transmitted by the subscribing end device in a two-phase commit method, where the to-be-synchronized data set is data obtained by the subscribing end device from the source end device;

Hash the to-be-synchronized data set into multiple files of WAL;

Transferring the multiple files to the target device.
The method according to claim 3, wherein the data set to be synchronized includes a plurality of data to be synchronized, each of the data to be synchronized corresponds to one piece of routing information, and the data set to be synchronized is hashed to WAL Among the multiple files, including:

Dividing the multiple data to be synchronized into multiple packets according to multiple routing information corresponding to the multiple data to be synchronized, and the data to be synchronized in each of the packets corresponds to the same routing information;

Write the to-be-synchronized data of each group in the plurality of groups into the file corresponding to the group in the WAL to obtain a plurality of files.
The method according to claim 4, wherein the writing the to-be-synchronized data of each of the plurality of groups into the file corresponding to the group in the WAL to obtain a plurality of files includes:

If the number of data types of all to-be-synchronized data in group i is greater than the preset value, divide all the to-be-synchronized data in said group i into multiple sub-groups, each of said sub-groups corresponding to said data to be synchronized The same data type, wherein the group i is any group of the plurality of groups;

Write the to-be-synchronized data of each sub-group in the multiple sub-groups of the group i into the file corresponding to the sub-group in the WAL to obtain multiple files.
The method according to claim 5, wherein the packet i includes m data to be synchronized, and the method further comprises:

Acquiring the type identifiers of the m to-be-synchronized data in the group i to obtain m type identifiers;

The m type identifiers are counted to obtain the number n of data types of all data to be synchronized in the group i, where n is less than or equal to m.
The method according to claim 5 or 6, wherein the writing the to-be-synchronized data of each sub-group of the multiple sub-groups of the group i into the file in the WAL to obtain multiple files comprises:

In the manner of writing files in sequence, the to-be-synchronized data of each sub-group of the multiple sub-groups of the group i is written into the file in the WAL to obtain multiple files.
The method according to claim 7, wherein the transferring the multiple files to the target device comprises:

The multiple files are transferred to the target device through multiple asynchronous threads.
A data transmission device, characterized in that it is applied to a subscribing device in a data transmission system, the data transmission system further includes a source device, a consumer device, and a target device, and the data transmission device includes:

The first obtaining unit is configured to obtain a data set to be synchronized from the source device;

The first transmission unit is configured to transmit the to-be-synchronized data set to the consumer device in a two-phase commit method, and instruct the consumer-end device to hash the to-be-synchronized data set to multiple files of WAL And transfer the multiple files to the target device.
The data transmission device according to claim 9, wherein in the aspect of obtaining the data set to be synchronized from the source device, the first obtaining unit is specifically configured to:

Acquire the to-be-synchronized data set transmitted by the source device through a single thread.
A data transmission device, characterized in that it is applied to a consumer device in a data transmission system, the data transmission system further includes a source device, a subscriber device, and a target device, and the data transmission device includes:

A receiving unit, configured to receive a data set to be synchronized transmitted by the subscribing end device in a two-phase commit manner, where the data set to be synchronized is data obtained by the subscribing end device from the source device;

The hash unit is used to hash the data set to be synchronized into multiple files of the WAL;

The second transmission unit is configured to transmit the multiple files to the target device.
The data transmission device according to any one of claims 7-9, wherein the set of to-be-synchronized data includes a plurality of to-be-synchronized data, and each of the to-be-synchronized data corresponds to one piece of routing information, and the set of data to be synchronized In terms of hashing the to-be-synchronized data set into multiple files of the WAL, the hash unit is specifically used for:

Dividing the multiple data to be synchronized into multiple packets according to multiple routing information corresponding to the multiple data to be synchronized, and the data to be synchronized in each of the packets corresponds to the same routing information;

Write the to-be-synchronized data of each of the multiple groups into the file corresponding to the group in the write-ahead log WAL to obtain multiple files.
The data transmission device according to any one of claims 7-9, wherein the data to be synchronized in each of the plurality of groups is written into the file corresponding to the group in the write-ahead log WAL , In terms of obtaining multiple files, the hash unit is specifically used for:

If the number of data types of all to-be-synchronized data in group i is greater than the preset value, divide all the to-be-synchronized data in said group i into multiple sub-groups, each of said sub-groups corresponding to said data to be synchronized The same data type, wherein the group i is any group of the plurality of groups;

Write the to-be-synchronized data of each sub-group in the multiple sub-groups of the group i into the file corresponding to the sub-group in the WAL to obtain multiple files.
The data transmission device according to claim 11, wherein the packet i includes m data to be synchronized, and the device further includes a second acquisition unit and a statistics unit, wherein,

The second acquiring unit is configured to acquire the type identifiers of the m to-be-synchronized data in the group i to obtain m type identifiers;

The statistics unit is configured to count the m type identifiers to obtain the number n of data types of all data to be synchronized in the group i, where n is less than or equal to m.
The data transmission device according to claim 11, wherein in the aspect of writing the to-be-synchronized data of each sub-group of the plurality of sub-groups of the group i into the file in the WAL, a plurality of files are obtained , The hash unit is specifically used for:

In the manner of writing files in sequence, the to-be-synchronized data of each sub-group of the multiple sub-groups of the group i is written into the file in the WAL to obtain multiple files.
The data transmission device according to claim 11, wherein in the aspect of transmitting the multiple files to the target device, the second transmission unit is specifically configured to:

The multiple files are transferred to the target device through multiple asynchronous threads.
A subscription terminal device, characterized by comprising a processor, a memory, a communication interface, and one or more programs, the one or more programs are stored in the memory and configured to be executed by the processor The program includes instructions for executing the steps in the method according to claim 1 or 2.
A consumer device, characterized by comprising a processor, a memory, a communication interface, and one or more programs, the one or more programs are stored in the memory and configured to be executed by the processor The program includes instructions for executing the steps in the method according to any one of claims 3-8.
A computer-readable storage medium, characterized by storing a computer program for electronic data exchange, wherein the computer program causes a computer to execute the method according to any one of claims 1-8.
A computer program product, characterized in that the computer program product comprises a non-transitory computer-readable storage medium storing a computer program, and the computer program causes a computer to execute the method according to any one of claims 1-8.