CN115102968A - Data synchronization method, device, system and storage medium - Google Patents
Data synchronization method, device, system and storage medium Download PDFInfo
- Publication number
- CN115102968A CN115102968A CN202210682589.2A CN202210682589A CN115102968A CN 115102968 A CN115102968 A CN 115102968A CN 202210682589 A CN202210682589 A CN 202210682589A CN 115102968 A CN115102968 A CN 115102968A
- Authority
- CN
- China
- Prior art keywords
- message
- data
- synchronization
- synchronous
- messages
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 230000001360 synchronised effect Effects 0.000 claims abstract description 136
- 230000005540 biological transmission Effects 0.000 claims abstract description 21
- 238000012986 modification Methods 0.000 claims description 21
- 230000004048 modification Effects 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 19
- 238000012545 processing Methods 0.000 claims description 13
- 238000012217 deletion Methods 0.000 claims description 9
- 230000037430 deletion Effects 0.000 claims description 8
- 230000008569 process Effects 0.000 abstract description 12
- 238000004891 communication Methods 0.000 abstract description 5
- 230000009286 beneficial effect Effects 0.000 abstract 1
- 230000003993 interaction Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000002457 bidirectional effect Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/141—Setup of application sessions
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to the field of communication and provides a data synchronization method, a data synchronization device, a data synchronization system and a storage medium. The method comprises the following steps: the first equipment and/or the second equipment sends a synchronous message to the message pool, wherein the synchronous message comprises a data identifier; when the second equipment and/or the first equipment pulls the synchronous message to the message pool, acquiring a data identifier in the synchronous message; and the second equipment determines the data to be synchronized which needs to be synchronized through data identification comparison, establishes a data transmission link between the first equipment and the second equipment, and transmits the data to be synchronized. Compared with a synchronization mode based on an http protocol, the method and the device for synchronizing the data between the first device and the second device can simplify the data synchronization process between the first device and the second device, can complete synchronization of data such as a plurality of pictures at one time, and are beneficial to improving data synchronization efficiency.
Description
Technical Field
The present application relates to the field of communications, and in particular, to a data synchronization method, apparatus, and system, and a storage medium.
Background
Data synchronization is a method of achieving data consistency between two or more devices. Through data synchronization, two or more devices can be enabled to have the same data, including, for example, the same pictures, videos, and the like, so that the same data processing or presentation can be achieved on the two or more devices.
At present, a synchronization method including large data (including, for example, pictures, face feature values, audio, and the like) is mainly implemented based on an http (chinese is fully referred to as hypertext Transfer Protocol, and english is fully referred to as Hyper Text Transfer Protocol) Protocol. That is, the server provides a url (english is called Universal Resource Locator, and chinese is called uniform Resource Locator) for downloading pictures, and the client initiates an http request to download pictures through the url. The number of interactions is large, and only one picture can be synchronized in one interaction, so that the synchronization efficiency is low.
Disclosure of Invention
In view of this, embodiments of the present application provide a data synchronization method, apparatus and system, and a storage medium, so as to solve the problems in the prior art that when data synchronization is performed, the number of interactions is large, the synchronization efficiency is low, and synchronization errors are easy to occur.
A first aspect of an embodiment of the present application provides a data synchronization method, where the data synchronization method is used to synchronize a first device and a second device, and the first device and the second device are connected through a message pool, and the method includes:
the first equipment and/or the second equipment sends a synchronous message to the message pool, wherein the synchronous message comprises a data identifier;
when the second equipment and/or the first equipment pulls the synchronous message to the message pool, acquiring a data identifier in the synchronous message;
and the second equipment determines the data to be synchronized which needs to be synchronized through data identification comparison, establishes a data transmission link between the first equipment and the second equipment, and transmits the data to be synchronized.
With reference to the first aspect, in a first possible implementation manner of the first aspect, the synchronization message further includes a message type, and after the first device and/or the second device sends the synchronization message to the message pool, before the second device and/or the first device pulls the synchronization message to the message pool, the method further includes:
and the message pool combines the synchronous messages according to the message types.
With reference to the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, the message type includes a record type and a sub-record type in a directory of the record type;
the message pool carries out merging processing on the synchronous messages according to the message types, and the merging processing comprises the following steps:
when the synchronous messages received by the message pool comprise the synchronous messages with the same record type and record subtype, combining two or more synchronous messages with the same record type and record subtype into one synchronous message, and adding the data identification of the received synchronous message in the combined synchronous message.
With reference to the first possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, the message type includes an operation type, and the operation type includes an add operation, a modify operation, and a delete operation;
according to the message type, the synchronization message is merged, which comprises the following steps:
and when the synchronous messages received by the message pool comprise the same data identification and the message type is the operation type, determining effective synchronous messages according to the influence of two or more operation types on the data.
With reference to the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, when the synchronization messages received by the message pool include the same data identifier and the message type is the operation type, determining an effective synchronization message according to influences of two or more operation types on the data, includes:
when the message types of the synchronous messages with the same data identification are the adding operation and the deleting operation and are the messages in the same direction, emptying the synchronous messages with the attribute of the adding operation and not storing the synchronous messages with the attribute of the deleting operation, or emptying the synchronous messages with the attribute of the deleting operation and not storing the synchronous messages with the attribute of the adding operation;
when the message types of the synchronous messages with the same data identification are the adding operation and the modifying operation in sequence and are all the homodromous messages, adding the content of the synchronous messages with the attribute of the modifying operation into the synchronous messages with the same data identification and the attribute of the adding operation, and not storing the synchronous messages with the attribute of the adding operation;
when the message types of the synchronous messages with the same data identification are modification operation and deletion operation in sequence, deleting the synchronous messages with the attribute of the modification operation, and storing the synchronous messages with the attribute of the deletion operation;
when the message types of the synchronous messages with the same data identification are all modification operations and are all homodromous messages, combining the message contents of the synchronous messages with the same data identification, modifying the combined message contents into the stored synchronous messages with the modification operations, and not storing the newly received synchronous messages;
and when the message types of the synchronous messages with the same data identification are all modification operations and are anisotropic messages, combining the message contents of the synchronous messages with the same data identification, and respectively storing the combined message contents into the anisotropic synchronous messages.
With reference to the first possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, the message type includes a control type, and the control type includes a full amount update and a restart message synchronization;
according to the message type, the synchronization message is merged, which comprises the following steps:
when receiving a synchronous message with the attribute of full update, emptying the synchronous message in the message pool, not receiving a new synchronous message, and controlling the message pool to enter a full synchronous state;
when the message pool is completely updated and the received attribute is the restart synchronous message, the synchronous message with the attribute being full synchronization is cleared and the receiving of a new incremental message is allowed again.
With reference to the first aspect, the first possible implementation manner of the first aspect, the second possible implementation manner of the first aspect, the third possible implementation manner of the first aspect, the fourth possible implementation manner of the first aspect, or the fifth possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, the establishing a data transmission link between the first device and the second device, and transmitting the data to be synchronized includes:
and establishing a data transmission link between the first equipment and the second equipment through a private protocol, and actively pushing the data to be synchronized through the private protocol.
With reference to the first aspect, in a seventh possible implementation manner of the first aspect, the data to be synchronized includes one or more segments of data streams, the data streams include a plurality of target objects, and the number of the target objects in the data streams is the same as the number of the target objects in the data to be synchronized.
A second aspect of the embodiments of the present application provides a data synchronization method, where the data synchronization method is used to synchronize a first device and a second device, and the first device and the second device are connected through a message pool, and the method includes:
receiving a synchronization message sent by first equipment and/or second equipment, wherein the synchronization message comprises a data identifier;
when a synchronization request that a second device pulls the synchronization message is received, sending a data identifier in the received synchronization message to the second device, wherein the data identifier is used for comparing the data identifier of the second device and/or the first device with the data identifier of the synchronization message in a message pool, determining to-be-synchronized data needing to be synchronized, establishing a data transmission link between the first device and the second device, and transmitting the to-be-synchronized data.
A third aspect of the embodiments of the present application provides a data synchronization apparatus, where the data synchronization apparatus is configured to synchronize a first device and a second device, where the first device and the second device are connected through a message pool, and the apparatus includes:
a synchronization message sending unit, configured to send a synchronization message to the message pool by the first device and/or the second device, where the synchronization message includes a data identifier;
a data identifier obtaining unit, configured to obtain a data identifier in the synchronization message when the synchronization message is pulled from the message pool by the second device and/or the first device;
and the data synchronization unit is used for comparing the data identifier of the second equipment and/or the first equipment with the data identifier of the synchronization message in the message pool, determining to-be-synchronized data needing to be synchronized, establishing a data transmission link between the first equipment and the second equipment, and transmitting the to-be-synchronized data.
A third aspect of embodiments of the present application provides a data synchronization system, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the method according to any one of the first aspect when executing the computer program.
A fourth aspect of embodiments of the present application provides a computer-readable storage medium, which stores a computer program that, when executed by a processor, implements the steps of the method according to any one of the first aspects.
Compared with the prior art, the embodiment of the application has the advantages that: according to the embodiment of the application, the data identification of the data to be synchronized of the first equipment is received through the message pool, the synchronization message is pulled from the second equipment, and the data identification is sent to the second equipment, so that the second equipment determines the data to be synchronized which needs to be synchronized according to comparison of the data identification, the data to be synchronized is transmitted through establishing a data transmission link between the first equipment and the second equipment, compared with a synchronization mode based on an http protocol, the application can simplify a data synchronization process between the first equipment and the second equipment, synchronization of a plurality of pictures and other data can be completed at one time, and improvement of data synchronization efficiency is facilitated.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for the embodiments or the prior art descriptions will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
Fig. 1 is a schematic view of an implementation scenario of a data synchronization method provided in an embodiment of the present application;
fig. 2 is a schematic flowchart illustrating an implementation process of a data synchronization method according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a data synchronization apparatus provided in an embodiment of the present application;
fig. 4 is a schematic diagram of a data synchronization system provided in an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
In order to explain the technical solution described in the present application, the following description will be given by way of specific examples.
Fig. 1 is a schematic view of an implementation scenario of a data synchronization method according to an embodiment of the present application. As shown in fig. 1, the implementation scenario includes a first device, a second device, and a message pool. Wherein the first device and the second device have data transceiving capability and data storage capability. The message pool is not limited to a specific form, and may be an independent server, or may be provided in any device for synchronization. The message pool is used as a transfer station of data information and can assist the first equipment and the second equipment in data synchronization.
In the embodiment of the present application, the first device and the second device for synchronization may be used for bidirectional synchronization, that is, data synchronization from the first device to the second device and data synchronization from the second device to the first device may be implemented. And realizing data synchronization between the first equipment and the second equipment based on a bidirectional synchronization protocol. The data synchronization from the second device to the first device is basically the same as the data synchronization from the first device to the second device.
In a possible implementation manner of the present application, an implementation scenario in which one system performs data synchronization with multiple devices may also be included. That is, one system (first device) connects a plurality of devices (second devices), and one device can be connected to only a single system. The single system is bound with the plurality of devices, so that the problem of data confusion caused by accessing the plurality of systems can be effectively avoided. When the device accesses the system, the two consult to determine the IP address of the message pool, the message subject, the transmission protocol and the like, and can immediately carry out one-time full synchronization on the device. After that, no matter the modification is performed at the system end or the device end, the incremental synchronization message can be sent to the message pool according to the message format specified in the scheme, and the message pool can perform certain merging processing on the incremental synchronization message. When the equipment end or the system end pulls the increment message, the data identification of the data to be synchronized is obtained from the message pool, the data identification is compared with the data identification of the local data, the data identification to be synchronized is determined, and a data link for synchronizing the data is established for data synchronization.
Fig. 2 is a schematic diagram of an implementation flow of a data synchronization method provided in an embodiment of the present application, where the synchronization method is used for a first device and a second device shown in fig. 1, and the first device and the second device are connected by a message pool, as shown in fig. 2, the method includes:
in S201, the first device and/or the second device sends a synchronization message to the message pool, where the synchronization message includes a data identifier.
In this embodiment of the present application, according to a first device and a second device connected to a message pool, the first device may send synchronization data to the second device, and the second device may also send synchronization data to the first device. The first device or the second device may send two or more synchronization messages to the message pool. If the sender of two or more synchronous messages is the same, it means that the two or more synchronous messages are the same-direction messages. For example, if two or more synchronization messages are sent to the message pool by the first device, the two or more synchronization messages are the same-direction messages. Correspondingly, if the two synchronization messages have different sending parties, for example, one of the synchronization messages is sent by the first device and the other synchronization message is sent by the second device, it indicates that the two synchronization messages are directed messages or directed synchronization messages.
When the first device and/or the second device sends the synchronization message to the message pool, including when the first device and/or the second device is connected to the data synchronization system, or incremental data appears in the first device and/or the second device, for example, a picture and a face feature are added in the first device and/or the second device to wait for synchronization data.
Before the first device and/or the second device synchronizes the data, marking the data to be synchronized in the first device and/or the second device, and determining the data identification of the data to be synchronized in the first device and/or the second device. The data identification is used for globally and uniquely identifying the data. The data identifier may determine the data identifier of the data to be synchronized through an encryption algorithm such as hash calculation.
In the embodiment of the application, when the first device and/or the second device synchronizes data to other devices, the data identifier of the data to be synchronized may be sent first, so that the data identifier of one or more data to be synchronized may be obtained between the message pool and the first device and/or the second device through a small amount of data.
In S202, when the second device and/or the first device pulls the synchronization message from the message pool, the data identifier in the synchronization message is obtained.
When the message pool is connected to the first device and the second device, the synchronization message received by the message pool may include a synchronization message sent by the first device, a synchronization message sent by the second device, or a synchronization message sent by the first device and the second device. When the synchronization message is pulled, the method may include that the first device pulls the synchronization message, the second device pulls the synchronization message, or the first device and the second device pull the synchronization message.
Before the second device and/or the first device pull the synchronization message to the message pool, the message pool may perform a merging process on the synchronization message according to the received message type. For example, several or tens of synchronization messages may be combined into one synchronization message, and when the second device and/or the first device pulls the synchronization message, unnecessary redundant messages in the message pool may be reduced, and the workload of the second device for synchronization may be reduced.
The message pool internal structure may adopt a Reactor mode, and includes a receiving thread group (Acceptor) and a processing thread group (Handler). The receiving thread group is used for monitoring network connection and generating a unique message identifier (MSGID) from the received message, and then the MSGID is delivered to the processing thread group for subsequent processing, including merging identified synchronous messages, maintaining a message list, persisting the synchronous messages and the like.
When the synchronous messages are merged, the synchronous messages can be merged according to the types of the messages, so that redundant synchronous messages in the message pool are eliminated as much as possible, and the situation that the same attribute is modified for multiple times is prevented. Thus, the type of synchronization message may be determined prior to synchronization message merging. The type of the synchronization message may be determined according to a predetermined correspondence between the synchronization message and the type. The determined type of the synchronization message may include a record class, an operation class, a control class, and the like.
For each type of synchronization message, a primary message type (fixed) and a secondary message type (optional) may be included. The primary message type may be fixed by a protocol, and each system or device may only use the primary message type specified by the protocol. The secondary message type can be designed to provide expansion and adaptation for each system or device, and the system or device can define the message content of the device. Therefore, the merging of the synchronization messages can be performed based on the primary message type.
The record type message may include a primary message type "record" and may represent all immutable records, including face snapshot record, attendance record, vehicle entry record, and the like.
The operation class message includes three kinds of messages for variable data, namely adding, modifying and deleting, which respectively represent operations for performing addition, modification and deletion of data.
The control type message is more specific to the message synchronization process itself, and the plan includes primary message types such as full synchronization, stop synchronization, restart, connection, and the like, which respectively indicate the meanings of one-time full synchronization, stop message synchronization, restart message synchronization, and negotiation of connection mode required for data.
In the embodiment of the present application, a field representing the message propagation direction may be included in the message to distinguish from which device the message is delivered to. Or may also include a message subject field that may be used to identify to which subject the message belongs.
When the synchronous messages are combined, a quick look-up table can be established based on a Key value pair mapping table, a message type and a theme are used as keys of a first layer, and for the messages of a record type and a control type, a value of the first layer is an actual value. For the operation type message, the unique message identifier can be used as a second layer key, and the value of the second layer key is incremental data which will take effect finally; after receiving the message, the processing thread group in the message pool scans the quick look-up table, merges the messages according with the merging type, and updates the quick look-up table.
The message merging in the embodiment of the present application may include merging of a record type synchronization message, merging of an operation type synchronization message, and merging of a control type message. Wherein:
1. the merging of class messages is recorded.
The method can comprise a record-record type, when the type of the primary message is a record, the subject of the synchronous message is the same, and the type of the secondary message is a sub-record type under the record type, the two synchronous messages are considered to be combined, the result of the combination is that a record message is added to the value of the entry and is used for recording the data identifier needing to be synchronized, and then the batch synchronization is carried out at one time.
2. And merging the operation type messages.
2.1 addition-deletion type, this type will only appear in the same direction message, if the received first class message is deletion, and there is operation type message with the same data identification to trigger, the result of merging is to empty the existing addition message, the newly added deletion message is not preserved.
2.2, adding-modifying type, the type only appears in the same-direction message, if the type of the received first-level message is modified and the added message with the same data identification exists, the first-level message and the added message are combined, the result is that the field attribute in the added message is modified, the field attribute is changed into the attribute of modified content, and the newly added modified message is not stored.
2.3, modify-delete type, the message type is not limited in direction, and is applicable to the following operations: if a delete type message is received and a modification message with the same data identifier exists, the merged structure is to delete the modification message and keep the deletion message.
2.4, modification-modification type, when receiving a modification type message, namely a modification message, if the modification message with the same data identifier and the same direction exists, combining the message contents of the two and modifying the message contents into the existing modification message, wherein the new modification message is not reserved; however, if there is a "modified" message with the same data identifier but different directions, the contents of the two messages need to be merged, and the merged message contents are stored in two different-direction synchronization messages respectively.
3. And controlling the merging of the messages.
And 3.1, a full update type, wherein when a message of the full update type is received, all the quick look-up tables are directly emptied, no new message is received any more, and the state of waiting for full synchronization is entered.
3.2, restart type, when the full synchronization is completed, the party receiving the data is responsible for sending a message of "restart" type to the message pool, at which point the message of "full update" type can be eliminated and the new incremental message is re-allowed to be received.
In S203, comparing the data identifier of the second device and/or the first device with the data identifier of the synchronization message in the message pool, determining to-be-synchronized data that needs to be synchronized, establishing a data transmission link between the first device and the second device, and transmitting the to-be-synchronized data.
When the second device and/or the first device pull data, the data processed by the message pool can be obtained. Because the message pool can combine the received synchronous messages, the redundant synchronous messages can be effectively removed, the interaction times of data synchronization are greatly reduced, the process is shorter, and the error probability is lower. And the method and the device can perform matching and data transmission based on a private protocol, and can improve the safety of data synchronization. According to the method and the device, the synchronous message and the synchronous data are processed separately, and the second device and/or the first device can obtain a plurality of data identifications to be synchronized at one time when pulling the data, so that the second device and/or the first device can efficiently obtain a plurality of synchronous data through the data transmission link, the synchronization process is simplified, and the data synchronization efficiency is improved.
The data link established between the first device and the second device may be a direct data link or an indirect data link. The data link can be established through a data transmission link based on a private protocol, and the device needing synchronization can actively push data based on the data transmission link. According to the proprietary protocol, the synchronization operation is enabled to complete the synchronization of various types of data (including pictures, videos, audios and the like) in one interaction. Without limitation, data links established by other communication protocols may also be included. When an indirect data link is established, data that needs to be synchronized can be forwarded through the message pool.
The private protocol can be privatized and expanded based on the RTSP, and compared with the current http protocol interaction mode, the private protocol has the advantages of fewer interaction times, shorter flow and lower error probability. Compared with the protocol based on the RFC standard, the private protocol has relatively smaller risk of being decoded and higher safety. By adopting the intelligent data format, a plurality of sections of data of different types, such as pictures, videos, audios and the like, can be intelligently identified and transmitted, and attribute information corresponding to the data type can be determined based on the identified data type, so that the requirements of the data transmission of different types can be met in a customized manner. The NVR, IPC and other devices are involved in the same system, and the integrity of the system can be better maintained by using a private protocol.
In addition, in the data stream of the private protocol described in this embodiment of the present application, a top layer thereof may transmit a plurality of target objects (for example, different types of target objects or a plurality of target objects of the same type), that is, a segment of the data stream includes a data stream header and one or more target objects, and the number of the target objects in the data stream is the same as the number of the target objects in the data to be synchronized, so as to reduce the number of interactions when the plurality of target objects are transmitted. The format of the private protocol may be a TLV (type-length-value in english, and type-length value in chinese), so that the private protocol has good expansibility, which facilitates to add attributes of the target object. That is, the length identifier in TLV format may define the data length of the information of the data stream, the target object, the attribute, etc., and based on the defined data length, complete determination of the specific information of the data stream, the target object, or the attribute may be facilitated. And when the specific information is changed, the defined data length is correspondingly adjusted. Based on the proprietary protocol format, the requirement of synchronizing a plurality of segments of data can be effectively met, and the data interaction efficiency is improved.
The data link established directly includes data link based on HTTP, FTP protocol, etc. Without being limited thereto, the data link may also be established based on a message pool, i.e. indirectly through the message pool. Or, the data identifier may be calculated based on the message pool, the data identifier may be compared with the message pool, the synchronization data that needs to be sent to the second device may be determined, and the synchronization message may be merged via the message pool.
It can be understood that, when the execution subject is a message pool, the synchronization method may receive a synchronization message sent by the first device and/or the second device through the message pool, where the synchronization message includes a data identifier; when a synchronization request that a second device and/or a first device pull the synchronization message is received, comparing the data identifier of the second device and/or the first device with the data identifier of the synchronization message in a message pool, determining data to be synchronized which need to be synchronized, so as to establish a data transmission link of the first device and the second device, and transmitting the data to be synchronized. The synchronization method corresponds to the synchronization method shown in fig. 2.
The data synchronized in the embodiment of the application may include data such as a photo, a face feature, a fingerprint feature, and the like.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
Fig. 3 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present application, and as shown in fig. 3, the apparatus includes:
a synchronization message sending unit 301, configured to send a synchronization message to the message pool by the first device and/or the second device, where the synchronization message includes a data identifier;
a data identifier obtaining unit 302, configured to obtain a data identifier in the synchronization message when the synchronization message is pulled from the message pool by the second device and/or the first device;
a data synchronization unit 303, configured to compare the data identifier of the second device and/or the first device with the data identifier of the synchronization message in the message pool, determine to-be-synchronized data that needs to be synchronized, establish a data transmission link between the first device and the second device, and transmit the to-be-synchronized data.
The data synchronization device shown in fig. 3 corresponds to the data synchronization method shown in fig. 2.
Fig. 4 is a schematic diagram of a data synchronization system according to an embodiment of the present application. As shown in fig. 4, the data synchronization system 4 of this embodiment includes: a processor 40, a memory 41 and a computer program 42, such as a data synchronization program, stored in said memory 41 and operable on said processor 40. The processor 40 implements the steps in the various data synchronization method embodiments described above when executing the computer program 42. Alternatively, the processor 40 implements the functions of the modules/units in the above-described device embodiments when executing the computer program 42.
Illustratively, the computer program 42 may be partitioned into one or more modules/units that are stored in the memory 41 and executed by the processor 40 to accomplish the present application. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution of the computer program 42 in the data synchronization system 4.
The data synchronization system may include, but is not limited to, a processor 40, a memory 41. Those skilled in the art will appreciate that fig. 4 is merely an example of the data synchronization system 4 and does not constitute a limitation of the data synchronization system 4 and may include more or fewer components than shown, or some components in combination, or different components, for example, the data synchronization system may also include input output devices, network access devices, buses, etc.
The Processor 40 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 41 may be an internal storage unit of the data synchronization system 4, such as a hard disk or a memory of the data synchronization system 4. The memory 41 may also be an external storage device of the data synchronization system 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the data synchronization system 4. Further, the memory 41 may also include both an internal storage unit and an external storage device of the data synchronization system 4. The memory 41 is used for storing the computer programs and other programs and data required by the data synchronization system. The memory 41 may also be used to temporarily store data that has been output or is to be output.
It should be clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional units and modules is only used for illustration, and in practical applications, the above function distribution may be performed by different functional units and modules as needed, that is, the internal structure of the apparatus may be divided into different functional units or modules to perform all or part of the above described functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. For the specific working processes of the units and modules in the system, reference may be made to the corresponding processes in the foregoing method embodiments, which are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when the actual implementation is performed, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, that is, may be located in one place, or may also be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above may be implemented by hardware related to instructions of a computer program, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the steps of the methods described above may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain other components which may be suitably increased or decreased as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media which may not include electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application, and are intended to be included within the scope of the present application.
Claims (12)
1. A data synchronization method is used for synchronizing a first device and a second device, wherein the first device and the second device are connected through a message pool, and the method comprises the following steps:
the first device and/or the second device sends a synchronization message to the message pool, wherein the synchronization message comprises a data identifier;
when the second equipment and/or the first equipment pulls the synchronous message to the message pool, acquiring a data identifier in the synchronous message;
and comparing the data identifier of the second equipment and/or the first equipment with the data identifier of the synchronization message in the message pool, determining to-be-synchronized data needing to be synchronized, establishing a data transmission link between the first equipment and the second equipment, and transmitting the to-be-synchronized data.
2. The method of claim 1, wherein the synchronization message further comprises a message type, and after the first device and/or the second device sends the synchronization message to the message pool, before the second device and/or the first device pulls the synchronization message to the message pool, the method further comprises:
and the message pool combines the synchronous messages according to the message types.
3. The method of claim 2, wherein the message type comprises a record type and a sub-record type under a directory of the record type;
the message pool carries out merging processing on the synchronous messages according to the message types, and the merging processing comprises the following steps:
when the synchronous messages received by the message pool comprise the synchronous messages with the same record type and record subtype, combining two or more synchronous messages with the same record type and record subtype into one synchronous message, and adding the data identification of the received synchronous message in the combined synchronous message.
4. The method of claim 2, wherein the message types include operation types including add, modify, and delete operations;
according to the message type, the synchronization message is merged, which comprises the following steps:
and when the synchronous messages received by the message pool comprise the same data identification and the message type is the operation type, determining effective synchronous messages according to the influence of two or more operation types on the data.
5. The method of claim 4, wherein when the synchronization messages received by the message pool include the same data identifier and the message type is the operation type, determining a valid synchronization message according to the influence of two or more operation types on the data comprises:
when the message types of the synchronous messages with the same data identification are the adding operation and the deleting operation and are the messages in the same direction, emptying the synchronous messages with the attribute of the adding operation and not storing the synchronous messages with the attribute of the deleting operation, or emptying the synchronous messages with the attribute of the deleting operation and not storing the synchronous messages with the attribute of the adding operation;
when the message types of the synchronous messages with the same data identification are the adding operation and the modifying operation in sequence and are the messages in the same direction, the content of the synchronous messages with the attribute of the modifying operation is added to the synchronous messages with the same data identification and the attribute of the synchronous messages with the adding operation, and the synchronous messages with the attribute of the adding operation are not stored;
when the message types of the synchronous messages with the same data identification are modification operation and deletion operation in sequence, deleting the synchronous messages with the attribute of the modification operation, and storing the synchronous messages with the attribute of the deletion operation;
when the message types of the synchronous messages with the same data identification are all modification operations and are all homodromous messages, combining the message contents of the synchronous messages with the same data identification, modifying the combined message contents into the stored synchronous messages with the modification operations, and not storing the newly received synchronous messages;
and when the message types of the synchronous messages with the same data identification are all modification operations and are anisotropic messages, combining the message contents of the synchronous messages with the same data identification, and respectively storing the combined message contents into the anisotropic synchronous messages.
6. The method of claim 2, wherein the message type comprises a control type, wherein the control type comprises a full update and a resume message synchronization;
according to the message type, the synchronization message is merged, which comprises the following steps:
when receiving a synchronous message with the attribute of full update, emptying the synchronous message in the message pool, not receiving a new synchronous message, and controlling the message pool to enter a full synchronous state;
when the full update of the message pool is completed and the received attribute is the restart synchronization message, the synchronization message with the attribute of full synchronization is cleared and the receiving of a new increment message is allowed again.
7. The method according to any one of claims 1-6, wherein establishing a data transmission link between the first device and the second device, transmitting the data to be synchronized, comprises:
and establishing a data transmission link between the first equipment and the second equipment through a private protocol, and actively pushing the data to be synchronized through the private protocol.
8. The method according to claim 7, wherein the data to be synchronized comprises one or more segments of data stream, the data stream comprises a plurality of target objects, and the number of target objects in the data stream is the same as the number of target objects in the data to be synchronized.
9. A data synchronization method is used for synchronizing a first device and a second device, wherein the first device and the second device are connected through a message pool, and the method comprises the following steps:
receiving a synchronization message sent by first equipment and/or second equipment, wherein the synchronization message comprises a data identifier;
when a synchronization request that a second device pulls the synchronization message is received, comparing the data identifier of the second device and/or the first device with the data identifier of the synchronization message in a message pool, determining data to be synchronized, so as to establish a data transmission link between the first device and the second device, and transmitting the data to be synchronized.
10. A data synchronization device, wherein the data synchronization device is configured to synchronize a first device and a second device, and the first device and the second device are connected through a message pool, the device comprising:
a synchronization message sending unit, configured to send a synchronization message to the message pool by the first device and/or the second device, where the synchronization message includes a data identifier;
a data identifier obtaining unit, configured to obtain a data identifier in the synchronization message when the synchronization message is pulled from the message pool by the second device and/or the first device;
and the data synchronization unit is used for comparing the data identifier of the second equipment and/or the first equipment with the data identifier of the synchronization message in the message pool, determining to-be-synchronized data needing to be synchronized, establishing a data transmission link between the first equipment and the second equipment, and transmitting the to-be-synchronized data.
11. A data synchronization system comprising a memory, a processor and a computer program stored in said memory and executable on said processor, characterized in that said processor implements the steps of the method according to any of claims 1 to 8 when executing said computer program.
12. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210682589.2A CN115102968B (en) | 2022-06-16 | 2022-06-16 | Data synchronization method, device, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210682589.2A CN115102968B (en) | 2022-06-16 | 2022-06-16 | Data synchronization method, device, system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115102968A true CN115102968A (en) | 2022-09-23 |
| CN115102968B CN115102968B (en) | 2024-08-30 |
Family
ID=83290838
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210682589.2A Active CN115102968B (en) | 2022-06-16 | 2022-06-16 | Data synchronization method, device, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115102968B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101500208A (en) * | 2008-01-31 | 2009-08-05 | 三星电子株式会社 | Data synchronization method and system between devices |
| CN104168335A (en) * | 2014-09-02 | 2014-11-26 | 东软熙康健康科技有限公司 | Data synchronization method and device |
| CN104184756A (en) * | 2013-05-21 | 2014-12-03 | 阿里巴巴集团控股有限公司 | Data synchronization method, device and system |
| CN105187475A (en) * | 2015-06-09 | 2015-12-23 | 深圳市金蝶友商电子商务服务有限公司 | Method and system for data synchronization |
| CN110493298A (en) * | 2018-05-15 | 2019-11-22 | 优酷网络技术（北京）有限公司 | Information synchronization method and device |
| CN112612799A (en) * | 2020-12-08 | 2021-04-06 | 福建天泉教育科技有限公司 | Data synchronization method and terminal |
| US20210281641A1 (en) * | 2020-03-03 | 2021-09-09 | Snap Inc. | Minimizing number of synchs |
| CN113868228A (en) * | 2021-09-29 | 2021-12-31 | 北京沃东天骏信息技术有限公司 | Data synchronization method, device, equipment and computer readable storage medium |
-
2022
- 2022-06-16 CN CN202210682589.2A patent/CN115102968B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101500208A (en) * | 2008-01-31 | 2009-08-05 | 三星电子株式会社 | Data synchronization method and system between devices |
| CN104184756A (en) * | 2013-05-21 | 2014-12-03 | 阿里巴巴集团控股有限公司 | Data synchronization method, device and system |
| CN104168335A (en) * | 2014-09-02 | 2014-11-26 | 东软熙康健康科技有限公司 | Data synchronization method and device |
| CN105187475A (en) * | 2015-06-09 | 2015-12-23 | 深圳市金蝶友商电子商务服务有限公司 | Method and system for data synchronization |
| CN110493298A (en) * | 2018-05-15 | 2019-11-22 | 优酷网络技术（北京）有限公司 | Information synchronization method and device |
| US20210281641A1 (en) * | 2020-03-03 | 2021-09-09 | Snap Inc. | Minimizing number of synchs |
| CN112612799A (en) * | 2020-12-08 | 2021-04-06 | 福建天泉教育科技有限公司 | Data synchronization method and terminal |
| CN113868228A (en) * | 2021-09-29 | 2021-12-31 | 北京沃东天骏信息技术有限公司 | Data synchronization method, device, equipment and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115102968B (en) | 2024-08-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20050235019A1 (en) | Method and system for transmitting data for data synchronization between server and client when data synchronization session was abnormally terminated | |
| CN109951546B (en) | Transaction request processing method, device, equipment and medium based on intelligent contract | |
| EP3820107A1 (en) | Method and device for transmitting data, and method and apparatus for receiving data | |
| CN108093015B (en) | File transmission system | |
| CN111193789B (en) | Subscription information pushing method, device, computer equipment and readable storage medium | |
| CN113709250B (en) | Cross-domain user data synchronization method based on subscription transmission mode | |
| CN112087475B (en) | Message pushing method and device for cloud platform component application and message server | |
| US5490088A (en) | Method of handling data retrieval requests | |
| US8745101B2 (en) | Terminal and method for identifying contents | |
| CN103401931A (en) | Method and system for downloading file | |
| US7779299B2 (en) | Efficiently re-starting and recovering synchronization operations between a client and server | |
| CN115102968B (en) | Data synchronization method, device, system and storage medium | |
| US20110055279A1 (en) | Application server, object management method, and object management program | |
| CN103327026A (en) | Data updating method and updating system | |
| CN113553206B (en) | Data event execution method and device, electronic equipment and computer readable medium | |
| CN113076380B (en) | Data synchronization method, device, system, equipment and storage medium | |
| JPH11265309A (en) | Partial extraction type remote maintenance system | |
| KR20110074244A (en) | Apparatus and method for synchronizing data between instant messaging clients in communication system | |
| CN114430496B (en) | Cross-equipment video searching method and related equipment | |
| CN113111042A (en) | Archive file management method and system and electronic equipment | |
| CN114745367A (en) | Data transmission method and device, electronic equipment and storage medium | |
| JPH11249977A (en) | Data transfer network system | |
| CN113810266B (en) | Retry operation method, device, equipment and storage medium for message object | |
| CN110929500B (en) | File comparison method and related device | |
| CN109688204A (en) | Document down loading method, device based on NDN network, node, terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |