CN111143382B - Data processing method, system and computer readable storage medium - Google Patents
Data processing method, system and computer readable storage medium Download PDFInfo
- Publication number
- CN111143382B CN111143382B CN201911292296.8A CN201911292296A CN111143382B CN 111143382 B CN111143382 B CN 111143382B CN 201911292296 A CN201911292296 A CN 201911292296A CN 111143382 B CN111143382 B CN 111143382B
- Authority
- CN
- China
- Prior art keywords
- service
- metadata
- child node
- database
- instruction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing method, a data processing system and a computer readable storage medium, and relates to the field of cluster data processing. The method is applied to a first service of a distributed data processing system and comprises the following steps: the plurality of services with the cache respectively register respective child nodes under a root node of the distributed application program coordination component; when the first service receives a metadata modification request, sending a first modification instruction to the database to enable the database to modify the first metadata information; updating the first service cache according to the modified first metadata information in the database; and sending a first metadata adjustment instruction to the child nodes of other services, and updating respective caches when the other services acquire the first metadata adjustment instruction by monitoring the corresponding child nodes. The first service receives the metadata modification request and sends a first modification instruction to the database, so that the database updates the first metadata information, and the first service updates the corresponding metadata in the cache, thereby improving the data processing efficiency.
Description
Technical Field
The present invention relates to the field of cluster data processing, and in particular, to a data processing method, system, and computer-readable storage medium.
Background
In the initial stage of website establishment, a single machine is generally used for providing centralized services, but as the traffic volume is larger and larger, the challenges in performance and stability are greater. To provide better service: multiple machines are typically grouped into a cluster to provide services. Currently, most internet systems use server clustering technology, such as web application server clustering, database server clustering, distributed cache server clustering, and the like.
In a cluster, in order to improve data processing efficiency, common data generally needs to be cached, and one prior art scheme is: storing the metadata into the Zookeeper, and synchronizing data from the Zookeeper node corresponding to each different service when the metadata in the Zookeeper node changes; obviously, the metadata is stored in the Zookeeper node, the maintenance difficulty is high, the data is not viewed and displayed, and the data reading speed is low. Another prior art scheme is: when the metadata needs to be read, reading the metadata from the cache, inquiring the original metadata database if the metadata does not exist in the cache, and regularly reading the whole amount of metadata from the original metadata database so as to update the cache; obviously, it cannot guarantee that the data taken from the cache each time is consistent with the original database, and the cache update time is difficult to determine. Therefore, in order to improve the efficiency of data processing, how to implement synchronous update of data cache when metadata changes is an urgent problem to be solved at present.
Disclosure of Invention
In view of the above, the present invention is directed to a data processing method, system and computer readable storage medium.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
in a first aspect, the present invention provides a data processing method, which is applied to a first service, where the first service is provided in a distributed data processing system, the distributed data processing system includes a database, a distributed application program coordination component, and a plurality of services with caches, and the first service is any one of the plurality of services with caches. The method comprises the following steps: the plurality of services with caches register respective child nodes under a root node of the distributed application coordination component; when the first service receives a metadata modification request, the first service sends a first modification instruction corresponding to the metadata modification request to the database, so that the database modifies first metadata information corresponding to the first modification instruction; wherein the database comprises first metadata information corresponding to the first service; the first service updates the first metadata information in the first service cache according to the modified first metadata information in the database; the first service sends a first metadata adjusting instruction to child nodes of other services; when the other services obtain the first metadata adjusting instruction by monitoring the corresponding child nodes, the other services update the first metadata information in the caches according to the first metadata adjusting instruction; the first metadata adjustment instruction is to indicate that first metadata information of the database has been modified.
In an optional embodiment, the distributed data processing system further includes a first child node corresponding to the first service, where the first child node is a child node registered under the root node for the first service, and the database further includes second metadata information corresponding to the other services, and the method further includes: the first service monitors the first child node; when the first child node has a second metadata adjustment instruction sent by the other service, the first service updates the second metadata information stored in the first service cache according to the modified second metadata information in the database; the second metadata adjustment instruction is used for indicating that second metadata information of the database is modified.
In an optional embodiment, when the first child node has a second metadata adjustment instruction sent by the other service, the updating, by the first service, the second metadata information stored in the first service cache according to the modified second metadata information in the database includes: when the first child node has the second metadata adjustment instruction, the first service judges whether second metadata information corresponding to the second metadata adjustment instruction is stored in the first service cache; if so, the first service modifies the second metadata information stored in the first service cache according to the modified second metadata information in the database; and if not, discarding the node information of the first child node.
In an optional implementation manner, the distributed data processing system further includes a second service and a second child node corresponding to the second service, where the second service is any one of the other services, the second child node is a child node registered by the second service under the root node, and the sending, by the first service, the first metadata adjustment instruction to the child node of the other service includes: the first service acquires version information of the second child node; the first service generates the first metadata adjustment instruction consistent with the version information; the first service sends the first metadata adjustment instruction to the second child node.
In an optional embodiment, the sending, by the first service, the first metadata adjustment instruction to the second child node includes: the first service acquires node data of the second child node and empties the node data; the first service generates a write instruction, wherein the write instruction comprises the node data and the first metadata adjustment instruction; the first service sends the write instruction to the second child node, so that the second child node receives and stores the write instruction.
In a second aspect, the present invention provides a distributed data processing system comprising: a database, a distributed application coordination component, and a plurality of services with caches. The plurality of services with the cache register respective child nodes under the root node corresponding to the distributed application program coordination component; the first service is used for sending a first modification instruction corresponding to a metadata modification request to the database when the first service receives the metadata modification request; wherein the first service is any one of the plurality of services with cache. The database is used for modifying first metadata information corresponding to the first modification instruction; the database comprises first metadata information corresponding to the first service; the first service is further configured to update the first metadata information in the first service cache according to the modified first metadata information in the database. The first service is further configured to send a first metadata adjustment instruction to child nodes of other services, so that when the other services obtain the first metadata adjustment instruction by monitoring the respective corresponding child nodes, the other services update the first metadata information in respective caches according to the first metadata adjustment instruction, where the first metadata adjustment instruction is used to indicate that the first metadata information of the database is modified.
In an optional embodiment, the distributed data processing system further includes a first child node corresponding to the first service, where the first child node is a child node registered by the first service under the root node, and the database further includes second metadata information corresponding to the other services; the first service is also used for monitoring the first child node; the first service is further configured to update the second metadata information stored in the first service cache according to the modified second metadata information in the database when the first child node has a second metadata adjustment instruction sent by the other service; the second metadata adjustment instruction is used for indicating that second metadata information of the database is modified.
In an optional implementation manner, the distributed data processing system further includes a second service and a second child node corresponding to the second service, where the second service is any one of the other services, and the second child node is a child node registered by the second service under the root node. The first service is further to: acquiring version information of the second child node; generating the first metadata adjustment instruction consistent with the version information; and sending the first metadata adjusting instruction to the second child node.
In an optional embodiment, the first service is further configured to acquire node data of the second child node and empty the node data; the first service is further to generate a write instruction; the write instruction includes the node data and the first metadata adjustment instruction; the first service is further configured to send the write instruction to the second child node, so that the second child node receives and stores the write instruction.
In a third aspect, the invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method according to any of the preceding embodiments.
Compared with the prior art, the invention provides a data processing method, a data processing system and a computer readable storage medium, and relates to the field of cluster data processing. The method is applied to a first service and comprises the following steps: when the first service receives the metadata modification request, the first service sends a first modification instruction corresponding to the metadata modification request to the database so that the database modifies first metadata information corresponding to the first modification instruction; the first service updates the first metadata information in the first service cache according to the modified first metadata information in the database; the first service sends a first metadata adjusting instruction to child nodes of other services; and when other services acquire the first metadata adjusting instruction by monitoring the corresponding child nodes, updating the first metadata information in the cache according to the first metadata adjusting instruction. The first service receives the metadata modification request and sends a first modification instruction corresponding to the metadata modification request to the database, so that the database updates first metadata information corresponding to the first modification instruction, the first service updates the first metadata information in a corresponding first service cache, the metadata information stored in the first service cache is consistent with the latest database, and the accuracy of data processing is improved.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
FIG. 1 is a schematic structural diagram of a ZooKeeper cluster;
FIG. 2 is a block diagram illustrating a multi-service metadata cache according to an embodiment of the present invention;
fig. 3 is a schematic flow chart of a data processing method according to an embodiment of the present invention;
FIG. 4 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
FIG. 5 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
FIG. 6 is a block diagram illustrating an alternative multi-service metadata cache according to an embodiment of the present invention;
FIG. 7 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
FIG. 8 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
FIG. 9 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
fig. 10 is a block diagram of an electronic device according to an embodiment of the present invention.
Icon: 100-distributed data processing system, 110-first service, 111-first child node, 120-second service, 121-second child node, 130-root node, 140-database, 150-third service, 151-third child node, 30-Zookeeper cluster, 31-service, 32-child node, 321-cache child node, 33-parent node, 70-electronic device, 71-memory, 72-processor, 73-communication interface.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
It is noted that relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Advanced distributed cluster management is to enable multiple servers to work cooperatively to complete tasks that cannot be processed by a single server, especially tasks with high concurrency or large data volume. Among them, in order to implement a coordination service between a plurality of servers, it is generally implemented in a ZooKeeper or the like, for example.
ZooKeeper as an open source distributed application coordination service that may provide functions including, but not limited to: configuration maintenance, domain name service, distributed synchronization, group service, etc. For example, referring to fig. 1, fig. 1 is a schematic structural diagram of a ZooKeeper cluster 30, which includes: a service 31, a child node 32 (Znode), and a parent node 33. In the ZooKeeper cluster 30 shown in fig. 1, taking as an example that the file data structure of the ZooKeeper cluster 30 is a tree structure, the tree structure includes a plurality of child nodes 32 (i.e., Znode); the path difference between the child node 32 and the conventional file system is that: in a file system, a path is only a name and does not contain data; each child node 32 (Znode) may not only serve as a unique identifier for data queries through its own path, but any child node 32 (Znode) may store data, such as metadata in the Zookeeper cluster 30, into the child node 32.
Further, Zookeeper in a distributed system may be used to deploy multiple services, and each service may create Znode when any one service is deployed, with the purpose of reducing system coupling of the Zookeeper cluster 30. Meanwhile, in order to acquire the service 31 to execute the business process, the service 31 may listen to the stored data change of the corresponding child node 32. In this case, the stored data in the child node 32 may be used to instruct the service 31 to implement the business logic corresponding to the stored data.
In the current technical solution, in order to implement metadata invocation of multiple services in a distributed cluster, metadata is generally stored in a child node 32 (Znode), and when the service 31 monitors that the metadata stored in the corresponding child node 32 changes, the service 31 may synchronize the metadata from the corresponding child node 32.
Whereas in the art, since the upper limit of each Znode stored data is generally small, e.g., 1MB, it is only appropriate to store a small amount of data. Referring to fig. 1, when the total amount of metadata of the service 31 is large, a plurality of metadata need to be stored in a plurality of child nodes 32 (Znode), that is, the metadata corresponding to the service 31 is stored in a plurality of child nodes 32 (Znode), at this time, in order to implement monitoring of metadata change by the service 31, the service 31 needs to establish a monitor for each child node 32 so as to obtain a metadata change condition of each child node 32 (Znode), thereby causing waste of service resources.
In addition, if there is an association relationship between metadata stored in a child node and the total amount of the metadata is large, in order to maintain the association between the metadata, a set of metadata with the association needs to be stored in a group of child nodes, please refer to fig. 1, the group of child nodes may include a child node 32 (Znode) created during service deployment and a plurality of cache child nodes 321 created by the child node 32 (Znode) according to actual service requirements for storing the metadata, and when the set of metadata with the association needs to be modified, the child node 32 and the cache child node 321 need to respectively modify the metadata stored therein. And also needs to consider the relevance between the metadata in the set of metadata. It is therefore difficult to ensure consistency of the set of metadata modifications under this mechanism. In addition, when a set of metadata having relevance is stored in the child node 32 and the plurality of cache child nodes 321, corresponding metadata needs to be acquired from the child node 32 and the plurality of cache child nodes 321 when the set of metadata needs to be viewed, thereby causing inconvenience in viewing and showing the metadata.
Based on the above problems and the deficiencies of the background art, an embodiment of the present invention provides an architecture for implementing a multi-service metadata cache, taking two services deployed on a distributed data processing system as an example, please refer to fig. 2, where fig. 2 is a schematic structural diagram of a multi-service metadata cache according to an embodiment of the present invention. The distributed data processing system 100 includes: a first service 110, a first child node 111, a second service 120, a second child node 121, a root node 130, and a database 140.
The first service 110 has a first service cache and the second service 120 has a second service cache, it being understood that the first child node 111 is a child node created by the first service 110 under the root node 130 of the distributed application coordination component; the second child node 121 is a child node created by the second service 120 under the root node 130, and the second service 120 may listen to the second child node 121.
The first service 110 and the second service 120 may be on the same electronic device or different electronic devices, and the form and topology of the hardware devices in which the services, nodes, and databases are located are not limited in the present invention.
It should be appreciated that the database 140 includes first metadata information corresponding to the first service 110 and second metadata information corresponding to the second service 120, and that the use of a single database to store multiple metadata facilitates management of all metadata in the cluster, as compared to the prior art in which multiple metadata are split and stored in multiple child nodes; especially when a plurality of metadata have an incidence relation, the operations of adding, deleting, changing and the like are carried out on a group of metadata with the incidence relation in the database, so that the metadata in the database can be ensured to be modified consistently, and the problem of data disorder is avoided; meanwhile, the database comprises all metadata information, so that when part or all of the metadata are required to be displayed, the display method is more convenient and faster compared with the prior art.
For example, when the distributed data processing system 100 described above is a Zookeeper cluster, the service is deployed in the Zookeeper cluster, and the root node 130 is a node of the Zookeeper cluster that is registered in order to implement application coordination of multiple services, and may be defined not to disappear due to disconnection of a user terminal connected to the Zookeeper cluster. The root node 130 may be created at the time of deployment of the first service 110 or may exist previously. The first sub-node 111 may be registered when the electronic device where the service is located is started, and when the electronic device fails or is actively turned off, and then the electronic device may be removed, the first service 110 may listen to the first sub-node 111, so as to obtain a node information change condition of the first sub-node 111.
For example, in a case that the Zookeeper cluster is deployed with a plurality of services, the reason for the change of the node data of the first child node 111 may be that the first service 110 acquires the node data of the first child node and empties the data in the first child node 111, or that the first child node 111 receives instruction information sent by other services in the Zookeeper cluster due to the change of the metadata.
Optionally, the node may also be implemented by other cluster systems, for example, Redis, which is a storage system with multiple data structures, such as an open source and supporting Key-Value (K-V) storage, and may be used in scenarios such as caching, event publishing or subscription, high-speed queue, and when the first service is deployed on an electronic device in the Redis, a process corresponding to a Key in the Redis is used as a first child node in the embodiment of the present invention, and the data processing method provided by the present invention may also be implemented; it is contemplated that other distributed clusters, such as Kafka, MQ, etc., may implement the data processing method provided by the present invention by deploying services on electronic devices on the cluster and creating nodes corresponding to the services.
The first service cache may be disposed on the electronic device where the first service is located, or may be disposed on another electronic device that the first service can access; the first service 110 and the database 140 may be on the same electronic device, or on different electronic devices, so that the first service 110 may access the database 140.
It should be understood that the distributed data processing system 100 described above may also be a child node with more services and corresponding services, and the embodiment of the present invention is taken as an example that the distributed data processing system 100 is deployed with two services, and should not be construed as limiting the present invention.
The following describes an exemplary data processing method provided by the embodiment of the present invention, taking the first service 110 as an example to perform cache update on the basis of the distributed data processing system 100 shown in fig. 2, and the purpose of the exemplary data processing method is to improve the data processing efficiency and implement synchronous cache of data. Optionally, an embodiment of the present invention provides a data processing method, please refer to fig. 3, and fig. 3 is a schematic flow chart of the data processing method according to the embodiment of the present invention. The data processing method comprises the following steps:
and S20, registering each child node under the root node of the distributed application program coordination component by the plurality of services with the cache respectively.
S21, when the first service receives the metadata modification request, the first service sends a first modification instruction corresponding to the metadata modification request to the database, so that the database modifies the first metadata information corresponding to the first modification instruction.
As can be appreciated, the database includes metadata information corresponding to the first service. For example, if a database stores a plurality of metadata tables with Identification (ID) fields as primary keys, in order to update the database, the metadata tables in the database corresponding to the ID fields of the first modification instruction may be modified by setting the ID fields of the objects in the first modification instruction.
S22, the first service updates the first metadata information in the first service cache according to the modified first metadata information in the database.
S23, the first service sends a first metadata adjustment instruction to the child nodes of other services.
It is understood that the manner of sending the message to the other child nodes by the first service may be unicast or broadcast, and the specific form may be determined according to the number of services actually deployed and the number of child nodes.
And S24, when other services obtain the first metadata adjustment instruction by monitoring the corresponding child nodes, other services update the first metadata information in the cache according to the first metadata adjustment instruction.
Each time the cache is updated, embodiments of the present invention may only update for the portion of the metadata that is modified. Namely, according to the modified first metadata information, updating the first metadata information of the corresponding part in the metadata stored in the first service cache so as to improve the efficiency of data processing. For example, the first metadata information that needs to be modified is determined by the object ID field. It can be understood that each service may cache corresponding metadata information according to the requirements of the service.
It should be noted that, the system for executing the above method flow is illustrated in fig. 3 only by the distributed data system including the first service and the second service. It is contemplated that for a distributed data processing system, it may deploy many more different services, which may be deployed on one or more electronic devices. The same or different metadata may be used between these services to implement the corresponding functionality. In this case, the first service in the steps shown in fig. 3 is one or more of a plurality of services.
The data processing method provided by the embodiment of the invention receives the metadata modification request through the first service, and the first service sends the first modification instruction corresponding to the metadata modification request to the database, so that the database updates the first metadata information corresponding to the first modification instruction. And the first service updates the first metadata information in the corresponding first service cache. Compared with the periodic cache updating, the method can keep the metadata information stored in the first service cache consistent with the latest database, and improve the accuracy of data processing.
Furthermore, only the first metadata information corresponding to the first modification instruction needs to be updated in the database, global modification is not needed, and only the first metadata information needs to be updated in the first service cache, so that all metadata in the database is prevented from being loaded into the first service cache, variable updating is realized, data processing capacity is reduced, and data processing efficiency of the cluster is improved.
Optionally, for the above metadata modification request, one possible implementation is given as follows:
the metadata modification request may be addition, deletion, or modification information, for example, when the metadata modification request is addition information, the purpose of the metadata modification request is to add new metadata information in the metadata table corresponding to the target ID field in the database, for example, a record with ID 1 in the metadata table of table _ test is added. The corresponding message may be defined as { "TYPE": CREATED "," TABLE _ NAME ": TABLE _ test", "RECORD _ ID": 1 ".
When the metadata modification request is modification information, for example, a record is updated, which aims to perform metadata adjustment on a metadata table corresponding to the target ID field in the database: a RECORD having an ID of 1 in the metadata TABLE of TABLE _ test is UPDATED, the message can be defined as { "TYPE": "UPDATED", "TABLE _ NAME": TABLE _ test "," RECORD _ ID ": 1".
When the above-described metadata modification request is deletion information, for example, a record is deleted with the purpose of deleting part or all of the metadata in the metadata table corresponding to the target ID field in the database: a RECORD having an ID of 1 in the metadata TABLE of TABLE _ test is DELETED, the message may be defined as { "TYPE": DELETED "," TABLE _ NAME ": TABLE _ test", "RECORD _ ID": 1 ", etc.; it is anticipated that in the above case, the message content of the metadata modification request may locate the metadata change to each record of each table, so that only the metadata information of the changed part needs to be updated when the cache is updated.
For the above process of sending the metadata adjustment instruction, when the distributed data processing system 100 is a Zookeeper cluster, after the first service 110 completes updating the first service cache corresponding to itself, there are two possible situations:
one situation is: the first service issues a message (a first metadata adjustment instruction) to other sub-nodes except the first sub-node under the same root node in the Zookeeper cluster, the message format can be a Json format, the message content is the change condition of data in the database, and other services monitor whether the node data change occurs in the respective corresponding sub-node, so that when the service monitors that the node data of the sub-node corresponding to the service itself changes, the node data corresponding to the service itself is read, the node data is analyzed, and the latest metadata information is read from the database according to the requirements of the service itself to update the cache itself, thereby realizing synchronization of a plurality of service caches and the database.
The other situation is that: maintaining the condition of cache storage metadata information of services corresponding to all child nodes under the root node in the first service, judging whether first metadata information is stored in a second service cache of a second service cache by the first service after the first service completes updating of the first service cache, and sending a first metadata adjusting instruction to a second child node corresponding to the second service by the first service when the first metadata information is stored in the second service cache; when the second service cache does not store the first metadata information, the first service does not send the first metadata adjustment instruction to the second child node.
Next, on the basis of the distributed data processing system 100 shown in fig. 2, taking cache update by the first service 110 as an example, a data processing method provided in the embodiment of the present invention is exemplarily described, and it is intended that when the first service does not receive a metadata modification request, and metadata information in a database is updated, in order to synchronize the cache of the first service with corresponding metadata information in the metadata, taking other services as the second service 120 shown in fig. 2 as an example, an embodiment of the present invention provides another data processing method, please refer to fig. 4, and fig. 4 is a flowchart of another data processing method provided in the embodiment of the present invention. The data processing method further comprises the following steps:
s25, the first service monitors the first child node.
The first child node is a child node created by the first service under the parent node.
Specifically, for example, taking the Zookeeper cluster as an example, the first child node may be a child node created by the first service under the parent node, and the first service may monitor data change of the first child node, so that the first service acquires progress information of executing other services by the Zookeeper cluster. Further, when the first service monitors the data change of the first child node, in order to match the progress of other services, the first service may further obtain node information of the first child node, so as to execute a process corresponding to the node information, so that the first service is matched with other services in the Zookeeper cluster.
S26, when the first child node has a second metadata adjustment instruction sent by the second service, the first service updates the second metadata information stored in the first service cache according to the modified second metadata information in the database.
Referring to the distributed data processing system shown in fig. 2, when the second service 120 receives the metadata modification request, the second service 120 sends a second adjustment instruction corresponding to the metadata modification request to the database 140, so that the database 140 updates the second metadata information stored in the database 140 according to the second adjustment instruction, and the second service sends a second metadata adjustment instruction to the first child node 111, where the second metadata adjustment instruction is used to indicate that the second metadata information of the database is modified. When the first child node 111 receives the second metadata adjustment instruction, the second metadata adjustment instruction needs to be stored; correspondingly, the first service 110 continuously listens to the first child node 111, and once it is known that the first child node 111 has the second metadata adjustment instruction sent by the second service 120, the first service 110 considers that the data stored in the first child node 111 has changed, and the first service 110 obtains node information on the first child node 111, where the node information is the data stored in the first child node 111 and includes at least the second metadata adjustment instruction: if the first child node stores the data to be processed before the second metadata adjustment instruction is acquired, the first service executes the process corresponding to the second metadata adjustment instruction to update the first service cache after the process corresponding to the data to be processed is executed; if the first child node does not store the data to be processed before the second metadata adjustment instruction is acquired, the first service executes a process corresponding to the second metadata adjustment instruction to update the first service cache.
Taking the distributed data processing system 100 shown in fig. 2 as an example, the data to be processed may already exist on the first child node 111 before receiving the second metadata adjustment instruction, or may not exist on the first child node, and in order to update the first service cache, a possible implementation manner is provided on the basis of fig. 4, please refer to fig. 5, where fig. 5 is a flowchart of another data processing method provided in the embodiment of the present invention. The above S26 may include:
s261, when the first child node has the second metadata adjustment instruction, the first service determines whether the second metadata information corresponding to the second metadata adjustment instruction is stored in the first service cache.
If yes, go to S262; if not, go to S263.
S262, the first service modifies the second metadata information stored in the first service cache according to the modified second metadata information in the database.
S263, discarding the second metadata adjustment instruction of the first child node.
It can be understood that there are two possible cases of discarding the node information of the first child node as described above:
one of the situations is: when the first child node itself does not have data to be processed, that is, the node information of the first child node before receiving the second metadata adjustment instruction is empty, and meanwhile, the first service cache corresponding to the first service does not store the second metadata information corresponding to the second metadata adjustment instruction, that is, when the second metadata information is not needed in the process of implementing the first service, the first service may directly discard the obtained second metadata adjustment instruction.
In another case: when the first child node has data to be processed before receiving the second metadata adjustment instruction, taking the first metadata adjustment instruction sent by the service other than the second service as an example, the first metadata adjustment instruction is used to indicate that the first metadata information in the database is modified. The first metadata adjustment instruction is unrelated to the metadata to be modified by the second metadata adjustment instruction. At this time, in order to ensure the consistency between the first service cache and the database, it is necessary to determine whether to execute the operation associated with the second metadata adjustment instruction after the first service finishes executing the action associated with the first metadata adjustment instruction. For example, when the node information obtained by the first service is empty, it is characterized that the first service has completed processing the data to be processed. At this time, the first service determines whether to discard the second metadata adjustment instruction according to the second metadata adjustment instruction (i.e., after updating the corresponding first metadata information in the first service cache), and when the first service is implemented and the second metadata information is not needed, discards the obtained second metadata adjustment instruction, so as to reduce useless data in the first service and effectively utilize resources of the first service.
Specifically, for example, when the first service monitors that the Data of the first child Node created by the first service changes (Node Data Changed event), the Node information of the first child Node is read, where the Node information is Data stored by the first child Node, and in the Zookeeper cluster, the Node information may represent to-be-processed Data that matches the first service with other service processes in the Zookeeper cluster. First, the first service determines whether the first child node data is empty according to the node information, for example, the node information acquired by the first service may be a character string of "00000000", and in the first service, when the node information is "00000000", the first child node data is defined as empty; if the data is empty, no processing is carried out, and if the data is not empty, a service process corresponding to the node information is executed according to the acquired node information; further, taking the Zookeeper cluster as an example, when the first child node is a Znode, since the Znode can generally store only 1MB of data, in order to effectively utilize the storage space of the first child node, after the first service acquires and executes the node information, the node information in the first child node may be set to be null, for example, the node information is set to be a character string of "00000000".
It is understood that the distributed data processing system 100 described above may further include more services, and a possible implementation manner is given on the basis of the distributed data processing system 100 shown in fig. 2, please refer to fig. 6, and fig. 6 is a schematic structural diagram of another multi-service metadata cache provided in the embodiment of the present invention. The distributed data processing system 100 also includes a third service 150 and a third child node 151.
The third child node 151 is a child node created by the third service 150 under the root node 130, and the third service 150 may listen to the third child node 151 to sense a data change of the third child node.
If the first child node 111 receives the third metadata adjustment instruction sent by the third service 150 while receiving the second metadata adjustment instruction, the third metadata adjustment instruction is used to indicate that the third metadata information in the database is modified: the data stored in the first child node 111 may be the second metadata adjustment instruction and the third metadata adjustment instruction, which are separated by a separator according to the received time sequence, the first service 110 obtains node information of the first child node 111, where the node information is the data stored in the first child node 111, and includes the second metadata adjustment instruction and the third metadata adjustment instruction, and the first service 110 sequentially parses each metadata adjustment instruction in the node information, so that data disorder can be avoided when the first child node receives metadata adjustment instructions sent by multiple services; meanwhile, when the cache is updated, only the first service cache corresponding to the first service is needed to be updated, and all metadata in the database is not needed to be loaded into the first service cache of the first service, so that the redundancy of data processing is reduced, and the efficiency of data processing is improved.
In the Zookeeper cluster, since the node data in each Znode has a version number, when the node data of the Znode needs to be modified, the data version number needs to be used to implement successful submission of new data, next, on the basis of the distributed data processing system 100 shown in fig. 2, in order to implement normal transmission of metadata instructions, taking the data stored in the second child node as an example that the data has version information, on the basis of fig. 3, a possible implementation manner is given, please refer to fig. 7, and fig. 7 is a flowchart of another data processing method provided by the embodiment of the present invention. The above S23 may include:
s231, the first service acquires version information of the second child node.
S232, the first service generates a first metadata adjusting instruction consistent with the version information.
S233, the first service sends the first metadata adjustment instruction to the second child node.
For example, before modifying the node data, the node data version number (version information) in the Zookeeper cluster may read the data and record the data version number, and when the node data needs to be updated, the node data version number may be carried to submit, and if the carried version number (that is, the version number read last time) is the same as the version number of the current node, it indicates that the node data is not modified, the submission may be successful; if the commit fails, the node data is modified in the period after the read and before the commit. For example, on the basis of the first metadata adjustment instruction, by using a set command to submit an update to the version number carried by the first metadata adjustment instruction, the update with the same version number as that of the second child node is successful, otherwise, the update fails.
It can be understood that, taking the distributed data processing system 100 shown in fig. 6 as an example below, in order to solve the problem of concurrently writing the metadata adjustment instruction and ensure that each metadata adjustment instruction is correctly analyzed and processed by the service without missing, when the second service 120 and the third service 150 issue the metadata adjustment instruction, the metadata adjustment instruction may be submitted to the first child node 111 according to the version information, and finally, the first service cache update corresponding to the first service is implemented.
Next, on the basis of the distributed data processing system 100 shown in fig. 6, in order to solve the problem of concurrent write messages and ensure that each metadata adjustment instruction is correctly analyzed and processed by the service without omission when the first child node 111 receives the second metadata adjustment instruction and also receives a third metadata adjustment instruction sent by a third service at the same time, on the basis of fig. 7, a possible implementation manner is provided, please refer to fig. 8, and fig. 8 is a flowchart of another data processing method provided in the embodiment of the present invention. The above S233 may include:
s233a, the first service acquires node data of the second child node and empties the node data.
S233b, the first service generates a write command.
The write command includes node data and a first metadata adjustment command.
S233c, the second child node receives and stores the write command.
For example, the first service obtains current node data and version information from the second child node, taking the current node data as a third metadata adjustment instruction as an example: the first service appends a new message (first metadata adjustment instruction) to the retrieved third metadata adjustment instruction with a specific symbol interval, resulting in data (write instruction) to be written to the second child node; and writing the write-in instruction into the second child node by carrying the obtained version information, and finally realizing accurate issuing of the metadata adjusting instruction, and realizing correct analysis processing without omission. It can be understood that when the first metadata adjustment instruction is sent, by means of additional issuing, unnecessary waste of service resources caused by repeated execution of a certain service process is avoided, and data processing efficiency is improved.
In an optional embodiment, taking the distributed data processing system 100 shown in fig. 2 as an example below, the data to be processed may already exist on the second child node 121 before the first metadata adjustment instruction is received, or the data to be processed may not exist, and in order to update the second cache, the second service 120 may also implement a process similar to the data flow method of S261 to S263, which is not described herein again.
Next, on the basis of the data system shown in fig. 6, when the second child node 121 receives metadata adjustment instructions sent by multiple services, for example, when the second child node 121 receives a first metadata adjustment instruction sent by a first service and a third metadata adjustment instruction sent by a third service 150, to avoid data disturbance, a similar process to that of the above S223a, S223b, and S223c may be used to process the write instruction, which is not described herein again.
When the second service updates the second service cache every time, only the first metadata information corresponding to the first modification instruction needs to be updated in the database, global modification is not needed, and only the first metadata information needs to be updated in the second service cache, so that all metadata in the database are prevented from being loaded into the second service cache, variable updating is realized, data processing capacity is reduced, and data processing efficiency of the cluster is improved.
It is foreseeable that after each service has updated its own cache, the service may also process subsequent business processes according to its corresponding child node data, and wait for the next metadata modification request or metadata adjustment instruction, so as to implement the update of the database and the cache.
For the first service 110 shown in fig. 2, when the first service does not receive a metadata modification request, and metadata information in the database is updated, in order to synchronize the first service cache with corresponding metadata information in the metadata, on the basis of the first service 110 shown in fig. 2, another data processing method is provided in an embodiment of the present invention, please refer to fig. 9, and fig. 9 is a flowchart of another data processing method provided in an embodiment of the present invention. The data processing method further comprises the following steps:
s60, the first service monitors the first child node.
The first child node is a child node created by the first service under the parent node.
S61, when the first child node has the first metadata adjustment instruction sent by the second service, the first service updates the first metadata information stored in the first service cache according to the modified first metadata information in the database.
The first metadata adjustment instruction is to indicate that first metadata information of a database has been modified.
The first service monitors whether the node data of the first sub-node corresponding to the first service changes or not, so that when the first service monitors that the node data of the first sub-node corresponding to the first service changes, the node data of the first sub-node corresponding to the first service is read, the node data is analyzed, the latest first metadata information is read from a database according to the requirement of the first service, and a first service cache is updated, so that the cache and the database of a plurality of services are synchronized.
It should be noted that, for the distributed data processing system 100 shown in fig. 2, the first service 110 and the second service 120 may be deployed on the same electronic device, or may be deployed on different electronic devices, for example, the first service 110 is deployed on a first electronic device, and the second service 120 is deployed on a second electronic device. It should be understood that the distributed data processing system 100 shown in fig. 2 may implement the data processing method proposed by any of the above embodiments.
For any of the electronic devices proposed above, which may implement any of the data processing methods provided by the present invention, the electronic device may be, but is not limited to, a mobile phone, a tablet computer, a notebook computer, a server, or other electronic devices with processing capabilities.
An embodiment of the present invention provides an electronic device, and as shown in fig. 10, fig. 10 is a block schematic diagram of an electronic device provided in an embodiment of the present invention. The electronic device 70 comprises a memory 71, a processor 72 and a communication interface 73. The memory 71, processor 72 and communication interface 73 are electrically connected to each other, directly or indirectly, to enable transmission or interaction of data. For example, the components may be electrically connected to each other via one or more communication buses or signal lines. The memory 71 can be used for storing software programs and modules, such as program instructions/modules corresponding to the data processing method provided by the embodiment of the present invention, and the processor 72 executes the software programs and modules stored in the memory 71, so as to execute various functional applications and data processing. The communication interface 73 may be used for communication of signaling or data with other node devices. The electronic device 70 may have a plurality of communication interfaces 73 in the present invention.
The Memory 71 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.
The processor 72 may be an integrated circuit chip having signal processing capabilities. The Processor may be a general-purpose Processor including a Central Processing Unit (CPU), a Network Processor (NP), etc.; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc.
An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the data processing method according to any one of the foregoing embodiments. The computer readable storage medium may be, but is not limited to, various media that can store program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a PROM, an EPROM, an EEPROM, a magnetic or optical disk, etc.
In summary, the present invention provides a data processing method, a system and a computer readable storage medium, which relate to the field of cluster data processing. The method is applied to a first service and comprises the following steps: when the first service receives the metadata modification request, the first service sends a first modification instruction corresponding to the metadata modification request to the database so that the database modifies first metadata information corresponding to the first modification instruction; the first service updates the first metadata information in the first service cache according to the modified first metadata information in the database; the first service sends a first metadata adjusting instruction to child nodes of other services; and when other services acquire the first metadata adjusting instruction by monitoring the corresponding child nodes, updating the first metadata information in the cache according to the first metadata adjusting instruction. The first service receives the metadata modification request and sends a first modification instruction corresponding to the metadata modification request to the database, so that the database updates first metadata information corresponding to the first modification instruction, the first service updates the first metadata information in a corresponding first service cache, the metadata information stored in the first service cache is consistent with the latest database, and the accuracy of data processing is improved.
In addition, the functional modules in the embodiments of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A data processing method applied to a first service, the first service being provided in a distributed data processing system, the distributed data processing system including a database, a distributed application coordination component, and a plurality of services with caches, the first service serving any one of the plurality of services with caches, the method comprising:
the plurality of services with caches register respective child nodes under a root node of the distributed application coordination component;
when the first service receives a metadata modification request, the first service sends a first modification instruction corresponding to the metadata modification request to the database, so that the database modifies first metadata information corresponding to the first modification instruction;
wherein the database comprises first metadata information corresponding to the first service;
the first service updates the first metadata information in the first service cache according to the modified first metadata information in the database;
the first service sends a first metadata adjusting instruction to child nodes of other services;
when the other services obtain the first metadata adjusting instruction by monitoring the corresponding child nodes, the other services update the first metadata information in the caches according to the first metadata adjusting instruction; the first metadata adjustment instruction is used for indicating that first metadata information of the database is modified.
2. The method of claim 1, wherein the distributed data processing system further comprises a first child node corresponding to the first service, wherein the first child node is a child node registered under the root node for the first service, wherein the database further comprises second metadata information corresponding to the other services, and wherein the method further comprises:
the first service monitors the first child node;
when the first child node has a second metadata adjustment instruction sent by the other service, the first service updates the second metadata information stored in the first service cache according to the modified second metadata information in the database; the second metadata adjustment instruction is used for indicating that second metadata information of the database is modified.
3. The method according to claim 2, wherein when the first child node has a second metadata adjustment instruction sent by the other service, the first service updating the second metadata information stored in the first service cache according to the modified second metadata information in the database, including:
when the first child node has the second metadata adjustment instruction, the first service judges whether second metadata information corresponding to the second metadata adjustment instruction is stored in the first service cache or not;
if so, the first service modifies the second metadata information stored in the first service cache according to the modified second metadata information in the database;
and if not, discarding the second metadata adjustment instruction of the first child node.
4. The method of claim 1, wherein the distributed data processing system further includes a second service and a second child node corresponding to the second service, the second service is any one of the other services, the second child node is a child node registered by the second service under the root node, and the sending, by the first service, the first metadata adjustment instruction to the child node of the other service includes:
the first service acquires version information of the second child node;
the first service generates the first metadata adjustment instruction consistent with the version information;
the first service sends the first metadata adjustment instruction to the second child node.
5. The method of claim 4, wherein the first service sending the first metadata adjustment instruction to the second child node comprises:
the first service acquires current node data and version information from the second child node; the current node data is a third element data adjusting instruction;
the first service adds the first metadata adjustment instruction to the back of the third metadata adjustment instruction by using a specific symbol interval to obtain a write-in instruction to be written in the second child node;
and the first service sends a writing instruction carrying the version information to a second child node so that the second child node receives and stores the writing instruction.
6. A distributed data processing system, comprising: a database, a distributed application coordination component, and a plurality of services having caches;
the plurality of services with the cache register respective child nodes under the root node corresponding to the distributed application program coordination component;
the system comprises a first service and a second service, wherein the first service is used for sending a first modification instruction corresponding to a metadata modification request to the database when the first service receives the metadata modification request; wherein the first service is any one of the plurality of services with cache;
the database is used for modifying the first metadata information corresponding to the first modification instruction; the database comprises first metadata information corresponding to the first service;
the first service is further configured to update the first metadata information in the first service cache according to the modified first metadata information in the database;
the first service is further configured to send a first metadata adjustment instruction to child nodes of other services, so that when the other services obtain the first metadata adjustment instruction by monitoring the respective corresponding child nodes, the other services update the first metadata information in respective caches according to the first metadata adjustment instruction, where the first metadata adjustment instruction is used to indicate that the first metadata information of the database is modified.
7. The system of claim 6, wherein the distributed data processing system further comprises a first child node corresponding to the first service, the first child node being a child node registered under the root node for the first service, and wherein the database further comprises second metadata information corresponding to the other services;
the first service is also used for monitoring the first child node;
the first service is further configured to update the second metadata information stored in the first service cache according to the modified second metadata information in the database when the first child node has a second metadata adjustment instruction sent by the other service; the second metadata adjustment instruction is used for indicating that second metadata information of the database is modified.
8. The system of claim 6, wherein the distributed data processing system further comprises a second service and a second child node corresponding to the second service, wherein the second service is any one of the other services, and the second child node is a child node registered by the second service under the root node;
the first service is further to:
acquiring version information of the second child node;
generating the first metadata adjustment instruction consistent with the version information;
and sending the first metadata adjusting instruction to the second child node.
9. The system of claim 8,
the first service is also used for acquiring current node data and version information from the second child node; the current node data is a third element data adjusting instruction;
the first service is further configured to append the first metadata adjustment instruction to the third metadata adjustment instruction with a specific symbol interval, so as to obtain a write instruction to be written into the second child node;
and the first service is also used for sending a writing instruction carrying the version information to a second child node so as to enable the second child node to receive and store the writing instruction.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911292296.8A CN111143382B (en) | 2019-12-13 | 2019-12-13 | Data processing method, system and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911292296.8A CN111143382B (en) | 2019-12-13 | 2019-12-13 | Data processing method, system and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111143382A CN111143382A (en) | 2020-05-12 |
| CN111143382B true CN111143382B (en) | 2022-08-09 |
Family
ID=70518419
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911292296.8A Active CN111143382B (en) | 2019-12-13 | 2019-12-13 | Data processing method, system and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111143382B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111984380B (en) * | 2020-08-21 | 2024-07-30 | 北京金山云网络技术有限公司 | Stream computing service system and control method and device thereof |
| CN112667650B (en) * | 2020-12-28 | 2024-05-28 | 北京华大智宝电子系统有限公司 | Database updating method and device |
| CN112905699B (en) * | 2021-02-23 | 2024-04-09 | 京东科技信息技术有限公司 | Full data comparison method, device, equipment and storage medium |
| CN113312404B (en) * | 2021-08-02 | 2021-11-02 | 北京华品博睿网络技术有限公司 | Method and system for collecting characteristic samples in real time |
| CN114945026A (en) * | 2022-04-24 | 2022-08-26 | 网易（杭州）网络有限公司 | Data processing method, device and system |
| CN115277735B (en) * | 2022-07-20 | 2023-11-28 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN116226203A (en) * | 2023-05-05 | 2023-06-06 | 北京零壹视界科技有限公司 | Single object data caching method and device, electronic equipment and storage medium |
| CN117056379B (en) * | 2023-10-11 | 2024-01-26 | 宁波银行股份有限公司 | Metadata caching method and device, electronic equipment and readable storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8015355B1 (en) * | 2005-11-01 | 2011-09-06 | Netapp, Inc. | System and method for managing hard lock state information in a distributed storage system environment |
| CN103294701A (en) * | 2012-02-24 | 2013-09-11 | 联想(北京)有限公司 | Distributed file system and data processing method |
| US9244958B1 (en) * | 2013-06-13 | 2016-01-26 | Amazon Technologies, Inc. | Detecting and reconciling system resource metadata anomolies in a distributed storage system |
| US9367301B1 (en) * | 2013-03-06 | 2016-06-14 | Attivio Inc. | Dynamic update of a distributed message processing system |
| CN109213792A (en) * | 2018-07-06 | 2019-01-15 | 武汉斗鱼网络科技有限公司 | Method, server-side, client, device and the readable storage medium storing program for executing of data processing |
| CN109815248A (en) * | 2019-01-15 | 2019-05-28 | 科大国创软件股份有限公司 | A kind of distributed structure/architecture data consistency method based on Zookeeper |
| WO2019127916A1 (en) * | 2017-12-29 | 2019-07-04 | 北京奇虎科技有限公司 | Data read/write method and device implemented on the basis of distributed consensus protocol |
| CN110046029A (en) * | 2019-03-13 | 2019-07-23 | 平安科技（深圳）有限公司 | Data processing method and device applied to multi-level buffer in cluster |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10585873B2 (en) * | 2017-05-08 | 2020-03-10 | Sap Se | Atomic processing of compound database transactions that modify a metadata entity |
| US10467112B2 (en) * | 2017-11-09 | 2019-11-05 | Bank Of America Corporation | Distributed data monitoring device |
-
2019
- 2019-12-13 CN CN201911292296.8A patent/CN111143382B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8015355B1 (en) * | 2005-11-01 | 2011-09-06 | Netapp, Inc. | System and method for managing hard lock state information in a distributed storage system environment |
| CN103294701A (en) * | 2012-02-24 | 2013-09-11 | 联想(北京)有限公司 | Distributed file system and data processing method |
| US9367301B1 (en) * | 2013-03-06 | 2016-06-14 | Attivio Inc. | Dynamic update of a distributed message processing system |
| US9244958B1 (en) * | 2013-06-13 | 2016-01-26 | Amazon Technologies, Inc. | Detecting and reconciling system resource metadata anomolies in a distributed storage system |
| WO2019127916A1 (en) * | 2017-12-29 | 2019-07-04 | 北京奇虎科技有限公司 | Data read/write method and device implemented on the basis of distributed consensus protocol |
| CN109213792A (en) * | 2018-07-06 | 2019-01-15 | 武汉斗鱼网络科技有限公司 | Method, server-side, client, device and the readable storage medium storing program for executing of data processing |
| CN109815248A (en) * | 2019-01-15 | 2019-05-28 | 科大国创软件股份有限公司 | A kind of distributed structure/architecture data consistency method based on Zookeeper |
| CN110046029A (en) * | 2019-03-13 | 2019-07-23 | 平安科技（深圳）有限公司 | Data processing method and device applied to multi-level buffer in cluster |
Non-Patent Citations (3)
| Title |
|---|
| 基于协同缓存的分布式数据库更新机制研究;符青云等;《计算机工程》;20070405(第07期);47-49 * |
| 大规模游戏平台云缓存技术研究与实践;陈杰等;《移动通信》;20160315(第05期);5-9 * |
| 网络文件系统的数据一致性研究;刘湛等;《舰船电子工程》;20080720(第07期);157-161 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111143382A (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111143382B (en) | Data processing method, system and computer readable storage medium | |
| CN112069265B (en) | Synchronization method of configuration data, business data system, computer system and medium | |
| CN108874558B (en) | Message subscription method of distributed transaction, electronic device and readable storage medium | |
| CN110515912A (en) | Log processing method, device, computer installation and computer readable storage medium | |
| US10860604B1 (en) | Scalable tracking for database udpates according to a secondary index | |
| US10127077B2 (en) | Event distribution pattern for use with a distributed data grid | |
| CN106874334B (en) | Data processing method and device and information processing system | |
| CN111881223B (en) | Data management method, device, system and storage medium | |
| CN107977396B (en) | Method and device for updating data table of KeyValue database | |
| CN115185705A (en) | Message notification method, device, medium and equipment | |
| JP2012234333A (en) | Cluster system, synchronization control method, server device and synchronization control program | |
| CN115934855A (en) | Full-link field level blood margin analysis method, system, equipment and storage medium | |
| CN113704790A (en) | Abnormal log information summarizing method and computer equipment | |
| WO2016082594A1 (en) | Data update processing method and apparatus | |
| CN111338834B (en) | Data storage method and device | |
| CN113032477B (en) | Long-distance data synchronization method and device based on GTID and computing equipment | |
| CN110798358B (en) | Distributed service identification method and device, computer readable medium and electronic equipment | |
| CN112015534A (en) | Configurated platform scheduling method, system and storage medium | |
| US20240028663A1 (en) | Web site generation based on streams of data records | |
| WO2024103898A1 (en) | Database cluster management method and apparatus | |
| CN109165259B (en) | Index table updating method based on network attached storage, processor and storage device | |
| CN114860782B (en) | Data query method, device, equipment and medium | |
| CN115189931A (en) | Distributed key management method, device, equipment and storage medium | |
| CN117827854A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN111061719B (en) | Data collection method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |