CN109284177B - Data updating method and device - Google Patents
Data updating method and device Download PDFInfo
- Publication number
- CN109284177B CN109284177B CN201710594880.3A CN201710594880A CN109284177B CN 109284177 B CN109284177 B CN 109284177B CN 201710594880 A CN201710594880 A CN 201710594880A CN 109284177 B CN109284177 B CN 109284177B
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- execution machine
- authority
- processing authority
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5022—Mechanisms to release resources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5018—Thread allocation
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data updating method and device, and relates to the technical field of computers. One embodiment of the method comprises: distributing data corresponding to the same service number in the input data to the same execution machine by using a preset algorithm; configuring the processing authority of the data in each execution machine according to a preset rule, wherein the preset rule is as follows: only one item of data corresponding to the same service number at the same time is configured with the processing authority; transmitting, by each execution engine, respective data configured with the processing privilege to a data storage cluster to perform a data update process; when the data configured with the processing authority in an execution machine is executed with the data updating processing, the execution machine releases the processing authority of the data. According to the embodiment, on the premise of ensuring correct updating of data, processing delay is reduced, and system throughput is improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data updating method and apparatus.
Background
Nowadays, big data is applied more and more in actual business, and technologies related to data processing are also rapidly developed. In a big data scene, in order to match the processing speeds of different systems, a pipeline is usually adopted to cache and transfer data, and in order to improve the processing capability of the system, a multi-instance parallel mode is adopted to process data in practical application, which increases the processing speed of the system, but also brings a problem of synchronous update of data.
In e-commerce, order status update is a common business scenario, a business party always wants to obtain the latest status of an order, and multiple pieces of update information of order status may be processed simultaneously on multiple instances, so the system ensures that the order status is updated correctly. The existing technical scheme adopts a method for updating the whole order or carries out transaction processing on each order, namely transaction locking. The method comprises the steps of performing locking on an entire order updating method, converting parallel processing into serial processing, considering locking and unlocking expenses, performing locking on each order, requiring locking and unlocking operations on each order, having high performance expense, and simultaneously locking the same order by different instances, which relates to distributed locking, so that the complexity of the system is increased.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
the existing scheme can cause the processing capacity of the system to be greatly reduced, and the delay of state updating is increased.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data updating method and apparatus, which can reduce processing delay and improve system throughput on the premise of ensuring correct data updating.
To achieve the above object, according to an aspect of an embodiment of the present invention, a data updating method is provided.
A data update method, comprising: distributing data corresponding to the same service number in the input data to the same execution machine by using a preset algorithm; configuring the processing authority of the data in each execution machine according to a preset rule, wherein the preset rule is as follows: only one item of data corresponding to the same service number at the same time is configured with the processing authority; transmitting, by each execution engine, respective data configured with the processing privilege to a data storage cluster to perform a data update process; when the data configured with the processing authority in an execution machine is executed with the data updating processing, the execution machine releases the processing authority of the data.
Optionally, the preset algorithm is a hash algorithm or a consistent hash algorithm.
Optionally, the step of configuring, in each execution machine, the processing authority of the data in the execution machine according to a preset rule includes: and configuring the processing authority of each item of data one by one according to the time of the data reaching the execution machine, wherein when the processing authority of each item of data is configured, the following steps are executed: step one, judging whether other data with the same service number as the data has the processing authority or not, if so, executing step two, and if not, executing step three; step two, waiting for a preset time interval, and then returning to execute the step one; and step three, configuring the data with the processing authority.
Optionally, the input data includes time stamp information, and the step of executing the data update process includes: for each item of data transmitted to the data storage cluster, the data storage cluster judges whether stored data with the same service number as the transmitted data exists or not; if so, comparing the timestamp information of the stored data and the transmitted data, and taking the latest timestamp data in the stored data and the transmitted data as new stored data; and if not, storing the transmitted data into the data storage cluster.
Optionally, the step of transmitting, by each execution machine, respective data configured with the processing permission to a data storage cluster, where the processing permission is a permission to use a thread, includes: transmitting, by each execution machine, data configured with respective authority to use a thread to a data storage cluster through a randomly assigned thread, the step of the execution machine releasing the processing authority of the data, comprising: the step of clearing the occupation of the data to the thread by the execution machine and judging whether data is configured with the processing authority in other data with the same service number as the data includes the following steps: and judging whether other data with the same service number as the data occupies the thread or not.
Optionally, a multithreading hash set built in Java is used in each execution machine to configure the processing authority of the data in the execution machine according to a preset rule.
According to another aspect of an embodiment of the present invention, there is provided a data updating apparatus.
A data update apparatus comprising: the data distribution module is used for distributing data corresponding to the same service number in the input data to the same execution machine by using a preset algorithm; the authority configuration module is used for configuring the processing authority of the data in each execution machine according to preset rules, wherein the preset rules are as follows: only one item of data corresponding to the same service number at the same time is configured with the processing authority; the data transmission module is used for transmitting the respective data configured with the processing authority to the data storage cluster through each execution machine so as to execute data updating processing; and the authority releasing module is used for releasing the processing authority of the data after the data configured with the processing authority in an execution machine is executed by the data updating processing.
Optionally, the preset algorithm is a hash algorithm or a consistent hash algorithm.
Optionally, the permission configuration module is further configured to: and configuring the processing authority of each item of data one by one according to the time of the data reaching the execution machine, wherein when the processing authority of each item of data is configured, the following steps are executed: step one, judging whether other data with the same service number as the data has the processing authority or not, if so, executing step two, and if not, executing step three; step two, waiting for a preset time interval, and then returning to execute the step one; and step three, configuring the data with the processing authority.
Optionally, the input data includes time stamp information, and the step of executing the data update process includes: for each item of data transmitted to the data storage cluster, the data storage cluster judges whether stored data with the same service number as the transmitted data exists or not; if so, comparing the timestamp information of the stored data and the transmitted data, and taking the latest timestamp data in the stored data and the transmitted data as new stored data; and if not, storing the transmitted data into the data storage cluster.
Optionally, the processing right is a right to use a thread, and the data transfer module is further configured to: transmitting, by each execution machine, data configured with respective permissions to use threads to the data storage cluster via randomly assigned threads, the permission release module further configured to: the execution machine clears the occupation of the data on the thread, and the permission configuration module is further used for: and judging whether other data with the same service number as the data occupies the thread or not.
Optionally, the permission configuration module is further configured to: and configuring the processing authority of the data in each execution machine according to a preset rule by using a multithread hash set built in Java in each execution machine.
According to yet another aspect of an embodiment of the present invention, a server is provided.
A server, comprising: one or more processors; a memory for storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement a data update method.
According to yet another aspect of an embodiment of the present invention, a computer-readable medium is provided.
A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out a data updating method.
One embodiment of the above invention has the following advantages or benefits: distributing data corresponding to the same service number in the input data to the same execution machine by using a preset algorithm; configuring the processing authority of the data in each execution machine according to a preset rule, wherein the preset rule is that only one data in the data corresponding to the same service number is configured with the processing authority at the same time; transmitting the respective data configured with the processing authority to the data storage cluster through each execution machine to execute data updating processing; when data configured with processing authority in an execution machine is executed with data updating processing, the execution machine releases the processing authority of the data. On the premise of ensuring correct updating of data, processing delay is reduced, and system throughput is improved.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of the main steps of a data update method according to an embodiment of the present invention;
FIG. 2 is a block diagram of an order status data update method according to an embodiment of the invention;
FIG. 3 is a schematic diagram of an order status data update flow for distributing order status data based on a consistent hashing algorithm according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of the main modules of a data update apparatus according to an embodiment of the present invention;
FIG. 5 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
FIG. 6 is a schematic block diagram of a computer system suitable for use with a server implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of main steps of a data updating method according to an embodiment of the present invention.
As shown in fig. 1, the data updating method according to the embodiment of the present invention mainly includes the following steps S101 to S104.
Step S101: and distributing the data corresponding to the same service number in the input data to the same execution machine by using a preset algorithm.
The input data may include timestamp information, and the data updating in the embodiment of the present invention mainly updates stored data to data with the latest timestamp according to the input data, so the data updating method in the embodiment of the present invention is applicable to various data updating scenarios, for example, an order status data updating scenario in the e-commerce field, and accordingly, the input data may be order status data, and the service number may be an order number.
The preset algorithm may be a hash algorithm or a consistent hash algorithm. It should be noted that the hash algorithm described herein may be various hash algorithms other than the consistent hash algorithm.
Step S102: and configuring the processing authority of the data in each execution machine according to a preset rule.
The processing authority of each item of data can be configured one by one according to the time when the data reaches the execution machine, wherein when the processing authority of each item of data is configured, the following steps are executed:
step one, judging whether other data with the same service number as the data has processing authority or not, if so, executing step two, and if not, executing step three;
step two, waiting for a preset time interval, and then returning to execute the step one;
and step three, configuring the data with processing authority.
Wherein, the preset rule is as follows: only one item of data corresponding to the same service number at the same time is configured with processing authority.
In each execution machine, a multithread hash set built in Java can be used for configuring the processing authority of the data in the execution machine according to a preset rule.
The processing right may specifically be a right to use a thread.
Step S103: and transmitting the respective data configured with the processing authority to the data storage cluster through each execution machine to execute data updating processing.
The step of transmitting the respective data configured with the processing authority to the data storage cluster through each execution machine may specifically include: data configured with respective permissions to use the threads is communicated by each of the execution machines to the data storage cluster via the randomly assigned threads.
After an item of data is configured with the authority to use a thread, the thread can be randomly allocated to process the item of data, the thread processing the item of data mainly transfers the item of data to a data storage cluster and performs data updating processing in the data storage cluster, specifically, the thread can call a method for performing data updating processing in the data storage cluster, transfer the data to the data storage cluster, and perform the method by the data storage cluster to realize the data updating processing.
A data storage cluster such as a Redis cluster.
The step of executing the data update process may specifically include: for each item of data transmitted to the data storage cluster, the data storage cluster judges whether stored data with the same service number as the transmitted data exists or not; if so, comparing the timestamp information of the stored data and the transmitted data, and taking the latest timestamp data in the stored data and the transmitted data as new stored data; if not, the transmitted data is stored in the data storage cluster.
Step S104: when data configured with processing authority in an execution machine is executed with data updating processing, the execution machine releases the processing authority of the data.
The processing authority of the execution machine for releasing the data is mainly to clear the occupation of the data on the thread.
The step of determining whether data has been configured with processing permission in other data with the same service number as the data in step S102 may specifically include: and judging whether other data with the same service number as the data occupies the thread or not.
The main flow of the data updating method according to the embodiment of the present invention is described in detail below by taking the input data as order status data and the service number as an order number.
Fig. 2 is a frame diagram of an order status data updating method according to an embodiment of the present invention. As shown in fig. 2, taking the input data as order status data as an example, the order status data updating method according to the embodiment of the present invention may be executed by a server, where the method may include a main server and a plurality of execution machines (execution machines may also be referred to as examples), receiving, by the main server, order status data input through respective order status data pipelines (order status data pipeline 1, order status data pipeline 2, order status data pipeline 3, order status data pipeline 4, … …), redistributing the input data by using a hash algorithm, distributing order status data corresponding to the same order number among the order status data to the same execution machine, and transmitting, by the respective execution machines (execution machine 1, execution machine 2, execution machine 3, execution machine 4, … …), the respective order status data to a data storage cluster to perform data updating processing.
In the implementation of the hash algorithm, the order state data may be distributed to the execution machines by a method that the number M of the order state data is left over by the number N of the execution machines (N is a natural number), and for example, 10 execution machines (N is 10) and 20 order state data (M is 20), when 10 is left over, the 1 st and 11 th order state data are distributed to the 1 st execution machine, the 2 nd and 12 th order state data are distributed to the 2 nd execution machine, … …, and so on. The hash algorithm used in the embodiment of the present invention is specifically a high-reliability high-fault-tolerance consistent hash algorithm, and the consistent hash algorithm is a hash algorithm with better stability.
Fig. 3 is a schematic diagram of an order status data updating process for distributing order status data based on a consistent hashing algorithm according to an embodiment of the present invention. As shown in fig. 3, after the execution machine receives the order state data distributed by the consistent hash algorithm, it configures processing authority (the processing authority may specifically be authority to use a thread) for the order state data in the execution machine through a ConcurrentHashMap (i.e. a multi-threaded hash set, also called a thread-safe hash set) built in Java, where, when the execution machine processes a certain piece of order state data, the ConcurrentHashMap determines whether to configure the processing authority for the piece of order state data, specifically, the ConcurrentHashMap determines whether there are other threads processing other order state data with the same order number, i.e. determines whether there are other order state data with the same order number occupying the thread, if so, waits for a preset time interval (i.e. a blocking time), the preset time interval (i.e. the blocking time) may be set to 1ms (millisecond), and then the ConcurrentHashMap determines again whether there are other threads processing other order state data with the same order number, until other threads finish processing other order state data with the same order number, after the ConcurrentHashMap releases the right of using the thread of other order state data with the same order number (namely clears the occupation of the thread by other order state data with the same order number), the ConcurrentHashMap configures the processing right (specifically the right of using the thread) to the order state data, so that the order state data is configured with the right of using the thread, and then the order state data is processed through the randomly allocated thread.
The specific algorithm for configuring the authority of using the thread for the order state data in the execution machine is as follows:
initializing ConcurrentHashMap, wherein the initialization comprises initialization processing such as applying for memory resources and the like;
and II, circularly calling the putIfAbsent method of ConcurrentHashMap. The putifAbsent method has two parameters, key and value. Taking two pieces of order state data distributed to the execution machine at the same time as the lock order state data and the delete order state data as an example, when the execution machine processes the lock order state data and the delete order state data, the execution machine respectively calls the putifAbsent method to respectively configure the authority of the respective use thread for the lock order state data and the delete order state data, wherein it is determined that only one of the lock order state data and the delete order state data is configured with the authority of the use thread in a preempting manner. Assuming that when the ticket state data is configured with the authority to use the thread, the putIfAbsent method invoked when processing the ticket state data takes the order number of the ticket state data as key and takes a random value as value, where the random value may specifically be the state value of the ticket state data or may also be another self-defined random number, and the return value of the putIfAbsent method is NULL, and the return value of the putIfAbsent method invoked when processing the ticket deleting state data is the above-mentioned random value (specifically, for example, the state value of the ticket state data or another self-defined random number);
and thirdly, if the return value of the putIfAbsent method called by configuring the authority of using the thread for a certain order state data is NULL, which indicates that other order state data without the same order number as the certain order state data are configured with the authority of using the thread, but the certain order state data are configured with the authority of using the thread, randomly allocating a thread for the certain order state data, and transmitting the certain order state data to the data storage cluster through the thread to perform data updating processing. After the data updating processing is finished, a remove method of ConcurrentHashMap is called, and a key and a corresponding order number of a putIfAbsent method called by configuring the authority of using the thread for the order state data are deleted, so that the authority of using the thread of the order state data is released, and the occupation of the order state data on the thread is eliminated;
and fourthly, if the called putifAbsent method for configuring the authority of using the thread for a certain order state data returns no NULL, other order state data which indicates that the order number is the same as the order state data is configured with the authority of using the thread, namely, the thread is processing other state information of the order number, after the thread processing for processing other state information is finished and the authority of using the thread is released, the putifAbsent method of ConcurrentHashMap is called to re-configure the authority of using the thread, if the configuration is successful, the order state information is configured with the authority of using the thread, and if the configuration is failed, a preset time interval (for example, 1 millisecond) is waited, and the putifAbsent method of ConcurrentHashMap is re-called to reconfigure the authority until the authority is configured. Whether the thread is processing other order state data of the order number can be polled at regular time, the interval time of polling can be set to be 1 millisecond or other values can be set according to requirements, the order number of the order state data is extracted, whether a key corresponding to the order number exists is judged, if yes, the thread is processing other order state data of the order number, and if not, the thread processing other order state data of the order number is finished.
The putIfAbsent method of the ConcurrentHashMap is a data synchronization method implemented by a segmented lock, taking the example that order state data a of order number 1 and order state data b of order number 2 are distributed to the execution machine (order number 1 is different from order number 2), due to the segmented lock mechanism of the ConcurrentHashMap, when the authority of using a thread is configured for the order state data a and the putIfAbsent method is called, the authority of using the thread is configured for the order state data b, the putIfAbsent method can also be called, the later does not need to wait for the calling of the former to be finished, and if the return values of calling the putIfAbsent methods for the two are both NULL, the order state data a and the order state data b can be simultaneously transmitted to the data storage cluster through the respective threads, so that the order state data among different threads can be synchronized, and data synchronization update can be efficiently implemented.
A data storage cluster such as a Redis cluster. Taking the data storage cluster as a Redis cluster as an example, a specific process of executing data update processing is described below. The data update process performed by the Redis cluster is mainly a two-step operation of checking updates (i.e., a two-step operation of checking whether data in the Redis cluster needs to be updated and performing the updates when the updates are needed), which needs to be wrapped into an atomic operation (the atomic operation is the most basic operation that is not interrupted by other threads when executing) in view of the multi-thread synchronization problem.
The procedure for the Redis cluster to perform the data update process is as follows: for the transmitted order state data, the Redis cluster firstly checks whether the stored data in the Redis cluster needs to be updated, wherein when the Redis cluster does not have the stored data with the same order state data order number as the order state data, or when the Redis cluster has the stored data with the same order state data order number as the order state data but the timestamp of the stored data is earlier than the timestamp of the order state data, the data in the Redis cluster needs to be updated, wherein whether the Redis cluster has the data with the same order state data order number as the order state data can be judged by using a get method of the Redis cluster. Then, the Redis cluster performs updating when data needs to be updated, accordingly, when there is no stored data in the Redis cluster that is the same as the order number of the order state data, the Redis cluster needs to insert the order state data into a storage record of the Redis cluster for each order state information so as to store the order state data, and when there is stored data in the Redis cluster that is the same as the order number of the order state data, but a timestamp of the stored data is earlier than that of the order state data, indicating that the order state recorded in the Redis cluster is the order state of the same order number that is earlier than the order state data, the Redis cluster updates the stored data of the order number recorded in the Redis cluster into the order state data. If the data in the Redis cluster does not need to be updated, namely the Redis cluster has stored data with the same order number as the order state data, but the timestamp of the stored data is later than that of the order state data, which indicates that the latest order state of the order number corresponding to the order state data is recorded in the Redis cluster, the stored data in the Redis cluster with the same order number as the order state data is kept unchanged.
In order to avoid data update errors, a data update scheme in the prior art needs to lock in a Redis cluster, so that when a certain order state is checked and updated, other order states are in a waiting state until the order state is checked and updated, and then check and update processing of other order states is performed, which seriously affects the performance of the entire Redis cluster, and a Java transaction locking mechanism has the problems of long locking process and high performance overhead. To overcome the above-mentioned defects of the prior art, the state information with the same order number can be processed by the same thread, however, this approach needs to redistribute the order state data received by each instance (i.e. execution machine) according to the thread ID (identification), which is more complex to implement, and the data is distributed multiple times, which increases the processing delay. According to the embodiment of the invention, after the execution machine receives the order state data distributed by the consistent hash algorithm, the permission of using the thread is configured for the order state data in the execution machine through the ConcurrentHashMap (namely, a multi-thread Hash set, also called a thread-safe Hash set) built in Java, so that only one piece of order state data in the order state data corresponding to the same order number is configured with the permission of using the thread at the same time, namely, only one thread processes the order state data of the order number at the same time, and therefore, after the order state data of the order number transmitted by the unique thread reaches the Redis cluster, the Redis cluster updates the order state data of the order number without errors, and locking is not needed when the Redis cluster updates the order state data, and the defects brought by locking in the prior art are overcome.
By using the technical scheme of the embodiment of the invention, the transaction locking operation with high performance overhead is converted into the distribution of the processing authority (specifically, the authority of using the thread) based on the ConcurrentHashMap, the processing energy and performance of the system are improved, the average processing time of single order state data can be reduced to 3ms from 10ms of the locking-based scheme in the prior art, the throughput of the system is greatly improved, and in addition, the stability of the system is further improved from the data distribution angle by using a consistent Hash algorithm.
Fig. 4 is a schematic diagram of main blocks of a data updating apparatus according to an embodiment of the present invention.
As shown in fig. 4, the data updating apparatus 400 according to the embodiment of the present invention mainly includes a data distribution module 401, a right configuration module 402, a data transmission module 403, and a right release module 404.
The data distribution module 401 is configured to distribute, by using a preset algorithm, data corresponding to the same service number in the input data to the same execution machine.
The preset algorithm may be a hash algorithm or a consistent hash algorithm.
The input data may include time stamp information.
An authority configuration module 402, configured to configure, in each execution machine, a processing authority of data in the execution machine according to a preset rule, where the preset rule is: only one item of data corresponding to the same service number at the same time is configured with processing authority.
The processing permission may be a permission to use the thread.
The permission configuration module 402 may also be configured to: and configuring the processing authority of each item of data one by one according to the time of the data reaching the execution machine, wherein when the processing authority of each item of data is configured, the following steps are executed: step one, judging whether other data with the same service number as the data has processing authority or not, if so, executing step two, and if not, executing step three; step two, waiting for a preset time interval, and then returning to execute the step one; and step three, configuring the data with processing authority.
The permission configuration module 402 may also be configured to: and judging whether other data with the same service number as the data occupies the thread or not.
The permission configuration module 402 may also be configured to: and configuring the processing authority of the data in each execution machine according to a preset rule by using a multithread hash set built in Java in each execution machine.
A data transmission module 403, configured to transmit the respective data configured with the processing authority to the data storage cluster through each execution machine to perform data updating processing.
The data transfer module 403 may also be configured to: data configured with respective permissions to use the threads is communicated by each of the execution machines to the data storage cluster via the randomly assigned threads.
The step of executing the data update process may specifically include: for each item of data transmitted to the data storage cluster, the data storage cluster judges whether stored data with the same service number as the transmitted data exists or not; if so, comparing the timestamp information of the stored data and the transmitted data, and taking the latest timestamp data in the stored data and the transmitted data as new stored data; if not, the transmitted data is stored in the data storage cluster.
The permission releasing module 404 is configured to, after data configured with processing permission in an execution machine is executed with data update processing, release the processing permission of the data by the execution machine, where the processing permission of the execution machine to release the data is mainly to clear occupation of a thread by the data.
Fig. 5 illustrates an exemplary system architecture 500 to which the data update method or data update apparatus of an embodiment of the present invention may be applied.
As shown in fig. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 serves to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. Network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 501, 502, 503 to interact with a server 505 over a network 504 to receive or send messages or the like. The terminal devices 501, 502, 503 may have various communication client applications installed thereon, such as a shopping application, a web browser application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.
The terminal devices 501, 502, 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 505 may be a server that provides various services, such as a background management server that supports shopping websites browsed by users using the terminal devices 501, 502, 503. The background management server can analyze and process the received data such as the product information inquiry request and feed back the processing result (the order state data of the product) to the terminal equipment.
It should be noted that the data updating method provided by the embodiment of the present invention is generally executed by the server 505, and accordingly, the data updating apparatus is generally disposed in the server 505.
It should be understood that the number of terminal devices, networks, and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 6, shown is a block diagram of a computer system 600 suitable for use in implementing a server according to embodiments of the present application. The server shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present application are executed when the computer program is executed by the Central Processing Unit (CPU) 601.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a data distribution module 401, a rights configuration module 402, a data transfer module 403, a rights release module 404. The names of these modules do not in some cases constitute a limitation on the modules themselves, and for example, the data distribution module 401 may also be described as "a module for distributing data corresponding to the same service number in input data to the same execution machine by using a preset algorithm".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: distributing data corresponding to the same service number in the input data to the same execution machine by using a preset algorithm; configuring the processing authority of the data in each execution machine according to a preset rule, wherein the preset rule is as follows: only one item of data corresponding to the same service number at the same time is configured with processing authority; transmitting the respective data configured with the processing authority to the data storage cluster through each execution machine to execute data updating processing; when data configured with processing authority in an execution machine is executed with data updating processing, the execution machine releases the processing authority of the data.
According to the technical scheme of the embodiment of the invention, the data corresponding to the same service number in the input data are distributed to the same execution machine by using a preset algorithm; configuring the processing authority of the data in each execution machine according to a preset rule, wherein the preset rule is that only one data in the data corresponding to the same service number is configured with the processing authority at the same time; transmitting the respective data configured with the processing authority to the data storage cluster through each execution machine to execute data updating processing; when data configured with processing authority in an execution machine is executed with data updating processing, the execution machine releases the processing authority of the data. By using the technical scheme of the embodiment of the invention, the processing delay can be reduced and the system throughput can be improved on the premise of ensuring the correct update of the data.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (14)
1. A method for updating data, comprising:
distributing data corresponding to the same service number in the input data to the same execution machine by using a preset algorithm;
configuring the processing authority of the data in each execution machine according to a preset rule according to the time of the data reaching the execution machine, wherein the preset rule is as follows: only one item of data corresponding to the same service number at the same time is configured with the processing authority;
transmitting, by each execution engine, respective data configured with the processing privilege to a data storage cluster to perform a data update process;
when the data configured with the processing authority in an execution machine is executed with the data updating processing, the execution machine releases the processing authority of the data.
2. The method of claim 1, wherein the predetermined algorithm is a hash algorithm or a consistent hash algorithm.
3. The method of claim 1, wherein the step of configuring, in each of the execution machines, the processing right of the data in the execution machine according to the preset rule based on the time of the data reaching the execution machine comprises:
and configuring the processing authority of each item of data one by one according to the time of the data reaching the execution machine, wherein when the processing authority of each item of data is configured, the following steps are executed:
step one, judging whether other data with the same service number as the data has the processing authority or not, if so, executing step two, and if not, executing step three;
step two, waiting for a preset time interval, and then returning to execute the step one;
and step three, configuring the data with the processing authority.
4. The method of claim 1, wherein the input data includes timestamp information,
the step of executing the data update process includes:
for each item of data transmitted to the data storage cluster, the data storage cluster judges whether stored data with the same service number as the transmitted data exists or not;
if so, comparing the timestamp information of the stored data and the transmitted data, and taking the latest timestamp data in the stored data and the transmitted data as new stored data;
and if not, storing the transmitted data into the data storage cluster.
5. The method of claim 3, wherein the processing permission is a permission to use a thread,
the step of transmitting, by each execution engine, respective data configured with the processing privilege to a data storage cluster, includes: transmitting, by each execution machine, respective data configured with permission to use the thread to the data storage cluster through the randomly assigned thread,
the step of releasing the processing authority of the data by the execution machine includes: the execution machine clears the data from occupying the thread,
the step of judging whether data is configured with the processing authority in other data with the same service number as the data comprises the following steps: and judging whether other data with the same service number as the data occupies the thread or not.
6. The method as claimed in claim 1, wherein the processing authority of the data in each execution machine is configured according to a preset rule by using a multithread hash set built in Java.
7. A data update apparatus, comprising:
the data distribution module is used for distributing data corresponding to the same service number in the input data to the same execution machine by using a preset algorithm;
the authority configuration module is used for configuring the processing authority of the data in each execution machine according to the time of the data reaching the execution machine and a preset rule, wherein the preset rule is as follows: only one item of data corresponding to the same service number at the same time is configured with the processing authority;
the data transmission module is used for transmitting the respective data configured with the processing authority to the data storage cluster through each execution machine so as to execute data updating processing;
and the authority releasing module is used for releasing the processing authority of the data after the data configured with the processing authority in an execution machine is executed by the data updating processing.
8. The apparatus of claim 7, wherein the predetermined algorithm is a hash algorithm or a consistent hash algorithm.
9. The apparatus of claim 7, wherein the permission configuration module is further configured to:
and configuring the processing authority of each item of data one by one according to the time of the data reaching the execution machine, wherein when the processing authority of each item of data is configured, the following steps are executed:
step one, judging whether other data with the same service number as the data has the processing authority or not, if so, executing step two, and if not, executing step three;
step two, waiting for a preset time interval, and then returning to execute the step one;
and step three, configuring the data with the processing authority.
10. The apparatus of claim 7, wherein the input data comprises timestamp information,
the step of executing the data update process includes:
for each item of data transmitted to the data storage cluster, the data storage cluster judges whether stored data with the same service number as the transmitted data exists or not;
if so, comparing the timestamp information of the stored data and the transmitted data, and taking the latest timestamp data in the stored data and the transmitted data as new stored data;
and if not, storing the transmitted data into the data storage cluster.
11. The apparatus of claim 9, wherein the processing permission is a permission to use a thread,
the data transfer module is further configured to: transmitting, by each execution machine, respective data configured with permission to use the thread to the data storage cluster through the randomly assigned thread,
the permission release module is further configured to: the execution machine clears the data from occupying the thread,
the permission configuration module is further configured to: and judging whether other data with the same service number as the data occupies the thread or not.
12. The apparatus of claim 7, wherein the permission configuration module is further configured to: and configuring the processing authority of the data in each execution machine according to a preset rule by using a multithread hash set built in Java in each execution machine.
13. A server, comprising:
one or more processors;
a memory for storing one or more programs,
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method recited in any of claims 1-6.
14. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710594880.3A CN109284177B (en) | 2017-07-20 | 2017-07-20 | Data updating method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710594880.3A CN109284177B (en) | 2017-07-20 | 2017-07-20 | Data updating method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109284177A CN109284177A (en) | 2019-01-29 |
| CN109284177B true CN109284177B (en) | 2021-03-30 |
Family
ID=65185367
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710594880.3A Active CN109284177B (en) | 2017-07-20 | 2017-07-20 | Data updating method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109284177B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110825762A (en) * | 2019-11-14 | 2020-02-21 | 北京融易做科技有限公司 | Data storage method, device and system |
| CN114358779B (en) * | 2022-03-18 | 2022-06-28 | 汇承金融科技服务(南京)有限公司 | Order processing method and device, storage medium and electronic device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102354316A (en) * | 2011-09-22 | 2012-02-15 | 用友软件股份有限公司 | Data updating device and data updating method |
| CN102467570A (en) * | 2010-11-17 | 2012-05-23 | 日电（中国）有限公司 | Connection query system and method for distributed data warehouse |
| CN103632250A (en) * | 2013-12-06 | 2014-03-12 | 湖南御家汇化妆品有限公司 | Quick sales order sorting, grouping and screening method |
| CN105045791A (en) * | 2014-03-26 | 2015-11-11 | 日本电气株式会社 | Database device |
| CN105139151A (en) * | 2015-09-25 | 2015-12-09 | 佛山市中格威电子有限公司 | Information management system based on two-dimensional code technology |
| CN106651402A (en) * | 2016-12-29 | 2017-05-10 | 江西博瑞彤芸科技有限公司 | Data processing method for goods return operation |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8677344B2 (en) * | 2008-10-10 | 2014-03-18 | Sony Corporation | Software upgrade using single source channel |
| US20150199318A1 (en) * | 2011-09-02 | 2015-07-16 | Micah Lemonik | System and Method for Using a Third-Party Add-On in a Collaborative On-Line Software Development Environment |
| US9684853B2 (en) * | 2014-04-15 | 2017-06-20 | Kyocera Document Solutions Inc. | Image forming apparatus that writes data from volatile memory to non-volatile memory |
| US9684689B2 (en) * | 2015-02-03 | 2017-06-20 | Ca, Inc. | Distributed parallel processing system having jobs processed by nodes based on authentication using unique identification of data |
-
2017
- 2017-07-20 CN CN201710594880.3A patent/CN109284177B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102467570A (en) * | 2010-11-17 | 2012-05-23 | 日电（中国）有限公司 | Connection query system and method for distributed data warehouse |
| CN102354316A (en) * | 2011-09-22 | 2012-02-15 | 用友软件股份有限公司 | Data updating device and data updating method |
| CN103632250A (en) * | 2013-12-06 | 2014-03-12 | 湖南御家汇化妆品有限公司 | Quick sales order sorting, grouping and screening method |
| CN105045791A (en) * | 2014-03-26 | 2015-11-11 | 日本电气株式会社 | Database device |
| CN105139151A (en) * | 2015-09-25 | 2015-12-09 | 佛山市中格威电子有限公司 | Information management system based on two-dimensional code technology |
| CN106651402A (en) * | 2016-12-29 | 2017-05-10 | 江西博瑞彤芸科技有限公司 | Data processing method for goods return operation |
Non-Patent Citations (1)
| Title |
|---|
| 基于数据服务的数据组合视图的优化更新;张鹏等;《计算机学报》;20111231;第34卷(第12期);第2344-2354页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109284177A (en) | 2019-01-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109274731B (en) | Method and device for deploying and calling web service based on multi-tenant technology | |
| CN109032796B (en) | Data processing method and device | |
| CN111897633A (en) | Task processing method and device | |
| CN110928912A (en) | Method and device for generating unique identifier | |
| CN111478781B (en) | Message broadcasting method and device | |
| CN110909022A (en) | Data query method and device | |
| CN109284177B (en) | Data updating method and device | |
| CN115525411A (en) | Method, device, electronic equipment and computer readable medium for processing service request | |
| CN112860505A (en) | Method and device for regulating and controlling distributed clusters | |
| CN112398669A (en) | Hadoop deployment method and device | |
| CN111290842A (en) | Task execution method and device | |
| CN110413427B (en) | Subscription data pulling method, device, equipment and storage medium | |
| CN113010238A (en) | Permission determination method, device and system for micro application call interface | |
| CN112306695A (en) | Data processing method and device, electronic equipment and computer storage medium | |
| CN108683608B (en) | Method and device for distributing flow | |
| CN113541987A (en) | Method and device for updating configuration data | |
| CN113765871A (en) | Fortress management method and device | |
| CN114374657B (en) | Data processing method and device | |
| CN112948138A (en) | Method and device for processing message | |
| CN112214500A (en) | Data comparison method and device, electronic equipment and storage medium | |
| CN114756173A (en) | Method, system, device and computer readable medium for file merging | |
| CN113760487A (en) | Service processing method and device | |
| CN112182003A (en) | Data synchronization method and device | |
| CN114528140A (en) | Method and device for service degradation | |
| CN110909269B (en) | Log reporting method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |