CN107370809A - Method of data synchronization and data search system - Google Patents
Method of data synchronization and data search system Download PDFInfo
- Publication number
- CN107370809A CN107370809A CN201710568992.1A CN201710568992A CN107370809A CN 107370809 A CN107370809 A CN 107370809A CN 201710568992 A CN201710568992 A CN 201710568992A CN 107370809 A CN107370809 A CN 107370809A
- Authority
- CN
- China
- Prior art keywords
- data search
- data
- cluster
- request
- search cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of method of data synchronization and data search system, belong to Internet technical field.Method includes：When data renewal operation occurs for the first data search cluster, data new information of the first data search cluster generation for data renewal operation；Data new information is distributed to the first subscription message server of matching by the first data search cluster；Data new information is pushed to the second data search cluster by the first subscription message server, is updated so that the second data search cluster synchronizes data according to data new information.The present invention is by being respectively one subscription message server of each data search clustered deploy(ment), so that on each data search cluster can complete paired data renewal, and support by subscribing to message server to be synchronized to the renewal of data other data search clusters.And even if service exception occurs for one or several data search clusters, it can still ensure that search service can use, greatly improve the disaster tolerance performance of system.
Description
Technical field
The present invention relates to Internet technical field, more particularly to a kind of method of data synchronization and data search system.
Background technology
Data volume is in the trend that explodes after Internet era is entered, and this for people while bringing great convenience, therewith
And also to have brought the difficulty of data screening, data search engine arises at the historic moment under this environment.For example, in recent years
With the continuous maturation of distributed computing technology, there are the data search such as Sphinx, Solr, Elasticsearch and draw
Hold up.Wherein, why above-mentioned data search engine possesses search process ability at high speed, is because having set up at least one search
Cluster.For example Elasticsearch can support search node scales up to a hundred as emerging distributed real-time search engine
Different clusters come carry out cluster from set up.And when carrying out data search, in order to ensure that data search engine can be user
There is provided accurately search result and ensure search service high disaster tolerance, how it is different search clusters between carry out data it is same
Step is always a focus of those skilled in the art's concern.
By taking Elasticsearch search engines as an example, prior art mainly takes following manner when carrying out data syn-chronization
Realize：As shown in figure 1, introduce third party cloud service and set up a main cluster and at least one secondary cluster.Wherein, main collection
The index data of group's renewal is uploaded to by Backup and Restore function on the cloud storage service device of third party cloud service offer, and is disposed
In other regional secondary clusters then by start by set date Backup and Restore function, main cluster is obtained more from above-mentioned cloud storage service device
New index data, to reach the data syn-chronization incremental update with main cluster.
During the present invention is realized, inventor has found that prior art at least has problems with：
The basis that above-mentioned data syn-chronization is realized is the paid paid service provided using third party cloud service, is not only expended big
Financial resources are measured, and the stability of search service is limited to third party cloud service.Further, since the renewal operation of index data is necessary
In the above-mentioned progress of main cluster, secondary cluster is used for queried access only as copy one by one, therefore this programme only supports area more
Local search, do not support the local updates in more areas, i.e., all renewal operations should necessarily return to main cluster and carry out, therefore meeting
The RTT (Round-Trip Time, round-trip delay) for causing renewal to operate is higher.Further, since the data of secondary cluster are all from
In main cluster, if therefore main cluster occur abnormal, secondary cluster just can not carry out data renewal, cause the disaster tolerance of search service
Can be poor.
The content of the invention
In order to solve problem of the prior art, the embodiments of the invention provide a kind of method of data synchronization and data search system
System.The technical scheme is as follows：
First aspect, there is provided a kind of method of data synchronization, applied to data search system, the data search system bag
Include request Distributor, at least two data search clusters and match with least two data search number of clusters
Message server is subscribed to, methods described includes：
When data renewal operation occurs for the first data search cluster, the first data search cluster generation is for described
The data new information of data renewal operation；
The data new information is distributed to the first subscription message server of matching by the first data search cluster；
The data new information is pushed to the second data search cluster by the first subscription message server, so that institute
State the second data search cluster and data renewal is synchronized according to the data new information；
Wherein, the first data search cluster is any data search collection at least two data search cluster
Group, the second data search cluster be at least two data search cluster in except the first data search cluster it
Other outer data search clusters.
In another embodiment, methods described also includes：
Described first subscribes to the message subscribing request that message server receives the second data search collection pocket transmission, described
The identification information of the second data search cluster is comprised at least in message subscribing request；
Described first subscribes to identification information of the message server according to the second data search cluster, is ordered to carrying out message
The the second data search cluster read is recorded；
The data new information is pushed to the second data search cluster by the first subscription message server, including：
Described first subscribes to identification information of the message server according to the second data search cluster, by the data more
New information pushes to the second data search cluster.
In another embodiment, methods described also includes：
The request Distributor is after the data search request of requesting terminal transmission is received, at least two described in acquisition
The network state of each data search cluster in individual data search cluster；
The network state of request Distributor each data search cluster according to, described at least two
The target data for determining to handle the data search request in data search cluster searches for cluster；
The data search request is forwarded to target data search cluster by the request Distributor；
The target data searches for cluster after the data search request is received, and the data search request is carried out
Processing, and obtained result is sent to the request Distributor；
The result is forwarded to the requesting terminal by the request Distributor.
In another embodiment, the request Distributor is according to the network-like of each data search cluster
State, determine to handle the target data search cluster of the data search request, bag at least two data search cluster
Include：
The request Distributor determines the minimum data of round-trip delay at least two data search cluster
Search for cluster；
The minimum data search cluster of the round-trip delay is defined as the target data by the request Distributor
Search for cluster.
In another embodiment, the request Distributor is according to the network-like of each data search cluster
State, determine to handle the target data search cluster of the data search request, bag at least two data search cluster
Include：
The request Distributor determines the minimum data of round-trip delay at least two data search cluster
Search for cluster；
The request Distributor judges that the minimum data search cluster of the round-trip delay is abnormal with the presence or absence of service；
If the data search cluster appearance service exception that the round-trip delay is minimum, described to ask Distributor root again
The target data search cluster is determined according to the network state of the 3rd data search cluster；
Wherein, the 3rd data search cluster is except the round-trip delay at least two data search cluster
Other data search clusters outside minimum data search cluster.
Second aspect, there is provided a kind of data search system, the data search system include request Distributor, extremely
Few two data search clusters and the subscription message server matched with least two data search number of clusters；
First data search cluster, for when data renewal operation occurs, generation to be for data renewal operation
Data new information；
The first data search cluster, it is additionally operable to the data new information being distributed to the first subscription message of matching
Server；
Described first subscribes to message server, for the data new information to be pushed into the second data search cluster；
The second data search cluster, for synchronizing data renewal according to the data new information；
Wherein, the first data search cluster is any data search collection at least two data search cluster
Group, the second data search cluster be at least two data search cluster in except the first data search cluster it
Other outer data search clusters.
In another embodiment, described first message server is subscribed to, is additionally operable to receive the second data search collection
The message subscribing request of pocket transmission, the mark that the second data search cluster is comprised at least in the message subscribing request are believed
Breath；
Described first subscribes to message server, the identification information according to the second data search cluster is additionally operable to, to entering
The second data search cluster of row message subscribing is recorded；
Described first subscribes to message server, for the identification information according to the second data search cluster, by described in
Data new information pushes to the second data search cluster.
In another embodiment, the request Distributor, for being searched in the data for receiving requesting terminal transmission
After rope request, the network state of each data search cluster at least two data search cluster is obtained；
The request Distributor, for the network state according to each data search cluster, it is described extremely
Determine that the target data for handling the data search request searches for cluster in few two data search clusters, and the data are searched
Rope request is forwarded to target data search cluster；
The target data searches for cluster, for after the data search request is received, being asked to the data search
Ask and handled, and obtained result is sent to the request Distributor；
The request Distributor, it is additionally operable to the result being forwarded to the requesting terminal.
In another embodiment, the request Distributor, at least two data search cluster
The minimum data search cluster of round-trip delay is determined, the minimum data search cluster of the round-trip delay is defined as the target
Data search cluster.
In another embodiment, the request Distributor, at least two data search cluster
Determine the minimum data search cluster of round-trip delay；Judge the minimum data search cluster of the round-trip delay with the presence or absence of service
It is abnormal；If it is abnormal that service occurs in the minimum data search cluster of the round-trip delay, further according to the 3rd data search cluster
Network state determines the target data search cluster；
Wherein, the 3rd data search cluster is except the round-trip delay at least two data search cluster
Other data search clusters outside minimum data search cluster.
The beneficial effect that technical scheme provided in an embodiment of the present invention is brought is：
By deploying the mutually redundant multiple data search clusters of data and being respectively each data search cluster
It is supporting deploy one subscription message server so that on each data search cluster can complete paired data renewal,
And support that the renewal of data will be synchronized to other trans-regional data search clusters, institute by the subscription message server matched
By compared to the RTT for being only capable of in a manner of main cluster carries out data renewal, greatly reducing renewal operation.It is further, since each
The data stored on individual data search cluster keep realtime uniform, and each data search cluster can provide complete search clothes
Business, even if service exception occurs for one or several data search clusters, it can still ensure that search service can use, therefore carry significantly
The disaster tolerance performance of system is risen, and the realization of data syn-chronization of the embodiment of the present invention is without third-party support so that search service
Stability be not only restricted to third party.
Brief description of the drawings
Technical scheme in order to illustrate the embodiments of the present invention more clearly, make required in being described below to embodiment
Accompanying drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the present invention, for
For those of ordinary skill in the art, on the premise of not paying creative work, other can also be obtained according to these accompanying drawings
Accompanying drawing.
Fig. 1 is the data search system involved by a kind of Elasticsearch search engines that background of invention provides
The Organization Chart of system；
Fig. 2A is a kind of Organization Chart of data search system provided in an embodiment of the present invention；
Fig. 2 B are a kind of flow charts of method of data synchronization provided in an embodiment of the present invention；
Fig. 3 A are a kind of flow charts of method of data synchronization provided in an embodiment of the present invention；
Fig. 3 B are a kind of flow charts of data search method provided in an embodiment of the present invention；
Fig. 4 is a kind of structural representation of data search system provided in an embodiment of the present invention；
Fig. 5 is a kind of structural representation of data search system provided in an embodiment of the present invention.
Embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with accompanying drawing to embodiment party of the present invention
Formula is described in further detail.
Before to the embodiment of the present invention carrying out that explanation is explained in detail, first the embodiment of the present invention may relate to one
A little nouns are explained.
RTT (Round Trip Time, round-trip delay)：It is an important performance indexes in internet, it is represented from hair
Sending end sends data and started, and receiving the confirmation from receiving terminal to transmitting terminal, (receiving terminal is soon sent really after receiving data
Recognize), the time delay undergone altogether.
Cluster：Cluster server, i.e. Cluster are referred in embodiments of the present invention.Wherein, cluster is such a skill
Art, it connects together multiple systems so that the server of multiple systems can be worked or seen as a server
Get up to seem a server.Using after Clustering, can greatly improve the stability of system, data-handling capacity and
Service ability.
Cloud storage：A kind of emerging Network storage technology, refers to by cluster application, network technology or distributed field system
The functions such as system, gather collaborative work, jointly by a large amount of various types of storage devices in network by application software
One system of data storage and Operational Visit function is externally provided.
Packet loss：Refer to the ratio that the data packet number lost accounts for transmitted packet total amount.
Load：Weigh the amount of being run in a server for task.The load of the bigger meaning server of this amount is bigger, has
The task of being likely to result in overstocks, and then causes performance loss.
Next, data search system provided in an embodiment of the present invention is once illustrated.
Referring to Fig. 2A, data search system provided in an embodiment of the present invention is made up of three parts, respectively request distribution
Server, data search cluster and subscription message server.Separately below to request Distributor, data search cluster with
And subscribe to message server and carry out that explanation is explained in detail.
Ask Distributor
In embodiments of the present invention, Distributor is asked to be referred to as intelligence request dissemination system again.Wherein, intelligence please
Dissemination system is asked to be deployed in all parts of the world as the access point of search service.That is, the clothes that intelligence request dissemination system includes
Business device is large number of.Wherein, the requesting terminal with search need can be asked by way of accessing nearby by neighbouring intelligence
Dissemination system is sought to forward data search request.
Intelligence request dissemination system can perceive the network state of all data search clusters in the whole world.Wherein, it is network-like
State includes but is not limited to the parameters such as round-trip delay, packet loss and load.So, intelligence request dissemination system receive it is neighbouring
Requesting terminal send data search request after, intelligence ask dissemination system can be according to the network of all data search clusters
State handles this data search request to select optimal data rope cluster to search for cluster as target data.
Wherein, when carrying out the selection of target data search cluster, the embodiment of the present invention is typically using RTT as mainly examining
Consider key element.Because data search cluster lower RTT is also nearer generally with the distance of requesting terminal, therefore target data search collection
Group's data search cluster generally neighbouring with requesting terminal.But if the target data search cluster currently selected goes out
The situation of service exception is showed, has been usually expressed as packet loss height or load is high, then intelligence request dissemination system still can be with guarantor
Card service availability is preferential selection principle, can carry out the choosing of target data search cluster in other data search clusters again
Take, i.e., the data search request can be switched to other normal data search clusters of service and handled.
Data search cluster
In embodiments of the present invention, in order to realize the low RTT in response data searching request, and make it that search service is high
(disaster tolerance performance is good) can be used, the mutually redundant data search cluster of multiple data can be established in each area in the whole world, and then form
Multicenter search system.Wherein, the mutually redundant data for meaning that each data search cluster possesses full dose of data are used for
Complete search service is provided.One data search cluster there is a problem that other data search clusters are due to storing full dose
Data, therefore search service also can use, and data search precision is not in the situation of reduction.In addition, a data are searched
There is data renewal in Suo Jiqun, and other data search clusters can synchronize data renewal.
In summary, by intelligence request dissemination system by the data search request of requesting terminal be distributed to it is neighbouring (i.e.
Round-trip delay is low) data search cluster handled, realize reduces respond request in a manner of asking response localization
RTT.And if there is service abnormal problem in the data search cluster selected originally, in the state that can not be serviced.Intelligence
Data search request can also be forwarded to other and service normal data search cluster by request dissemination system, ensure that service is high
It can use.
Subscribe to message server
In embodiments of the present invention, it is each in order to ensure because center search system is made up of multiple data search clusters
Data search cluster can provide complete search service, and the data that need to ensure to store in each data search cluster keep real-time
Unanimously.And because data search cluster is distributed across each different regions, therefore in order to realize the real-time trans-regional same of data
Step, the embodiment of the present invention, which is that each data search cluster is supporting, is provided with a subscription message server.Such as if data
Searching for cluster A, there occurs data modification, then data search cluster A just disappears the data publication of modification to matching subscription
Cease in server A.And other data search clusters in addition to data search cluster A are by subscribing to data search cluster A's
Message server A message is subscribed to, can be by the real time data synchronization in data search cluster A to locally.Namely data search
Between cluster, the trans-regional real-time synchronization of data is realized by subscribing to the message of other data search clusters mutually.
Fig. 2 B are a kind of flow charts of method of data synchronization provided in an embodiment of the present invention, and the method for data synchronization is applied to
Data search system, the data search system include request Distributor, at least two data search clusters and with least
The subscription message server of two data search number of clusters matchings.Referring to Fig. 2 B, method flow provided in an embodiment of the present invention
Including：
201st, when data renewal operation occurs for the first data search cluster, the generation of the first data search cluster is directed to data
The data new information of operation is updated, the first data search cluster is any data search at least two data search clusters
Cluster.
202nd, data new information is distributed to the first subscription message server of matching by the first data search cluster.
203rd, data new information is pushed to the second data search cluster by the first subscription message server, so that the second number
Data renewal is synchronized according to data new information according to search cluster, the second data search cluster is at least two data search
Other data search clusters in cluster in addition to the first data search cluster.
Method provided in an embodiment of the present invention, by deploying the mutually redundant multiple data search clusters of data and dividing
Not Wei each data search cluster it is supporting deploy one subscription message server so that on each data search cluster
Can complete paired data renewal, and support by matching subscription message server the renewal of data will be synchronized to it is trans-regional
Other data search clusters, so compared to be only capable of main cluster carry out data renewal mode, greatly reduce renewal behaviour
The RTT of work.Further, since the data stored on each data search cluster keep realtime uniform, each data search collection
Group can provide complete search service, even if service exception occurs for one or several data search clusters, can still ensure
Search service can use, therefore greatly improve the disaster tolerance performance of system, and the realization of data syn-chronization of the embodiment of the present invention is without the
The support of tripartite so that the stability of search service is not only restricted to third party.
In another embodiment, this method also includes：
First subscribes to the message subscribing request that message server receives the second data search collection pocket transmission, message subscribing request
In comprise at least the second data search cluster identification information；
First subscribes to identification information of the message server according to the second data search cluster, and second to carrying out message subscribing
Data search cluster is recorded；
Data new information is pushed to the second data search cluster by the first subscription message server, including：
First subscribes to message server according to the identification information of the second data search cluster, and data new information is pushed to
Second data search cluster.
In another embodiment, this method also includes：
Request Distributor obtains at least two data and searched after the data search request of requesting terminal transmission is received
The network state of each data search cluster in Suo Jiqun；
Network state of the Distributor according to each data search cluster is asked, at least two data search clusters
The middle target data search cluster for determining processing data searching request；
Data search request is forwarded to target data search cluster by request Distributor；
Target data searches for cluster after data search request is received, and data search request is handled, and will
To result be sent to request Distributor；
Result is forwarded to requesting terminal by request Distributor.
In another embodiment, network state of the Distributor according to each data search cluster is asked, extremely
The target data search cluster of processing data searching request is determined in few two data search clusters, including：
Request Distributor determines the minimum data search cluster of round-trip delay at least two data search clusters；
The minimum data search cluster of round-trip delay is defined as target data search cluster by request Distributor.
In another embodiment, network state of the Distributor according to each data search cluster is asked, extremely
The target data search cluster of processing data searching request is determined in few two data search clusters, including：
Request Distributor determines the minimum data search cluster of round-trip delay at least two data search clusters；
Request Distributor judges that the minimum data search cluster of round-trip delay is abnormal with the presence or absence of service；
If it is abnormal that service occurs in the minimum data search cluster of round-trip delay, request Distributor is further according to the 3rd number
Determine that target data searches for cluster according to the network state of search cluster；
Wherein, the 3rd data search cluster is except the minimum data of round-trip delay are searched at least two data search clusters
Other data search clusters outside Suo Jiqun.
Above-mentioned all optional technical schemes, any combination can be used to form the alternative embodiment of the disclosure, herein no longer
Repeat one by one.
Fig. 3 A are a kind of flow charts of method of data synchronization provided in an embodiment of the present invention, applied to data search system,
Referring to Fig. 3 A, method flow provided in an embodiment of the present invention includes：
301a, first subscribe to the message subscribing request that message server receives the second data search collection pocket transmission, the message
Subscribe to the identification information that the second data search cluster is comprised at least in request.
What it is firstly the need of explanation is a little that the first data search cluster is that at least two data are searched in embodiments of the present invention
Any one data search cluster in Suo Jiqun, and the second data search cluster is except the at least two data search clusters
Other data search clusters outside one data search cluster.
In addition, the quantity of message server and the quantity Matching of data search cluster are subscribed to, to ensure that each data is searched
The corresponding subscription message servers of Suo Jiqun.Wherein, the first subscription message server matches with the first data search cluster,
That is the first data search cluster is published to the first subscription message server whenever having data renewal.
In embodiments of the present invention, can be mutual between data search cluster in order to realize the trans-regional real-time synchronization of data
Between carry out message subscribing.So, can be to pass through this subscription machine when data renewal be present on a certain data search cluster
System causes other data search cluster synchronizations to carry out correspondingly data and update.
Wherein, the identification information of the second data search cluster can be IP (Internet Protocol, network address) address
And/or port information, the embodiment of the present invention are limited without specific this.
302a, first subscribe to identification information of the message server according to the second data search cluster, to carrying out message subscribing
The second data search cluster recorded.
In embodiments of the present invention, in order to subsequently the data new information of generation can be pushed to except the first data are searched
The second data search cluster outside Suo Jiqun, first subscribes to message server can be to the second data search of progress message subscribing
Cluster is recorded.For example the total data for covering the second data search cluster searches for the unified record of identification information of cluster
In a file or a list.
303a, when the first data search cluster occur data renewal operation when, the first data search cluster generation for number
According to the data new information of renewal operation.
Wherein, data renewal operation both may include the modification to data storage, may also comprise newly-increased data, and the present invention is real
Example is applied to limit this without specific.
First data search cluster is when generating data new information, if data renewal operation is that data storage is repaiied
Change, then may include that target say by the identification information of data storage and the modification to target data storage in the data new information
Bright information, so that follow-up second data search cluster is the identification information of data storage finds target and stored according to target
After data, the modification to target data storage is completed according to the revised comment information.Or, it can also be wrapped in the data new information
The target identification information of data storage and amended target data storage is included, follow-up second data search cluster is in basis
The identification information of target data storage finds target after data storage, directly utilizes amended target data storage
Carry out data replacement.In addition, if data renewal operation is newly-increased data, the data new information includes newly-increased number
According to.
The data new information is distributed to the first subscription message server of matching by 304a, the first data search cluster.
In embodiments of the present invention, occur any data renewal in the first data search cluster and can report to first ordering
Read message server.
The data new information is pushed to the second data search cluster by 305a, the first subscription message server.
Because the first subscription message server has been recorded the second data search cluster for carrying out message subscribing, because
This first subscription message server can push to the data new information according to the identification information of the second data search cluster
Second data search cluster.
306a, the second data search cluster synchronize data renewal according to the data new information.
In embodiments of the present invention, the second data search cluster is receiving the data of the first subscription message server push
, can be with reference to the synchronized update of the complete paired data of the several ways shown in above-mentioned steps 303a so that differently after new information
The data stored on each data search cluster in area keep realtime uniform.So, even if some or a few individual data are searched
There are the abnormal conditions that can not provide search service in Suo Jiqun, and other data search clusters can still provide the search clothes of completion
Business, there is provided the disaster tolerance performance of system.
In another embodiment, the embodiment of the present invention is after request Distributor is introduced so that response data is searched
The response time of rope request is also reduced significantly, embodiment of the detailed process description request shown in referring to following Fig. 3 B：
301b, requesting terminal send data search request.
From Fig. 2A, requesting terminal is specifically to send data search request to neighbouring request Distributor, i.e.,
Requesting terminal forwards data search request based on the mode accessed nearby by neighbouring request Distributor.
302b, request Distributor obtain at least two after the data search request of requesting terminal transmission is received
The network state of each data search cluster in data search cluster, and according to the network-like of each data search cluster
State, determine to handle the target data search cluster of the data search request at least two data search clusters.
In embodiments of the present invention, the determination mode of target data search cluster includes but is not limited to following two kinds：
First way, request Distributor determine the minimum number of round-trip delay at least two data search clusters
According to search cluster；The minimum data search cluster of round-trip delay is defined as target data search cluster by request Distributor.
It is low as most important consideration key element using round-trip delay, therefore ask Distributor for first way
The round-trip delay of each data search cluster can be obtained, and determines the minimum data search cluster of round-trip delay wherein,
And search for cluster using the data search cluster as the target data for handling the data search request.
The second way, request Distributor determine the minimum number of round-trip delay at least two data search clusters
According to search cluster, and judge that the minimum data search cluster of round-trip delay is abnormal with the presence or absence of service；If round-trip delay minimum
There is service exception in data search cluster, then asks Distributor to be determined further according to the network state of the 3rd data search cluster
Target data searches for cluster.
Wherein, the 3rd data search cluster is except the minimum data of round-trip delay are searched at least two data search clusters
Other data search clusters outside Suo Jiqun.
For the second way, after determining that there is the target data search cluster of minimum round-trip delay, general request
Distributor can also determine whether the cluster of target data search, and whether storage service is abnormal, for example, packet loss whether
It is high whether to be higher by a predetermined threshold value to beyond a defined threshold or load.If the target data searches for cluster presence service
Abnormal, then the embodiment of the present invention still can be to ensure that service availability as criterion, continues further according to the 3rd data search cluster
Network state determines that the target data for handling the data search request searches for cluster, that is, switches the data search request to other clothes
Normal data search cluster of being engaged in is handled.Specific selection flow is consistent with aforesaid way, and here is omitted.
The data search request is forwarded to target data search cluster by 303b, request Distributor.
304b, target data search for cluster after the data search request is received, at the data search request
Reason, and obtained result is sent to request Distributor.
In embodiments of the present invention, if the data search request is only to be related to the inquiry to a certain item data, target
Data search cluster carries out data search according only to the data search request in data storage, and is finding needed for user
Data after, be sent to request Distributor using the data needed for the user as result.If the data search please
The modification further related to a certain item data is asked, for example changes the pet name of the user in social networking application, then target data is searched for
Cluster is carrying out data search according to the data search request in data storage, and data storage is complete finding target
Into after modification, the modification result of target data storage can be sent to request Distributor as result.
Result is forwarded to requesting terminal by 305b, request Distributor.
It should be noted that be, above-mentioned steps 301b to step 305b is step 1 shown in corresponding diagram 2A to 3.In addition,
, can also be according to above-mentioned steps 303a to step if the data search request that this requesting terminal is sent is related to the modification to data
The synchronized update of the rapid 306a complete paired data of mode.Accordingly, i.e., the step 4 shown in corresponding diagram 2A is to 5.
Method provided in an embodiment of the present invention, by deploying the mutually redundant multiple data search clusters of data and dividing
Not Wei each data search cluster it is supporting deploy one subscription message server so that on each data search cluster
Can complete paired data renewal, and support by matching subscription message server the renewal of data will be synchronized to it is trans-regional
Other data search clusters, so compared to be only capable of main cluster carry out data renewal mode, greatly reduce renewal behaviour
The RTT of work.
Further, since the data stored on each data search cluster keep realtime uniform, each data search collection
Group can provide complete search service, even if service exception occurs for one or several data search clusters, can still ensure
Search service can use, therefore greatly improve the disaster tolerance performance of system.
In addition, the realization of data syn-chronization of the embodiment of the present invention is without third-party support so that the stability of search service
It is not only restricted to third party.
In addition, by deploying request Distributor so that data search request can be assigned to neighbouring data
Search cluster is handled, and realizes the localized responses of request, is greatly reduced transmission of the request in network link and is taken,
So that the RTT of respond request is substantially reduced, ensure that the request of different regions can meet with a response in the short period of time.
The embodiments of the invention provide a kind of data search system.Wherein, the data search system includes request distribution clothes
Business device, at least two data search clusters and the subscription message server matched with least two data search number of clusters.
Referring to Fig. 4, the first data search cluster 401, for when data renewal operation occurs, generation to update for data
The data new information of operation；
First data search cluster 401, it is additionally operable to data new information being distributed to the first subscription messenger service of matching
Device；
First subscribes to message server 402, for data new information to be pushed into the second data search cluster；
Second data search cluster 403, for synchronizing data renewal according to data new information；
Wherein, the first data search cluster 401 is that any data at least two data search clusters searches for cluster, the
Two data search clusters 403 are other numbers in addition to the first data search cluster 401 at least two data search clusters
According to search cluster.
In another embodiment, first message server 402 is subscribed to, is additionally operable to receive the second data search cluster 403
The message subscribing request of transmission, the identification information of the second data search cluster 403 is comprised at least in message subscribing request；
First subscribes to message server 402, is additionally operable to the identification information according to the second data search cluster 403, to carrying out
Second data search cluster 403 of message subscribing is recorded；
First subscribes to message server 402, for the identification information according to the second data search cluster 403, by data more
New information pushes to the second data search cluster 403.
In another embodiment, referring to Fig. 5, Distributor 404 is asked, for receiving requesting terminal transmission
After data search request, the network state of each data search cluster at least two data search clusters is obtained；
Distributor 404 is asked, for the network state according to each data search cluster, at least two data
Search in cluster and determine the target data search cluster of processing data searching request, and data search request is forwarded to number of targets
According to search cluster；
Target data searches for cluster 405, for after data search request is received, at data search request
Reason, and obtained result is sent to request Distributor；
Distributor 404 is asked, is additionally operable to result being forwarded to requesting terminal.
In another embodiment, Distributor 404 is asked, it is past for being determined at least two data search clusters
The minimum data search cluster of time delay is returned, the minimum data search cluster of round-trip delay is defined as target data search cluster
405。
In another embodiment, Distributor 404 is asked, it is past for being determined at least two data search clusters
Return the minimum data search cluster of time delay；Judge that the minimum data search cluster of round-trip delay is abnormal with the presence or absence of service；It is if past
Return the minimum data search cluster of time delay and service exception occur, then determine mesh further according to the network state of the 3rd data search cluster
Mark data search cluster 405；
Wherein, the 3rd data search cluster is except the minimum data of round-trip delay are searched at least two data search clusters
Other data search clusters outside Suo Jiqun.
System provided in an embodiment of the present invention, by deploying the mutually redundant multiple data search clusters of data and dividing
Not Wei each data search cluster it is supporting deploy one subscription message server so that on each data search cluster
Can complete paired data renewal, and support by matching subscription message server the renewal of data will be synchronized to it is trans-regional
Other data search clusters, so compared to be only capable of main cluster carry out data renewal mode, greatly reduce renewal behaviour
The RTT of work.
Further, since the data stored on each data search cluster keep realtime uniform, each data search collection
Group can provide complete search service, even if service exception occurs for one or several data search clusters, can still ensure
Search service can use, therefore greatly improve the disaster tolerance performance of system.
In addition, the realization of data syn-chronization of the embodiment of the present invention is without third-party support so that the stability of search service
It is not only restricted to third party.
In addition, by deploying request Distributor so that data search request can be assigned to neighbouring data
Search cluster is handled, and realizes the localized responses of request, is greatly reduced transmission of the request in network link and is taken,
So that the RTT of respond request is substantially reduced, ensure that the request of different regions can meet with a response in the short period of time.
One of ordinary skill in the art will appreciate that hardware can be passed through by realizing all or part of step of above-described embodiment
To complete, by program the hardware of correlation can also be instructed to complete, described program can be stored in a kind of computer-readable
In storage medium, storage medium mentioned above can be read-only storage, disk or CD etc..
The foregoing is only presently preferred embodiments of the present invention, be not intended to limit the invention, it is all the present invention spirit and
Within principle, any modification, equivalent substitution and improvements made etc., it should be included in the scope of the protection.
Claims (10)
1. a kind of method of data synchronization, it is characterised in that applied to data search system, the data search system includes request
Distributor, at least two data search clusters and disappear with the subscription that at least two data search number of clusters matches
Server is ceased, methods described includes：
When data renewal operation occurs for the first data search cluster, the first data search cluster generation is directed to the data
Update the data new information of operation；
The data new information is distributed to the first subscription message server of matching by the first data search cluster；
The data new information is pushed to the second data search cluster by the first subscription message server, so that described the
Two data search clusters synchronize data renewal according to the data new information；
Wherein, the first data search cluster is that any data at least two data search cluster searches for cluster,
The second data search cluster be at least two data search cluster in addition to the first data search cluster
Other data search clusters.
2. according to the method for claim 1, it is characterised in that methods described also includes：
Described first subscribes to the message subscribing request that message server receives the second data search collection pocket transmission, the message
Subscribe to the identification information that the second data search cluster is comprised at least in request；
Described first subscribes to identification information of the message server according to the second data search cluster, to carrying out message subscribing
The second data search cluster is recorded；
The data new information is pushed to the second data search cluster by the first subscription message server, including：
Described first subscribes to identification information of the message server according to the second data search cluster, and data renewal is disappeared
Breath pushes to the second data search cluster.
3. according to the method for claim 1, it is characterised in that methods described also includes：
The request Distributor obtains at least two number after the data search request of requesting terminal transmission is received
According to the network state of each data search cluster in search cluster；
The network state of request Distributor each data search cluster according to, at least two data
Search in cluster and determine the target data search cluster for handling the data search request；
The data search request is forwarded to target data search cluster by the request Distributor；
The target data searches for cluster after the data search request is received, at the data search request
Reason, and obtained result is sent to the request Distributor；
The result is forwarded to the requesting terminal by the request Distributor.
4. according to the method for claim 3, it is characterised in that the request Distributor is according to each described data
The network state of cluster is searched for, determines to handle the target of the data search request at least two data search cluster
Data search cluster, including：
The request Distributor determines the minimum data search of round-trip delay at least two data search cluster
Cluster；
The minimum data search cluster of the round-trip delay is defined as the target data and searched for by the request Distributor
Cluster.
5. according to the method for claim 3, it is characterised in that the request Distributor is according to each described data
The network state of cluster is searched for, determines to handle the target of the data search request at least two data search cluster
Data search cluster, including：
The request Distributor determines the minimum data search of round-trip delay at least two data search cluster
Cluster；
The request Distributor judges that the minimum data search cluster of the round-trip delay is abnormal with the presence or absence of service；
If service occur abnormal for the minimum data search cluster of the round-trip delay, the request Distributor is further according to the
The network state of three data search clusters determines the target data search cluster；
Wherein, the 3rd data search cluster is except the round-trip delay is minimum at least two data search cluster
Data search cluster outside other data search clusters.
6. a kind of data search system, it is characterised in that the data search system includes request Distributor, at least two
Data search cluster and the subscription message server matched with least two data search number of clusters；
First data search cluster, for when data renewal operation occurs, generation to be directed to the data of data renewal operation
New information；
The first data search cluster, it is additionally operable to the data new information being distributed to the first subscription messenger service of matching
Device；
Described first subscribes to message server, for the data new information to be pushed into the second data search cluster；
The second data search cluster, for synchronizing data renewal according to the data new information；
Wherein, the first data search cluster is that any data at least two data search cluster searches for cluster,
The second data search cluster be at least two data search cluster in addition to the first data search cluster
Other data search clusters.
7. system according to claim 6, it is characterised in that described first subscribes to message server, is additionally operable to receive institute
The message subscribing request of the second data search collection pocket transmission is stated, is searched in the message subscribing request including at least second data
Suo Jiqun identification information；
Described first subscribes to message server, the identification information according to the second data search cluster is additionally operable to, to disappearing
The second data search cluster that breath is subscribed to is recorded；
Described first subscribes to message server, for the identification information according to the second data search cluster, by the data
New information pushes to the second data search cluster.
8. system according to claim 6, it is characterised in that the request Distributor, for receiving request
After the data search request that terminal is sent, the net of each data search cluster at least two data search cluster is obtained
Network state；
The request Distributor, for the network state according to each data search cluster, described at least two
Determine that the target data for handling the data search request searches for cluster in individual data search cluster, and please by the data search
Ask and be forwarded to target data search cluster；
The target data searches for cluster, for after the data search request is received, entering to the data search request
Row processing, and obtained result is sent to the request Distributor；
The request Distributor, it is additionally operable to the result being forwarded to the requesting terminal.
9. system according to claim 8, it is characterised in that the request Distributor, for described at least two
The minimum data search cluster of round-trip delay is determined in individual data search cluster, by the data search collection that the round-trip delay is minimum
Group is defined as the target data search cluster.
10. system according to claim 8, it is characterised in that the request Distributor, for described at least two
The minimum data search cluster of round-trip delay is determined in individual data search cluster；Judge the minimum data search of the round-trip delay
Cluster is abnormal with the presence or absence of service；If service occur abnormal for the minimum data search cluster of the round-trip delay, further according to the
The network state of three data search clusters determines the target data search cluster；
Wherein, the 3rd data search cluster is except the round-trip delay is minimum at least two data search cluster
Data search cluster outside other data search clusters.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710568992.1A CN107370809A (en) | 2017-07-13 | 2017-07-13 | Method of data synchronization and data search system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710568992.1A CN107370809A (en) | 2017-07-13 | 2017-07-13 | Method of data synchronization and data search system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107370809A true CN107370809A (en) | 2017-11-21 |
Family
ID=60308119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710568992.1A Pending CN107370809A (en) | 2017-07-13 | 2017-07-13 | Method of data synchronization and data search system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107370809A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109831473A (en) * | 2017-11-23 | 2019-05-31 | 菜鸟智能物流控股有限公司 | Logistics service providing method and equipment |
| CN110278109A (en) * | 2019-05-21 | 2019-09-24 | 阿里巴巴集团控股有限公司 | A kind of disaster recovery method and system |
| CN110532102A (en) * | 2019-09-02 | 2019-12-03 | 中国工商银行股份有限公司 | Implementation method, device, electronic equipment and the readable storage medium storing program for executing of application function |
| CN110609845A (en) * | 2018-06-15 | 2019-12-24 | 网宿科技股份有限公司 | Big data redundancy disaster recovery method, big data service system and query method |
| CN111314416A (en) * | 2019-12-30 | 2020-06-19 | 广州市百果园网络科技有限公司 | Data synchronization method, device, system, equipment and storage medium |
| CN113709250A (en) * | 2021-08-31 | 2021-11-26 | 中国电子科技集团公司第二十八研究所 | Cross-domain user data synchronization method based on subscription sending mode |
| CN115766303A (en) * | 2022-10-14 | 2023-03-07 | 中信科移动通信技术股份有限公司 | Broadband cluster system group information synchronization method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104462483A (en) * | 2014-12-18 | 2015-03-25 | 北京奇虎科技有限公司 | Data synchronizing method, device and system |
| CN106375362A (en) * | 2015-07-24 | 2017-02-01 | 广州市百果园信息技术有限公司 | Cache synchronization method and system for distributed server |
-
2017
- 2017-07-13 CN CN201710568992.1A patent/CN107370809A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104462483A (en) * | 2014-12-18 | 2015-03-25 | 北京奇虎科技有限公司 | Data synchronizing method, device and system |
| CN106375362A (en) * | 2015-07-24 | 2017-02-01 | 广州市百果园信息技术有限公司 | Cache synchronization method and system for distributed server |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109831473A (en) * | 2017-11-23 | 2019-05-31 | 菜鸟智能物流控股有限公司 | Logistics service providing method and equipment |
| CN110609845A (en) * | 2018-06-15 | 2019-12-24 | 网宿科技股份有限公司 | Big data redundancy disaster recovery method, big data service system and query method |
| CN110278109A (en) * | 2019-05-21 | 2019-09-24 | 阿里巴巴集团控股有限公司 | A kind of disaster recovery method and system |
| CN110278109B (en) * | 2019-05-21 | 2022-02-01 | 创新先进技术有限公司 | Disaster recovery method and system |
| CN110532102A (en) * | 2019-09-02 | 2019-12-03 | 中国工商银行股份有限公司 | Implementation method, device, electronic equipment and the readable storage medium storing program for executing of application function |
| CN111314416A (en) * | 2019-12-30 | 2020-06-19 | 广州市百果园网络科技有限公司 | Data synchronization method, device, system, equipment and storage medium |
| CN111314416B (en) * | 2019-12-30 | 2022-12-09 | 广州市百果园网络科技有限公司 | Data synchronization method, device, system, equipment and storage medium |
| CN113709250A (en) * | 2021-08-31 | 2021-11-26 | 中国电子科技集团公司第二十八研究所 | Cross-domain user data synchronization method based on subscription sending mode |
| CN113709250B (en) * | 2021-08-31 | 2023-09-15 | 中国电子科技集团公司第二十八研究所 | Cross-domain user data synchronization method based on subscription transmission mode |
| CN115766303A (en) * | 2022-10-14 | 2023-03-07 | 中信科移动通信技术股份有限公司 | Broadband cluster system group information synchronization method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107370809A (en) | Method of data synchronization and data search system | |
| CN107801086B (en) | The dispatching method and system of more cache servers | |
| CN100588172C (en) | System and method for realizing network reserved storage | |
| CN101535965B (en) | Technique for improving scalability and portability of a storage management system | |
| Ramanathan et al. | Finding good peers in peer-to-peer networks | |
| CN111405019B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN106202416B (en) | Table data write method and device, table data read method and device | |
| CN102882985A (en) | File sharing method based on cloud storage | |
| CN102365634A (en) | System for managing distributed assets and metadata | |
| US8140645B2 (en) | Index server support to file sharing applications | |
| US20030101267A1 (en) | Peer-to-peer caching network | |
| JP2004192170A (en) | Access relay device | |
| CN104011701A (en) | Content delivery network | |
| CN112035420B (en) | Data sharing method, sharing device and system | |
| KR101545626B1 (en) | System for interoperation between dds and dbms | |
| CN107888666A (en) | A kind of cross-region data-storage system and method for data synchronization and device | |
| CN110399348A (en) | File deletes method, apparatus, system and computer readable storage medium again | |
| EP2203840A2 (en) | Exchange of syncronization data and metadata | |
| CN110784498B (en) | Personalized data disaster tolerance method and device | |
| CN109325056A (en) | A kind of big data processing method and processing device, communication equipment | |
| EP2025133B1 (en) | Repository synchronization in a ranked repository cluster | |
| US10025859B2 (en) | Method and system for second-degree friend query | |
| CN107493309B (en) | File writing method and device in distributed system | |
| CN110290196A (en) | A kind of content distribution method and device | |
| CN111400266B (en) | Data processing method and system, and diagnosis processing method and device for operation event |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20171121 |