WO2023185071A1

WO2023185071A1 - Data query method, data writing method, related apparatus and system

Info

Publication number: WO2023185071A1
Application number: PCT/CN2022/136753
Authority: WO
Inventors: 张进
Original assignee: 北京沃东天骏信息技术有限公司; 北京京东世纪贸易有限公司
Priority date: 2022-03-31
Filing date: 2022-12-06
Publication date: 2023-10-05
Also published as: CN116932505A

Abstract

Disclosed are a data query method, a data writing method, a related apparatus and a system. A specific embodiment comprises: in response to a received data query request, obtaining a liveness probe result for each storage cluster, the liveness probe result indicating whether a corresponding storage cluster is currently available; according to the liveness probe result for each storage cluster, determining an available storage cluster list; and selecting a target available storage cluster from the available storage cluster list according to a cluster selection setting, and sending the data query request to the target available storage cluster, so that the target available storage cluster returns a corresponding data query result.

Description

Data query method, data writing method, related devices and systems

Cross-references to related applications

This application claims priority to Chinese patent application No. 202210331107.9 titled "A data query method, data writing method, related devices and systems" submitted on March 31, 2022. The above Chinese patent application is cited in full here. The disclosed content shall be included as part or all of this application.

Technical field

The present disclosure relates to the field of computer technology, and in particular, to a data query method and device, a data writing method and device, and a data query system.

Background technique

There are currently two options for cluster deployment: Option 1 is a cluster deployment based on ElasticSearch (a full-text retrieval framework, a search engine written in Java language), with one cluster limited to the same computer room; Option 2 is a cluster deployment across computer rooms, to When the cluster writes data or reads data, communication needs to occur within the cluster.

In the process of realizing the present disclosure, the inventor found that there are at least the following problems in the prior art:

Option 1 cannot achieve disaster recovery deployment between computer rooms. Once a computer room-level failure occurs, the entire cluster will be unavailable, and eventually the service will be unavailable. Option 2 will reduce the efficiency of communication within the cluster. When writing data to the cluster or reading data, When fetching data, it takes a lot of time due to the need for communication within the cluster.

Contents of the invention

In view of this, embodiments of the present disclosure provide a data query method, a data writing method, related devices and systems, which can achieve consistency in writing data to multiple clusters, automatically switch clusters, and improve the disaster tolerance of cluster deployment. , which solves the problem of unavailability of the entire cluster when a computer room level failure occurs, and at the same time overcomes the time-consuming drawback of communication within the cluster.

To achieve the above object, according to one aspect of an embodiment of the present disclosure, a data query method is provided.

A data query method, including: in response to a received data query request, obtaining an activity detection result for each storage cluster, the activity detection result indicating whether the corresponding storage cluster is currently available; according to the activity detection result for each storage cluster, Determine a list of available storage clusters; select a target available storage cluster from the list of available storage clusters according to the cluster selection settings, and send the data query request to the target available storage cluster, so that the corresponding target available storage cluster is returned data query results.

In one or more embodiments of the present disclosure, obtaining the live detection results for each storage cluster includes: obtaining the live detection results for each storage cluster from a configuration center or a local cache, where , the configuration center, through a dynamic listening mechanism, after listening to the detection results of the storage cluster by the detection node cluster, caches the corresponding detection results into the local cache.

In one or more embodiments of the present disclosure, the active detection node cluster includes one or more active detection nodes, and the active detection nodes are used to send health degree requests to each storage cluster respectively to determine the health status according to the The storage cluster's response to the health request determines whether the corresponding storage cluster is available.

In one or more embodiments of the present disclosure, determining the list of available storage clusters based on the detection results of each storage cluster includes: for each storage cluster, determining the activity of each detection node for the storage cluster. In the detection result, it is indicated whether the ratio of the number of detection nodes available to the storage cluster to the total number of detection nodes exceeds the proportion threshold. If so, it is determined that the storage cluster is available and added to the list of available storage clusters.

In one or more embodiments of the present disclosure, according to the cluster selection setting, a target available storage cluster is selected from the list of available storage clusters, and the data query request is sent to the target available storage cluster to be sent by The target available storage cluster returns corresponding data query results, including: taking all available storage clusters in the available storage cluster list as the target available storage clusters, and receiving the data query results returned by one of the target available storage clusters. Finally, output the data query results.

In one or more embodiments of the present disclosure, according to the cluster selection setting, a target available storage cluster is selected from the list of available storage clusters, and the data query request is sent to the target available storage cluster to be sent by The target available storage cluster returns corresponding data query results, including: selecting an available storage cluster currently polled from the list of available storage clusters according to the polling strategy as the target available storage cluster. After the target can store the data query results returned by the cluster, it outputs the data query results.

In one or more embodiments of the present disclosure, according to the cluster selection setting, a target available storage cluster is selected from the list of available storage clusters, and the data query request is sent to the target available storage cluster to be sent by The target available storage cluster returns corresponding data query results, including: randomly selecting an available storage cluster from the list of available storage clusters as the target available storage cluster, and receiving the returned data from the target available storage cluster. After the data query results are obtained, the data query results are output.

According to another aspect of an embodiment of the present disclosure, a data writing method is provided.

A data writing method includes: initiating multiple monitoring of message middleware topics according to the number of storage clusters, each monitoring corresponding to a storage cluster, the message middleware topic includes data to be written; in response to the Each listener of the message middleware topic writes the data to be written to the corresponding storage cluster.

According to another aspect of the embodiments of the present disclosure, a data query device is provided.

A data query device, including: a detection result acquisition module, configured to obtain the detection results of each storage cluster in response to a received data query request, the detection results indicating whether the corresponding storage cluster is currently available; available storage The cluster determination module is used to determine the available storage cluster list based on the detection results of each storage cluster; the data query module is used to select the target available storage cluster from the available storage cluster list according to the cluster selection settings, and add the The data query request is sent to the target available storage cluster, so that the corresponding data query result is returned by the target available storage cluster.

According to yet another aspect of an embodiment of the present disclosure, a data writing device is provided.

A data writing device includes: a monitoring module, configured to initiate multiple monitoring of message middleware topics according to the number of storage clusters. Each monitoring corresponds to a storage cluster, and the message middleware topic includes data to be written. ; A data writing module, configured to write the data to be written to the corresponding storage cluster in response to each monitoring of the message middleware topic.

According to yet another aspect of the embodiments of the present disclosure, a data query system is provided.

A data query system includes: the data query device provided by the embodiment of the present disclosure, and also includes each storage cluster, a configuration center and a live detection node cluster, wherein: the live detection node cluster includes one or more live detection nodes, so The health detection node is used to send health request to each storage cluster respectively, so as to determine the health detection result of the storage cluster according to the response of the storage cluster to the health request. The health detection result indicates Whether the corresponding storage cluster is currently available; the configuration center is configured to receive the detection results reported by each of the detection nodes, and provide the detection results to the data query device.

According to yet another aspect of embodiments of the present disclosure, an electronic device is provided.

An electronic device, including: one or more processors; a memory for storing one or more programs, when the one or more programs are executed by the one or more processors, such that the one or more programs Multiple processors implement the data query method or data writing method provided by the embodiments of the present disclosure.

According to yet another aspect of embodiments of the present disclosure, a computer-readable medium is provided.

A computer-readable medium on which a computer program is stored. When the computer program is executed by a processor, the data query method or the data writing method provided by the embodiment of the present disclosure is implemented.

Further effects of the above-mentioned non-conventional optional methods will be described below in conjunction with specific implementations.

Description of drawings

The accompanying drawings are used for a better understanding of the present disclosure and do not constitute an improper limitation of the present disclosure. in:

Figure 1 is a schematic diagram of the main steps of a data query method according to one or more embodiments of the present disclosure;

Figure 2 is a schematic diagram of the main steps of a data writing method according to one or more embodiments of the present disclosure;

Figure 3 is a flow diagram of data query and data writing according to one or more embodiments of the present disclosure;

Figure 4 is a schematic diagram of the main modules of a data query device according to one or more embodiments of the present disclosure;

Figure 5 is a schematic diagram of the main modules of a data writing device according to one or more embodiments of the present disclosure;

Figure 6 is a schematic diagram of the main architecture of a data query system according to one or more embodiments of the present disclosure;

Figure 7 is an exemplary system architecture diagram to which one or more embodiments of the present disclosure may be applied;

Figure 8 is a schematic structural diagram of a computer system suitable for implementing a terminal device or server in one or more embodiments of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the present disclosure are included to facilitate understanding and should be considered to be exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

Figure 1 is a schematic diagram of the main steps of a data query method according to one or more embodiments of the present disclosure.

As shown in Figure 1, the data query method in one or more embodiments of the present disclosure mainly includes the following steps S101 to S103.

Step S101: In response to the received data query request, obtain the activity detection results of each storage cluster. The activity detection results indicate whether the corresponding storage cluster is currently available.

Obtaining the live detection results of each storage cluster may include: obtaining the live detection results of each storage cluster from the configuration center or the local cache. Among them, the configuration center uses a dynamic monitoring mechanism to detect the live node cluster. After storing the cluster's liveness detection results, cache the corresponding liveness detection results in the local cache.

The live detection node cluster may include one or more live detection nodes. The live detection nodes are used to send health status requests to each storage cluster respectively to determine whether the corresponding storage cluster is available based on the storage cluster's response to the health status request. There are multiple storage clusters in this disclosure. The storage cluster is such as an ES (Elastic Search) cluster. When the active node sends a health request to each storage cluster, the command provided by the ES cluster specifically for obtaining health can be used to obtain the health status of the cluster. Cluster health.

Step S102: Determine the list of available storage clusters based on the detection results of each storage cluster.

Determining the list of available storage clusters based on the detection results of each storage cluster may include: for each storage cluster, determining the detection results of each detection node for the storage cluster to indicate the number of detection nodes available for the storage cluster. Whether the proportion of the total number of active nodes exceeds the proportion threshold, if so, the storage cluster is determined to be available and added to the list of available storage clusters. The liveness detection result can include a timeout field. When making a health request, you need to specify a storage cluster response time, such as 1s (seconds). If the storage cluster returns the result within 1s, the storage cluster is considered alive. Timeout=false in the result indicates that the storage cluster is alive. If it times out, the storage cluster is considered to have failed. For multiple live detection nodes, the proportion threshold can be set as needed. For example, if the detection results corresponding to a storage cluster returned by more than half (proportion threshold = 50%) of the live detection nodes are that the storage cluster is alive, it indicates that the storage cluster is alive. If the storage cluster is available, it is determined that the storage cluster is available based on the detection results.

Step S103: According to the cluster selection settings, select the target available storage cluster from the available storage cluster list, and send the data query request to the target available storage cluster, so that the target available storage cluster returns the corresponding data query result.

In one embodiment, according to the cluster selection settings, select the target available storage cluster from the available storage cluster list, send the data query request to the target available storage cluster, so that the target available storage cluster returns the corresponding data query result, which may include: Use all available storage clusters in the available storage cluster list as the target available storage cluster, and after receiving the data query result returned by one of the target available storage clusters, output the data query result.

In another embodiment, according to the cluster selection settings, select the target available storage cluster from the available storage cluster list, send the data query request to the target available storage cluster, so that the target available storage cluster returns the corresponding data query result, which may include : According to the polling policy, select an available storage cluster currently polled from the list of available storage clusters as the target available storage cluster. After receiving the data query results returned by the target available storage cluster, output the data query results.

In another embodiment, according to the cluster selection settings, select the target available storage cluster from the available storage cluster list, send the data query request to the target available storage cluster, so that the target available storage cluster returns the corresponding data query result, which may include : Randomly select an available storage cluster from the list of available storage clusters as the target available storage cluster. After receiving the data query results returned by the target available storage cluster, output the data query results.

FIG. 2 is a schematic diagram of the main steps of a data writing method according to one or more embodiments of the present disclosure.

As shown in Figure 2, the data writing method according to one embodiment of the present disclosure mainly includes the following steps S201 to S202.

Step S201: According to the number of storage clusters, initiate multiple monitoring of the message middleware topic. Each monitoring corresponds to a storage cluster. The message middleware topic includes data to be written.

Step S202: In response to each monitoring of the message middleware topic, write the data to be written into the corresponding storage cluster.

Message middleware such as Message Queuing (MQ).

In response to each monitoring of the message middleware topic, writing the data to be written to the corresponding storage cluster may include: in response to each monitoring of the message middleware topic, writing the data to be written through the preset writing interface. The data is written to the corresponding storage cluster. The parameters of the preset writing interface include the identification (ID) of each storage cluster. If the data to be written in any storage cluster fails to be written, the storage cluster will be retried. The operation to write the data to be written.

In this disclosed embodiment, when the business needs to write data to the ES, the data is sent to a message queue topic, and the data is written to the ES by monitoring the message queue topic. In this way, the writing operation and the final The write action is decoupled, and multiple storage clusters will listen to this message queue topic multiple times when writing. If a storage cluster fails to write, it only needs to be retried by the storage cluster. Currently, most message queue middleware Final consistency can be achieved, and this feature can be applied to achieve final consistency of multi-storage cluster data and greatly enhance scalability.

Figure 3 is a schematic flowchart of data query and data writing according to one or more embodiments of the present disclosure. The client may be a device used for data query and/or data writing.

As shown in Figure 3, in one embodiment, the configuration center uses a dynamic monitoring mechanism to listen to the detection results of the storage cluster by the detection node cluster, and caches the corresponding detection results into the local cache, and the client responds The received data query request obtains the live detection results of each storage cluster by the active node cluster from the configuration center or local cache.

Specifically, the storage cluster can be an ES (ElasticSearch) cluster, and ES clusters (clusters for short) are deployed in multiple computer rooms. Each cluster can be named cluster1, cluster2..., and all clusters can be collectively called es-cluster; build a configuration center , the configuration center can be zookeeper (a high-availability distributed configuration center); build a live detection node cluster, the live detection node cluster can include at least 3 live detection nodes, and the live detection nodes can be named node1, node2, node3...( As shown in Figure 3, three nodes are used as an example, namely node 1, node 2, and node 3). Each active node is connected to each ES cluster; build a client for data query, and establish a client for each ES. Connection to the cluster and connection to the configuration center. Start the liveness detection program. Each liveness detection node in the liveness detection node cluster periodically initiates a health request to each ES cluster. The ES cluster responds to the health request to generate a liveness detection result and stores it in the es-cluster directory. The configuration center dynamically monitors the es-cluster directory. When the directory changes, the configuration center modifies the client's local cache and saves the detection results to the client's local cache. The client obtains the live detection results from the configuration center or local cache. Among them, local caching can be implemented through Map (an object that maps keys to values).

In one embodiment, for each storage cluster, it is determined whether the ratio of the number of active detection nodes available to the storage cluster to the total number of active detection nodes in the detection result of each active detection node for the storage cluster exceeds the proportion threshold. If so, , it is determined that the storage cluster is available and added to the list of available storage clusters. For example, there are two storage clusters (a, b), the number of active detection nodes is 3, and the detection results of the three active detection nodes on the storage cluster are (1, 1), (0, 1), (0 , 0), where 1 means available and 0 means unavailable. The first number in the detection result is the available status of the first cluster, and the second number is the available status of the second cluster. Then there is a detection result. The activity detection results of nodes indicate that cluster a is available, and the activity detection results of two active nodes indicate that cluster b is available. If the proportion threshold is set to 0.5, then for cluster a, the proportion of the number of active detection nodes available in the cluster to the total number of active detection nodes is 1/3. If the proportion threshold of 0.5 is not exceeded, cluster a will be unavailable and cluster a will not be used. Add to the list of available storage clusters; for cluster b, the ratio of the number of active active nodes indicating that the cluster is available to the total number of active active nodes is 2/3, exceeding the proportion threshold of 0.5, then cluster b is available, and cluster b is added to the available storage Cluster list. Unavailable storage clusters can be reported for repair.

In one embodiment, according to the cluster selection settings, the target available storage cluster is selected from the available storage cluster list, and the data query request is sent to the target available storage cluster, so that the target available storage cluster returns the corresponding data query result. Specifically, the load balancing strategy is determined according to the load size of the cluster to perform cluster selection settings. The load balancing strategy can include all request strategies, polling strategies, and random strategies. Among them, the all request strategy uses all available storage clusters in the available storage cluster list as the target available storage cluster. After receiving the data query result returned by one of the target available storage clusters, the data query result is output. This strategy can flatten the network. The impact of fluctuations and jitter of a certain storage cluster on the entire cluster is that as long as one storage cluster is normal, services can be provided normally, which is suitable for situations where the overall pressure on the cluster is not very high. Polling strategy, that is, according to the polling strategy, select an available storage cluster currently polled from the list of available storage clusters as the target available storage cluster. After receiving the data query results returned by the target available storage cluster, output the data query result. The random strategy uses a random algorithm to randomly select an available storage cluster from the list of available storage clusters each time as the target available storage cluster. After receiving the data query results returned by the target available storage cluster, the data query results are output.

In one embodiment, multiple monitoring of the message middleware topic is initiated according to the number of storage clusters. Each monitoring corresponds to a storage cluster. In response to each monitoring of the message middleware topic, through the preset writing interface Write the data to be written into the corresponding storage cluster. The parameters of the preset writing interface include the identification of each storage cluster. Specifically, when there is a writing action, the data to be written is sent to the message middleware topic. According to the identification of each storage cluster to be written included in the writing interface parameters, the data to be written is established by mapping. Each storage cluster subscribes to the message middleware topic to monitor the message middleware topic. The number of subscriptions is the same as the number of storage clusters to be written. Each storage cluster writes the data to be written to each storage cluster by listening to this message middleware topic multiple times. If a storage cluster fails to write, you only need to retry the storage cluster. Based on the message middleware, the disclosed embodiment can achieve the ultimate consistency of each storage cluster and greatly enhance the scalability.

Figure 4 is a schematic diagram of the main modules of a data query device according to one or more embodiments of the present disclosure.

As shown in Figure 4, a data query device 400 according to an embodiment of the present disclosure mainly includes: a life detection result acquisition module 401, an available storage cluster determination module 402, and a data query module 403.

The activity detection result acquisition module 401 is used to obtain the activity detection results of each storage cluster in response to the received data query request, and the activity detection results indicate whether the corresponding storage cluster is currently available.

The available storage cluster determination module 402 is used to determine a list of available storage clusters based on the detection results of each storage cluster.

The data query module 403 is used to select a target available storage cluster from the available storage cluster list according to the cluster selection settings, and send a data query request to the target available storage cluster, so that the target available storage cluster returns the corresponding data query result.

In one embodiment, the active detection result acquisition module 401 is specifically used to: obtain the active detection result of each storage cluster by the active detection node cluster from the configuration center or the local cache, wherein the configuration center uses a dynamic monitoring mechanism to detect the active detection result. After the live node cluster stores the liveness detection results of the cluster, it caches the corresponding liveness detection results into the local cache.

In one embodiment, the available storage cluster determination module 402 is specifically configured to: for each storage cluster, determine the active detection results of each active node for the storage cluster, and indicate the number of active active nodes available for the storage cluster. Whether the proportion of the total number of nodes exceeds the proportion threshold, if so, the storage cluster is determined to be available and added to the list of available storage clusters.

In one embodiment, the data query module 403 is specifically configured to: use all available storage clusters in the available storage cluster list as target available storage clusters, and output the data query after receiving the data query result returned by one of the target available storage clusters. result.

In one embodiment, the data query module 403 is specifically configured to: select an available storage cluster that is currently polled from the list of available storage clusters according to the polling policy, as the target available storage cluster, and return after receiving the target available storage cluster. After the data query results, output the data query results.

In one embodiment, the data query module 403 is specifically configured to: randomly select an available storage cluster from the list of available storage clusters as the target available storage cluster, and output the data after receiving the data query results returned by the target available storage cluster. search result.

In addition, the specific implementation content of the data query device in the embodiment of the present disclosure has been described in detail in the above data query method, so the repeated content will not be described here.

Figure 5 is a schematic diagram of the main modules of a data writing device according to one or more embodiments of the present disclosure.

As shown in Figure 5, a data writing device 500 according to an embodiment of the present disclosure mainly includes: a monitoring module 501 and a data writing module 502.

The monitoring module 501 is used to initiate multiple monitoring of message middleware topics according to the number of storage clusters. Each monitoring corresponds to a storage cluster, and the message middleware topic includes data to be written.

The data writing module 502 is configured to respond to each monitoring of the message middleware topic and write the data to be written into the corresponding storage cluster.

In one embodiment, the data writing module 502 is specifically configured to: in response to each monitoring of the message middleware topic, write the data to be written to the corresponding storage cluster through the preset writing interface. The preset writing interface The parameters of the incoming interface include the identification of each storage cluster. If the writing of the data to be written in any storage cluster fails, the operation of writing the data to be written to the storage cluster is retried.

In addition, the specific implementation content of the data writing device in the embodiment of the present disclosure has been described in detail in the above data writing method, so the repeated content will not be described here.

Figure 6 is a schematic diagram of the main architecture of a data query system according to one or more embodiments of the present disclosure.

As shown in Figure 6, a data query system 600 according to an embodiment of the present disclosure mainly includes: a data query device 601, and also includes a storage cluster 602, a configuration center 603, and a live detection node cluster 604. The number of storage clusters 602 is multiple.

The data query device 601 has the same functional modules as the data query device 400 in the above embodiment.

The live detection node cluster 604 includes one or more live detection nodes. The live detection nodes are used to send health request to each storage cluster respectively to determine the live detection result of the storage cluster based on the response of the storage cluster to the health request. The live result indicates whether the corresponding storage cluster is currently available.

The configuration center 603 is configured to receive the activity detection results reported by each activity detection node, and provide the activity detection results to the data query device.

In one embodiment, the configuration center 603 is specifically configured to provide the data query device 601 with a probe result in response to the probe result acquisition request of the data query device 601 .

In one embodiment, the configuration center 603 is specifically configured to: use a dynamic listening mechanism to, after listening to the detection results of the storage cluster by the detection node cluster, cache the corresponding detection results to the local cache corresponding to the data query device 601 .

The content that has been introduced in the above embodiment will not be described again in this embodiment.

FIG. 7 shows an exemplary system architecture 700 to which the data query method, data writing method, data query device, data writing device or data query system in one or more embodiments of the present disclosure can be applied.

As shown in Figure 7, the system architecture 700 may include

terminal devices

701, 702, 703, a network 704 and a server 705. Network 704 is a medium used to provide communication links between

terminal devices

701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

Users can use

terminal devices

701, 702, 703 to interact with the server 705 through the network 704 to receive or send messages, etc. Various communication client applications can be installed on the

terminal devices

701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

The

terminal devices

701, 702, and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and so on.

The server 705 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using the

terminal devices

701, 702, and 703 (example only). The background management server can analyze and process the received product information query request and other data, and feed back the processing results (such as target push information, product information - only examples) to the terminal device.

It should be noted that, in one embodiment, the data query method provided by the embodiment of the present disclosure is generally executed by the server 705 or the

terminal devices

701, 702, and 703. Correspondingly, the data query device is generally provided on the server 705 or the terminal device 701. , 702, 703.

In another embodiment, the data writing method provided by the embodiment of the present disclosure is generally executed by the server 705 or the

terminal devices

701, 702, 703. Correspondingly, the data writing device is generally provided on the server 705 or the

terminal devices

701, 702. , 703.

It should be understood that the number of terminal devices, networks and servers in Figure 7 is only illustrative. Depending on implementation needs, there can be any number of end devices, networks, and servers.

Referring now to FIG. 8 , a schematic structural diagram of a computer system 800 suitable for implementing a terminal device or server in one or more embodiments of the present disclosure is shown. The terminal device or server shown in FIG. 8 is only an example and should not impose any restrictions on the functions and usage scope of the embodiments of the present disclosure.

As shown in FIG. 8, computer system 800 includes a central processing unit (CPU) 801 that can operate according to a program stored in a read-only memory (ROM) 802 or loaded from a storage portion 808 into a random access memory (RAM) 803. And perform various appropriate actions and processing. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. CPU 801, ROM 802 and RAM 803 are connected to each other via bus 804. An input/output (I/O) interface 805 is also connected to bus 804.

The following components are connected to the I/O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., speakers, etc.; and a storage section 808 including a hard disk, etc. ; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. Driver 810 is also connected to I/O interface 805 as needed. Removable media 811, such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage portion 808 as needed.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such embodiments, the computer program may be downloaded and installed from the network via communications portion 809 and/or installed from removable media 811 . When the computer program is executed by the central processing unit (CPU) 801, the above-described functions defined in the system of the present disclosure are performed.

It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read only memory (ROM), removable Programmed read-only memory (EPROM or flash memory), fiber optics, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device . Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the foregoing.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved. It will also be noted that each block in the block diagram or flowchart illustration, and combinations of blocks in the block diagram or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or may be implemented by special purpose hardware-based systems that perform the specified functions or operations. Achieved by a combination of specialized hardware and computer instructions.

The modules involved in the embodiments of the present disclosure can be implemented in software or hardware. The described module can also be provided in a processor. For example, it can be described as follows: a processor includes a detection result acquisition module, an available storage cluster determination module, and a data query module. Among them, the names of these modules do not constitute a limitation on the module itself under certain circumstances. For example, the detection result acquisition module can also be described as "used to obtain data about each storage cluster in response to the received data query request." Module for detecting live results".

As another aspect, the present disclosure also provides a computer-readable medium. The computer-readable medium may be included in the device described in the above embodiments; it may also exist separately without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by a device, the device includes: in response to the received data query request, obtain the detection results of each storage cluster, The detection result indicates whether the corresponding storage cluster is currently available; according to the detection result of each storage cluster, the list of available storage clusters is determined; according to the cluster selection settings, the target available storage cluster is selected from the list of available storage clusters, and the data query request is sent Go to the target available storage cluster to return the corresponding data query results from the target available storage cluster.

According to the technical solution of the embodiment of the present disclosure, in response to the received data query request, the detection results of each storage cluster are obtained, and the detection results indicate whether the corresponding storage cluster is currently available; according to the detection results of each storage cluster, it is determined List of available storage clusters; according to the cluster selection settings, select the target available storage cluster from the list of available storage clusters, send the data query request to the target available storage cluster, and return the corresponding data query results from the target available storage cluster. It can realize the consistency of writing data to multiple clusters, automatically switch clusters, improve the disaster recovery capability of cluster deployment, solve the problem of unavailability of the entire cluster in the event of a computer room level failure, and overcome the need for communication within the cluster. The disadvantage is that it takes a lot of time.

The above-mentioned specific embodiments do not constitute a limitation on the scope of the present disclosure. It will be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this disclosure shall be included in the protection scope of this disclosure.

Claims

A data query method, including:

In response to the received data query request, obtain the activity detection results for each storage cluster, and the activity detection results indicate whether the corresponding storage cluster is currently available;

Determine the list of available storage clusters based on the detection results of each storage cluster;

According to the cluster selection setting, a target available storage cluster is selected from the available storage cluster list, and the data query request is sent to the target available storage cluster, so that the corresponding data query result is returned by the target available storage cluster.
The method according to claim 1, wherein said obtaining the detection results of each storage cluster includes:

Obtain the detection results of each storage cluster by the detection node cluster from the configuration center or the local cache, wherein the configuration center uses a dynamic monitoring mechanism to detect the detection of the storage cluster by the detection node cluster. After the activation result is obtained, the corresponding detection result is cached in the local cache.
The method according to claim 2, wherein the active detecting node cluster includes one or more active detecting nodes, and the active detecting nodes are used to send health degree requests to each storage cluster respectively to determine the health status according to the storage cluster. The cluster's response to the health request determines whether the corresponding storage cluster is available.
The method according to claim 2, wherein determining the list of available storage clusters based on the detection results of each storage cluster includes:

For each storage cluster, determine whether the ratio of the number of active detection nodes available to the storage cluster to the total number of active detection nodes in the detection result of each active detection node for the storage cluster exceeds the proportion threshold, and if so, determine the The storage cluster is available and added to the list of available storage clusters.
The method according to claim 1, wherein according to the cluster selection setting, a target available storage cluster is selected from the available storage cluster list, and the data query request is sent to the target available storage cluster to obtain the information from the target available storage cluster. The above target can use the storage cluster to return the corresponding data query results, including:

All available storage clusters in the available storage cluster list are used as the target available storage clusters, and after receiving the data query result returned by one of the target available storage clusters, the data query result is output.
The method according to claim 1, wherein according to the cluster selection setting, a target available storage cluster is selected from the available storage cluster list, and the data query request is sent to the target available storage cluster to obtain the information from the target available storage cluster. The above target can use the storage cluster to return the corresponding data query results, including:

Select an available storage cluster that is currently polled from the list of available storage clusters according to the polling strategy as the target available storage cluster, and after receiving the data query results returned by the target available storage cluster, output the Describe the data query results.
The method according to claim 1, wherein according to the cluster selection setting, a target available storage cluster is selected from the available storage cluster list, and the data query request is sent to the target available storage cluster to obtain the information from the target available storage cluster. The above target can use the storage cluster to return the corresponding data query results, including:

An available storage cluster is randomly selected from the available storage cluster list as the target available storage cluster, and after receiving the data query result returned by the target available storage cluster, the data query result is output.
A data writing method including:

According to the number of storage clusters, initiate multiple monitoring of message middleware topics, each monitoring corresponds to a storage cluster, and the message middleware topic includes data to be written;

In response to each monitoring of the message middleware topic, the data to be written is written to the corresponding storage cluster.
The method according to claim 8, wherein in response to each monitoring of the message middleware topic, writing the data to be written to the corresponding storage cluster includes:

In response to each monitoring of the message middleware topic, the data to be written is written to the corresponding storage cluster through the preset writing interface. The parameters of the preset writing interface include the parameters of each storage cluster. Identification, wherein if the writing of the data to be written in any storage cluster fails, the operation of writing the data to be written to the storage cluster is retried.
A data query device, including:

The activity detection result acquisition module is used to obtain the activity detection results of each storage cluster in response to the received data query request, and the activity detection results indicate whether the corresponding storage cluster is currently available;

The available storage cluster determination module is used to determine the list of available storage clusters based on the detection results of each storage cluster;

A data query module, configured to select a target available storage cluster from the available storage cluster list according to the cluster selection setting, and send the data query request to the target available storage cluster, so that the target available storage cluster returns the corresponding data query results.
The device according to claim 10, wherein the detection result acquisition module is also used to:

Obtain the detection results of each storage cluster by the detection node cluster from the configuration center or the local cache, wherein the configuration center uses a dynamic monitoring mechanism to detect the detection of the storage cluster by the detection node cluster. After the activation result is obtained, the corresponding detection result is cached in the local cache.
The device according to claim 11, wherein the available storage cluster determination module is further configured to:

For each storage cluster, determine whether the ratio of the number of active detection nodes available to the storage cluster to the total number of active detection nodes in the detection result of each active detection node for the storage cluster exceeds the proportion threshold, and if so, determine the The storage cluster is available and added to the list of available storage clusters.
The device according to claim 10, wherein the data query module is also used for:

All available storage clusters in the available storage cluster list are used as the target available storage clusters, and after receiving the data query result returned by one of the target available storage clusters, the data query result is output.
The device according to claim 10, wherein the data query module is also used for:

Select an available storage cluster that is currently polled from the list of available storage clusters according to the polling strategy as the target available storage cluster, and after receiving the data query results returned by the target available storage cluster, output the Describe the data query results.
The device according to claim 10, wherein the data query module is also used for:

An available storage cluster is randomly selected from the available storage cluster list as the target available storage cluster, and after receiving the data query result returned by the target available storage cluster, the data query result is output.
A data writing device including:

A monitoring module, configured to initiate multiple monitoring of message middleware topics according to the number of storage clusters. Each monitoring corresponds to a storage cluster, and the message middleware topic includes data to be written;

A data writing module is configured to write the data to be written to the corresponding storage cluster in response to each monitoring of the message middleware topic.
The device according to claim 16, wherein the data writing module is also used for:

In response to each monitoring of the message middleware topic, the data to be written is written to the corresponding storage cluster through the preset writing interface. The parameters of the preset writing interface include the parameters of each storage cluster. Identification, wherein if the writing of the data to be written in any storage cluster fails, the operation of writing the data to be written to the storage cluster is retried.
A data query system, including: the data query device according to any one of claims 10 to 15, and further including each storage cluster, a configuration center and a detection node cluster, wherein:

The active detection node cluster includes one or more active detection nodes, and the active detection nodes are used to send health degree requests to each storage cluster respectively, so as to determine the response of the storage cluster to the health degree request. Determine the liveness detection result of the storage cluster, and the liveness detection result indicates whether the corresponding storage cluster is currently available;

The configuration center is configured to receive the activity detection results reported by each activity detection node, and provide the activity detection results to the data query device.
The system according to claim 18, wherein the configuration center is further configured to provide the data query device with the probe result in response to a probe result acquisition request from the data query device.
The system according to claim 18, wherein the configuration center is further configured to: through a dynamic monitoring mechanism, after monitoring the detection result of the storage cluster by the detection node cluster, the corresponding detection result The live results are cached in the local cache corresponding to the data query device.
An electronic device including:

one or more processors;

a storage device for storing one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any one of claims 1-9.
A computer-readable medium having a computer program stored thereon, which implements the method according to any one of claims 1-9 when executed by a processor.