[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

CN112799849B - Data processing method, device, equipment and storage medium - Google Patents

Data processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN112799849B
CN112799849B CN202110187719.0A CN202110187719A CN112799849B CN 112799849 B CN112799849 B CN 112799849B CN 202110187719 A CN202110187719 A CN 202110187719A CN 112799849 B CN112799849 B CN 112799849B
Authority
CN
China
Prior art keywords
configuration information
connection configuration
access server
access
client
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110187719.0A
Other languages
Chinese (zh)
Other versions
CN112799849A (en
Inventor
刘鑫伟
张世维
杨绍鹏
高绍涵
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN202110187719.0A priority Critical patent/CN112799849B/en
Publication of CN112799849A publication Critical patent/CN112799849A/en
Application granted granted Critical
Publication of CN112799849B publication Critical patent/CN112799849B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/505Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

A data processing method, apparatus, device and storage medium, the method includes: when detecting the adjustment operation of the access servers, acquiring initial connection configuration information, wherein the initial connection configuration information comprises a mapping relation between the access servers and the clients, and the number of the access servers allocated for each client is matched with the target number; when the adjustment operation is that an access server is newly added, determining the reference number of clients to be connected of each access server according to the initial connection configuration information and the current number of the access servers, and determining new connection configuration information according to the reference number and the initial connection configuration information; when the adjustment operation is to delete the access server, new connection configuration information is determined according to the information of deleting the access server, the initial connection configuration information, and the target number. By the method and the device, high availability of database services and load balancing of all access servers can be guaranteed.

Description

Data processing method, device, equipment and storage medium
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
With the rapid development of electronic commerce and information technology, a database management system is under tremendous pressure, and for better coordination of database management and work, more consideration is given to adopting a database cluster scheme in the industry, and the solution is to actually form a database cluster by using multiple servers to provide parallel transaction processing services.
In the actual use process of the database cluster, when a large number of users access, reasonable allocation of the requests is needed to be realized by utilizing a load balancing scheme, so that servers in the database cluster can timely respond to the requests of the users, and processing queuing or network congestion is avoided. At present, although load balancing schemes such as a load balancer can achieve the effect of load balancing, the number of access servers associated with a single client cannot be limited, if the connection from the client to the access servers is single connection, after the connection fails, the connection needs to be re-authenticated to be established, if the number of connections is too large, resource waste is caused, and meanwhile, under the scenes of capacity expansion and capacity shrinkage, the connection between a newly added access server and the client cannot be established, or the client continues to send a data operation request to a deleted access server, and cannot receive a response packet.
Disclosure of Invention
The embodiment of the invention provides a data processing method, a device, equipment and a storage medium, which can ensure high availability of database services and load balancing of various access servers.
In one aspect, an embodiment of the present invention provides a data processing method, applied to a data processing system, where the data processing system includes a directory server, at least two access servers, one or more clients, and a database, where the directory server is configured to allocate the access servers to the clients, and the clients perform data operations on the database through the allocated access servers, and the method includes:
when detecting the adjustment operation of the access servers, acquiring initial connection configuration information, wherein the initial connection configuration information comprises a mapping relation between the access servers and the clients, and the number of the access servers allocated for each client is matched with the target number;
when the adjustment operation is that an access server is newly added, determining the current number of the access servers, determining the reference number of clients to be connected of each access server according to the initial connection configuration information and the current number, and determining new connection configuration information according to the reference number and the initial connection configuration information;
And when the adjustment operation is to delete the access server, determining the information of deleting the access server, and determining new connection configuration information according to the information of deleting the access server, the initial connection configuration information and the target number.
In another aspect, an embodiment of the present invention provides a data processing apparatus, including:
the system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is used for acquiring initial connection configuration information when detecting the adjustment operation of the access servers, the initial connection configuration information comprises a mapping relation between the access servers and the clients, and the number of the access servers allocated for each client is matched with the target number;
the processing module is used for determining the current number of the access servers when the adjustment operation is that the access servers are newly added, determining the reference number of clients which each access server needs to be connected according to the initial connection configuration information and the current number, and determining new connection configuration information according to the reference number and the initial connection configuration information;
and the processing module is further configured to determine information of deleting the access server when the adjustment operation is performed to delete the access server, and determine new connection configuration information according to the information of deleting the access server, the initial connection configuration information and the target number.
Accordingly, an embodiment of the present invention provides a computer device, where the device includes a processor, a communication interface, and a memory, where the processor, the communication interface, and the memory are connected to each other, and the memory stores executable program code, and the processor is configured to invoke the executable program code to execute a data processing method according to any of the foregoing possible implementations.
Accordingly, an embodiment of the present invention provides a computer readable storage medium storing a computer program, where the processor executes a program related to a data processing method according to any one of the foregoing possible implementations.
Accordingly, embodiments of the present invention provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium and executes the computer instructions to cause the computer device to perform the data processing method of any of the possible implementations described above.
In the embodiment of the invention, when detecting that an access server is newly added, determining the reference number of clients to be connected of each access server according to the initial connection configuration information and the current number of the access servers, thereby determining new connection configuration information according to the reference number and the initial connection configuration information; when detecting that the access server is deleted, determining new connection configuration information according to the information of deleting the access server, the initial connection configuration information and the target number; the number of access servers allocated for each client indicated by the initial connection configuration information and the new connection configuration information matches the target number. By the method, the number of the access servers associated with the single client can be configured, meanwhile, when the capacity is expanded or contracted, the access servers associated with each client can be updated in time, fault connection is removed, and high availability of database service and load balancing of all the access servers are guaranteed.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data processing system according to an embodiment of the present application;
fig. 2 is a schematic flow chart of a data processing method according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a parameter configuration page according to an embodiment of the present application;
fig. 4 is a schematic diagram of a process flow related to a newly added access server according to an embodiment of the present application;
fig. 5 is a schematic diagram of a process flow related to deleting an access server according to an embodiment of the present application;
FIG. 6 is a flowchart of another data processing method according to an embodiment of the present disclosure;
FIG. 7 is a schematic diagram of queue management provided by an embodiment of the present application;
FIG. 8 is a flowchart of another data processing method according to an embodiment of the present disclosure;
FIG. 9 is a flowchart of another data processing method according to an embodiment of the present disclosure;
FIG. 10 is a flowchart of another data processing method according to an embodiment of the present disclosure;
FIG. 11 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;
fig. 12 is a schematic structural diagram of a computer device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
It should be noted that the descriptions of "first," "second," and the like in the embodiments of the present application are for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a technical feature defining "first", "second" may include at least one such feature, either explicitly or implicitly.
Cloud Technology (Cloud Technology): the method is a hosting technology for integrating hardware, software, network and other series resources in a wide area network or a local area network to realize calculation, storage, processing and sharing of data, namely a general term of network technology, information technology, integration technology, management platform technology, application technology and the like based on cloud computing business model application, can form a resource pool, and is flexible and convenient as required. Cloud computing technology will become an important support in the field of cloud technology. Background services of technical networking systems require a large amount of computing, storage resources, such as video websites, picture-like websites, and more portals. Along with the high development and application of the internet industry, each article possibly has an own identification mark in the future, the identification mark needs to be transmitted to a background system for logic processing, data of different levels can be processed separately, and various industry data all need strong system rear shield support and can be realized through cloud computing.
Cloud Storage (Cloud Storage): the distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system which integrates a large number of storage devices (storage devices are also called storage nodes) of different types in a network through application software or application interfaces to cooperatively work and provides data storage and service access functions together through functions of cluster application, grid technology, distributed storage file systems and the like.
Database (Database): in short, the system can be regarded as an electronic filing cabinet, namely a place for storing electronic files, and a user can perform operations such as adding, inquiring, updating, deleting and the like on data in the files. A "database" is a collection of data stored together in a manner that can be shared with multiple users, with as little redundancy as possible, independent of the application.
The data processing method provided by the embodiment of the application is mainly used for better realizing the load balance of the database cluster, and mainly relates to the technologies such as a cloud storage technology and a database technology in the cloud technology.
With reference to fig. 1, fig. 1 is a schematic architecture diagram of a data processing system according to an embodiment of the present application, where the data processing system includes one or more clients 101, a directory server 102, at least two access servers 103, and a database 104, and the one or more clients 101, the directory server 102, the at least two access servers 103, and the database 104 may be connected through a network, for example, a wired network or a wireless network. The directory server 102 is configured to allocate access servers 103 to the clients 101, where one or more directory servers 102 may be used, and a client may randomly select one directory server 102 to allocate access servers 103; the client 101 performs data operations (including read-write operations and the like) on the database 104 through the assigned access server 103, specifically, the access server 103 is an entry where the client 101 sends requests, all data operation requests are coordinated through the access server 103, and the access server 103 is responsible for forwarding the corresponding data operation requests to corresponding sharded nodes in the database 104 so as to perform data operations on the database 104.
The client 101 may be, but is not limited to, various personal computers, notebook computers, smartphones, tablet computers, in-vehicle devices, wearable devices, or other smart devices, etc.
The directory server 102 and the access server 103 may be independent physical servers, or may be a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.
The database 104 is a distributed database, such as MySql database, tcaplus database, etc., which divides data into a large number of pieces of data, dispersedly stores the pieces of data on a plurality of independent data storage nodes, and connects a plurality of physically dispersed data storage units by using a network to form a logically unified database. The database adopts an extensible system structure, and a plurality of servers are utilized to share the storage load.
In one embodiment, when the directory server 102 detects that the adjustment operation is to newly add the access server 103, the current number of the access servers 103 is determined, the reference number of the clients 101 to be connected to each access server 103 is determined according to the initial connection configuration information and the current number, so that new connection configuration information is determined according to the reference number and the initial connection configuration information, when the adjustment operation is detected to delete the access server 103, the information of deleting the access server 103 is determined, the new connection configuration information is determined according to the information of deleting the access server 103, the initial connection configuration information and the target number, and meanwhile, the obtained initial connection configuration information includes a mapping relationship between the access server 103 and the clients 101, and the number of the access servers 103 allocated to each client 101 is matched with the target number. By the method, the number of the access servers 103 associated with each client 101 can be configured, and when the capacity is expanded and contracted, the access servers 103 associated with each client 101 can be updated in time, fault connection is eliminated, and high availability of database service and load balancing of each access server 103 are ensured.
It may be understood that, the schematic architecture diagram of the system described in one or more embodiments of the present application is for more clearly describing the technical solutions of the embodiments of the present application, and is not limited to the technical solutions provided in the embodiments of the present application, and those skilled in the art may know that, with the evolution of the system architecture and the appearance of new service scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.
In one embodiment, as shown in FIG. 2, a data processing method is provided by an embodiment of the present invention based on the data processing system of FIG. 1. The directory server 102 in fig. 1 is exemplified by this method, which includes the following steps:
s201, when detecting the adjustment operation of the access servers, obtaining initial connection configuration information, wherein the initial connection configuration information comprises a mapping relation between the access servers and the clients, and the number of the access servers allocated for each client is matched with the target number.
In one embodiment, the adjustment operation of the access server may be performed by the management user, or may be performed according to a preset policy, which is not limited in this application. When the directory server detects the adjustment operation performed on the access server, the directory server acquires initial connection configuration information from the database, where the initial connection configuration information includes a mapping relationship between the access server and the client, and in a possible embodiment, the initial connection configuration information is saved by using a structured statement, for example: "Multimap < association number, proxyID > g_proxy_num_Multimap" holds the number of clients associated with each access server (proxyID); "map < proxyID, api set > g_proxy_conn_with_api_map" holds the client (api set) associated with each access server; "map < apiID, proxy set > g_api_conn_with_proteins_map" holds the access server (proxy set) associated with each client (apiID).
In one embodiment, the number of access servers allocated to each client in the initial connection configuration information is matched with the target number, and the management user can configure the target number of access servers associated with each client on the parameter configuration page shown in fig. 3, so as to ensure that the client can establish a mapping relationship with a plurality of access servers. Wherein, api is client, proxy is access server, dir is directory server. The page parameters can be modified in real time through parameter configuration, the client does not need to be restarted, and the user does not feel.
S202, when the adjustment operation is that the access server is newly added, determining the reference number of the clients to be connected of each access server according to the initial connection configuration information and the current number of the access servers, and determining new connection configuration information according to the reference number and the initial connection configuration information.
In one embodiment, when an access server is newly added, the current number of access servers is determined, and the reference number of clients to be connected to each access server is determined according to the initial connection configuration information and the current number, for example, 200 access servers are included in the initial connection configuration information, 1000 clients are included in the initial connection configuration information, 100 access servers are required to be associated with each client, each access server is required to be associated with (1000×100)/200=500 clients, after the access server is newly added, assuming that 30 access servers are newly added, the current number is determined to be 230, so that the reference number of the clients required to be associated with each access server is (1000×100)/230=435 (rounded up), and transfer client information of each access server is determined according to the clients and the reference number associated with each access server in the initial connection configuration information, so as to generate new connection configuration information.
S203, when the adjustment operation is to delete the access server, determining the information of deleting the access server, and determining new connection configuration information according to the information of deleting the access server, the initial connection configuration information and the target number.
In one embodiment, when the access server is deleted, determining information of the access server to be deleted, determining a first client from the initial connection configuration information according to the information of the access server to be deleted, wherein the first client is a client having a mapping relationship with the access server to be deleted, and reassigning the access server having the mapping relationship to the first client to generate new connection configuration information. The execution order of S202 and S203 is not limited in this application.
In the embodiment of the application, when detecting that an access server is newly added, determining the reference number of clients to be connected of each access server according to the initial connection configuration information and the current number of the access servers, so as to determine new connection configuration information according to the reference number and the initial connection configuration information; when detecting that the access server is deleted, determining new connection configuration information according to the information of deleting the access server, the initial connection configuration information and the target number; the number of access servers allocated for each client indicated by the initial connection configuration information and the new connection configuration information matches the target number. By the method, the number of the access servers associated with the single client can be configured, meanwhile, when the capacity is expanded or contracted, the access servers associated with each client can be updated in time, fault connection is removed, and high availability of database service and load balancing of all the access servers are guaranteed. The indirect effect ensures the rate and success rate of the data operation of the database by the client through the access server configured for the client.
In one embodiment, said determining new connection configuration information from said reference number and said initial connection configuration information comprises: determining transfer client information corresponding to each access server in the initial connection configuration information according to the reference number and the initial connection configuration information; the initial connection configuration information is adjusted according to the transfer client information corresponding to each access server, and adjusted connection configuration information is obtained; determining a mapping relation between each newly added access server and the client according to the transfer client information corresponding to each access server and the reference quantity; and determining new connection configuration information according to the mapping relation between each newly added access server and the client and the adjusted connection configuration information.
In one embodiment, transfer client information corresponding to each access server in the initial connection configuration information is determined according to the reference number and the initial connection configuration information, the transfer client information includes the number of transferred clients and the transferred client information, then the initial connection configuration information is adjusted according to the transfer client information corresponding to each access server, if newly added access server information is inserted into connection configuration information stored by using g_proxy_num_multimap, adjusted connection configuration information is obtained, then a mapping relation between each newly added access server and the client is determined according to the transfer client information corresponding to each access server and the reference number, and finally new connection configuration information is determined according to the mapping relation between each newly added access server and the client and the adjusted connection configuration information.
Specifically, as shown in fig. 4, for example, there are 200 access servers in the initial connection configuration information, there are 1000 clients, each client needs to be associated with 100 access servers, each access server needs to be associated with (1000×100)/200=500 clients, after the access servers are newly added, assuming that 30 access servers are newly added, the current number is determined to be 230, so that the reference number of the clients needed to be associated with each access server is (1000×100)/230=435 (rounded up), then the total requirement of each access server in the initial configuration information is changed to 13050 clients as 30 clients associated with the newly added access servers, so that each access server in the initial configuration information needs to be approximately transferred with 66 clients (rounded up), the first 197 access servers can be transferred with 197×66=13002 in total, the 198 th access server can be transferred with 48, and the latter two access servers are not transferred at all, so that after the access servers are newly added, the number of the clients associated with each access server is: the number of 2 access server associated clients is 500, the number of 1 access server associated clients is 452, the number of 30 access server associated clients is 435, and the number of 197 access server associated clients is 434. When transferring, if the transferred client is already in the client associated with the access client, the next access server is selected for transferring. And determining new connection configuration information according to the mapping relation between the newly added access server and the client after transfer and the mapping relation between each access server and the client in the connection configuration information after transfer.
In one embodiment, the determining new connection configuration information according to the information of the deletion access server, the initial connection configuration information, and the target number includes: determining a first client according to the information of the deletion access server and the initial connection configuration information, wherein the first client is any client with a mapping relation with the deletion access server; adjusting the initial connection configuration information according to the information of deleting the access server to obtain adjusted connection configuration information; determining a first access server according to the adjusted connection configuration information and the target number; and establishing a mapping relation between the first access server and the first client, and determining new connection configuration information according to the mapping relation between the first access server and the first client and the adjusted connection configuration information.
In one embodiment, a first client is determined according to the information of the deletion access server and the initial connection configuration information, the first client is any client having a mapping relationship with the deletion access server, and the mapping relationship between the deletion access server and the client is deleted from the initial connection configuration information, so as to obtain the adjusted connection configuration information.
Further, since the access server associated with the first client is deleted, the first access server is determined by the number of clients and the target number associated with each access server in the adjusted connection configuration, and after the first access server is determined, a mapping relationship between the first access server and the first client is established, so that new connection configuration information is determined according to the mapping relationship between the first access server and the first client and the adjusted connection configuration information.
In one embodiment, the determining the first access server according to the adjusted connection configuration information and the target number includes: determining the current number of access servers allocated to the first client according to the adjusted connection configuration information; determining one or more first access servers from the at least two access servers according to the target number, the current number and the adjusted connection configuration information; the first access server is an access server which orders the access servers from small to large according to the number of the corresponding clients and then arranges the access servers in the front M bits, and M is determined according to the target number and the current number.
In one embodiment, the current number of access servers allocated to the first client is determined according to the adjusted connection configuration information, the number M of access servers to which the first client needs to be associated can be determined according to the target number and the current number of access servers allocated to the first client, the number of associated clients in the adjusted connection configuration is ordered from small to large or from large to small, and after the ordering, the access servers in the M-bit row or the access servers in the M-bit row are selected as the first access server.
In one embodiment, as shown in fig. 5, after deleting the access servers, each client is traversed, the number of access servers associated with each client is determined, when the number of access servers associated with any client is detected to be not matched with the target number, a first access server is determined, and a mapping relationship with the first access server is established.
In one embodiment, the method further comprises: determining, for a second client, a current number of access servers allocated to the second client according to initial connection configuration information; detecting whether the current quantity is matched with the target quantity; when the current number is not matched with the target number, determining a second access server according to the initial connection configuration information; establishing a mapping relation between the second access server and the second client so that the number of the access servers distributed for the second client is matched with the target number; and determining new connection configuration information according to the mapping relation between the second access server and the second client and the initial connection configuration information.
In one embodiment, the second client may be any client in the initial connection configuration information, or may be a new client that is not associated with the access server, which is not limited in this application. Determining the current number of access servers allocated to the second client according to the initial connection configuration information, when the second client is a new client, the current number is 0, when the fact that the current number is not matched with the target number is detected, selecting the access server with the minimum number of associated clients in the initial connection configuration information as the second access server, establishing a mapping relation between the second access server and the second client, repeatedly executing the steps until the number of the access servers associated with the second client is equal to the target number, and finally determining the new connection configuration information according to the servers associated with the second client. In each round of determining the second access server, the number of clients associated with the second access server in the initial connection configuration information needs to be updated, and connection configuration information stored by using g_proxy_conn_with_api_map and g_api_conn_with_Proxies_map is updated at the same time.
In one embodiment, the method further comprises: determining whether a load balancing condition is met according to the initial connection configuration information; if the load balancing condition is not met, determining a third access server and a fourth access server from the at least two access servers according to the initial connection configuration information; the mapping relation between the third access server and the fourth access server and the clients is adjusted so that the number of the clients associated with the third access server and the fourth access server is matched; the third access server is the access server with the largest number of the clients associated with the at least two access servers, and the fourth access server is the access server with the smallest number of the clients associated with the at least two access servers.
In one embodiment, when a client sends a configuration information acquisition request to a directory server, the directory server detects whether the number of access servers associated with the client is matched with a target number, and when the number of access servers is matched with the target number, the target server judges whether initial connection configuration information meets a load balancing condition; or the directory server may determine at regular time whether the initial connection configuration information satisfies the load balancing condition. Whether the load balancing condition is satisfied may be determined by various manners, such as calculating whether a variance of the number of clients associated with each access server in the initial connection configuration information satisfies a preset threshold, or determining whether a percentage between an access server with a minimum number of associated clients and an access server with a maximum number of associated clients satisfies a preset threshold, for example, the access server with the maximum number of associated clients does not exceed 5% of the access server with the minimum number of associated clients.
Further, when the initial connection configuration information meets the load balancing condition, the connection configuration information is not changed, and when the load balancing condition is not met, a third access server (the access server with the smallest number of associated clients) and a fourth access server (the access server with the largest number of associated clients) are determined from the initial connection configuration information, and the mapping relation between the third access server and the fourth access server and the clients is subjected to equipartition adjustment so that the number of the clients associated with the third access server and the fourth access server is matched.
According to the embodiment of the application, even if the load imbalance is caused in a certain link, the load imbalance can be balanced again through one or more rounds of operation. Moreover, the performance overhead is relatively small because the work of adjusting the load balance is dispersed to a plurality of rounds of operation.
In one embodiment, as shown in fig. 6, when a client acquires connection configuration information, a directory server responds to a configuration information acquisition request sent by the client to determine whether the client is a new client, and determines the number of clients associated with each access server in initial connection configuration information for the new client, then selects the access server with the least number of associated clients as a second access server, establishes a mapping relationship between the second access server and the new client, and obtains new connection configuration information according to the mapping relationship between the second access server and the new client. When the client is not a new client, judging whether the initial connection configuration information meets the load balancing condition, if not, determining a third access server and a fourth access server from the initial connection configuration information, equally dividing and adjusting the mapping relation between the third access server and the fourth access server and the client so that the number of the clients associated with the third access server and the fourth access server are matched, obtaining new connection configuration information according to the adjusted third access server and fourth access server, and returning the new connection configuration information to the client.
In one embodiment, the method further comprises: receiving a configuration information acquisition request sent by a client; inquiring the matching connection configuration information of the client in response to the configuration information acquisition request; and sending the matched connection configuration information to the client so that the client performs data operation on the database through the access server indicated by the matched connection configuration information.
In one embodiment, when the directory server receives the configuration information acquisition request, the directory server queries the matching connection configuration information of the client from the database in response to the configuration information acquisition request, and sends the matching connection configuration information to the client, and the client can establish a mapping relationship with the access server indicated by the matching connection configuration information, and perform data operation on the database by using the access server for which the mapping relationship is established. When the matching connection configuration information acquired by the client is changed compared with the matching connection configuration information in use, the access server is either newly added or deleted, the mapping relation between the client and the newly added access server needs to be timely created for the newly added access server, and the request is not sent to the deleted access server any more because the client needs to actively reject the deleted access server.
In one embodiment, as shown in fig. 7, the client may use three queues to manage the connection to the access server, e.g., use the queues to save the matching connection configuration information the client is using, prepare the queues to save the matching connection configuration information this time acquired from the directory server, delete the queues to save and delete the access server.
In one embodiment, the client may send configuration information acquisition requests to the directory server at regular intervals during operation to acquire matching connection configuration information. Or when the connection configuration information is updated, the directory server may send new connection configuration information corresponding to each client to the corresponding client. Or when the connection configuration information of a certain client is adjusted, actively sending the corresponding matched connection configuration information to the corresponding client.
In one embodiment, as shown in fig. 8, after the client acquires the configuration information of the matching connection, the configuration information of the matching connection is put into a preparation queue, the preparation queue is compared with a use queue, whether a new access server exists or not is judged, for the new access server, a mapping relation between the new access server and the client is established, further, secondary comparison is performed, a deletion access server is put into the deletion queue, after the response packet on the deletion access server is received, the mapping relation between the deletion access server and the client is disconnected, and when the mapping relation between all access servers on the preparation queue and the client is established, the preparation queue is replaced with the use queue, and a user is unaware in the whole process.
In one embodiment, in the capacity expansion scenario, the directory server can adjust the connection between each client and the access server in time, as shown in fig. 9, a new access server 6 is allocated to the client 2, after the client 2 obtains the matching connection configuration information from the directory server, the connection configuration information of the connection is changed from the access server (4, 5, 1) to the access server (4, 5, 6), and the access server 6 is allocated to the new connection, so that the load of the access server 1 is reduced, the number of clients managed by each access server is balanced, and the process is not perceived by the user.
In one embodiment, as shown in fig. 10, after the management user configures the target number of each client to the access server by using the parameter configuration page, the configuration parameters are written into the database, and the directory server periodically pulls the configuration parameters from the database and updates the connection configuration information in the memory. The configuration parameters and the connection configuration information are stored in the database, so that the configuration can still be effective after the directory server is restarted or migrated.
As shown in fig. 11, fig. 11 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application, where the apparatus includes:
An obtaining module 1101, configured to obtain initial connection configuration information when an adjustment operation of an access server is detected, where the initial connection configuration information includes a mapping relationship between the access server and a client, and the number of access servers allocated to each client is matched with a target number;
a processing module 1102, configured to determine, when the adjustment operation is to add an access server newly, a current number of the access servers, determine, according to the initial connection configuration information and the current number, a reference number of clients that each access server needs to connect to, and determine, according to the reference number and the initial connection configuration information, new connection configuration information;
the processing module 1102 is further configured to determine, when the adjustment operation is to delete the access server, information of deleting the access server, and determine new connection configuration information according to the information of deleting the access server, the initial connection configuration information, and the target number.
In one embodiment, the processing module 1102 is specifically configured to:
determining transfer client information corresponding to each access server in the initial connection configuration information according to the reference number and the initial connection configuration information;
The initial connection configuration information is adjusted according to the transfer client information corresponding to each access server, and adjusted connection configuration information is obtained;
determining a mapping relation between each newly added access server and the client according to the transfer client information corresponding to each access server and the reference quantity;
and determining new connection configuration information according to the mapping relation between each newly added access server and the client and the adjusted connection configuration information.
In one embodiment, the processing module 1102 is specifically configured to:
determining a first client according to the information of the deletion access server and the initial connection configuration information, wherein the first client is any client with a mapping relation with the deletion access server;
adjusting the initial connection configuration information according to the information of deleting the access server to obtain adjusted connection configuration information;
determining a first access server according to the adjusted connection configuration information and the target number;
and establishing a mapping relation between the first access server and the first client, and determining new connection configuration information according to the mapping relation between the first access server and the first client and the adjusted connection configuration information.
In one embodiment, the processing module 1102 is specifically configured to:
determining the current number of access servers allocated to the first client according to the adjusted connection configuration information;
determining one or more first access servers from the at least two access servers according to the target number, the current number and the adjusted connection configuration information;
the first access server is an access server which orders the access servers from small to large according to the number of the corresponding clients and then arranges the access servers in the front M bits, and M is determined according to the target number and the current number.
In one embodiment, the processing module 1102 is specifically configured to:
determining, for a second client, a current number of access servers allocated to the second client according to initial connection configuration information;
detecting whether the current quantity is matched with the target quantity;
when the current number is not matched with the target number, determining a second access server according to the initial connection configuration information;
establishing a mapping relation between the second access server and the second client so that the number of the access servers distributed for the second client is matched with the target number;
And determining new connection configuration information according to the mapping relation between the second access server and the second client and the initial connection configuration information.
In one embodiment, the processing module 1102 is specifically configured to:
determining whether a load balancing condition is met according to the initial connection configuration information;
if the load balancing condition is not met, determining a third access server and a fourth access server from the at least two access servers according to the initial connection configuration information;
the mapping relation between the third access server and the fourth access server and the clients is adjusted so that the number of the clients associated with the third access server and the fourth access server is matched;
the third access server is the access server with the largest number of the clients associated with the at least two access servers, and the fourth access server is the access server with the smallest number of the clients associated with the at least two access servers.
In one embodiment, the processing module 1102 is specifically configured to:
receiving a configuration information acquisition request sent by a client;
Inquiring the matching connection configuration information of the client in response to the configuration information acquisition request;
and sending the matched connection configuration information to the client so that the client performs data operation on the database through the access server indicated by the matched connection configuration information.
In the embodiment of the application, when detecting that an access server is newly added, determining the reference number of clients to be connected of each access server according to the initial connection configuration information and the current number of the access servers, so as to determine new connection configuration information according to the reference number and the initial connection configuration information; when detecting that the access server is deleted, determining new connection configuration information according to the information of deleting the access server, the initial connection configuration information and the target number; the number of access servers allocated for each client indicated by the initial connection configuration information and the new connection configuration information matches the target number. By the method, the number of the access servers associated with the single client can be configured, meanwhile, when the capacity is expanded or contracted, the access servers associated with each client can be updated in time, fault connection is removed, and high availability of database service and load balancing of all the access servers are guaranteed. The indirect effect ensures the rate and success rate of the data operation of the database by the client through the access server configured for the client.
Fig. 12 is a schematic structural diagram of a computer device according to an embodiment of the present application, where the internal structure of the device is as shown in fig. 12, and includes: one or more processors 1201, memory 1202, and a communication interface 1203. The processor 1201, the memory 1202, and the communication interface 1203 described above may be connected via the bus 1204 or otherwise, and the embodiments of the present application are exemplified as being connected via the bus 1204.
Among them, the processor 1201 (or CPU (Central Processing Unit, central processing unit)) is a computing core and a control core of a computer device, which can parse various instructions in the computer device and process various data of the computer device, for example: the CPU can be used for analyzing a startup and shutdown instruction sent by a user to the computer equipment and controlling the computer equipment to perform startup and shutdown operation; and the following steps: the CPU may transmit various types of interaction data between internal structures of the computer device, and so on. The communication interface 1203 may optionally include a standard wired interface, a wireless interface (e.g., wi-Fi, mobile communication interface, etc.), controlled by the processor 1201 for transceiving data. Memory 1202 (Memory) is a Memory device in a computer device for storing programs and data. It is to be appreciated that the memory 1202 herein may include both built-in memory of the computer device and extended memory supported by the computer device. Memory 1202 provides storage space that stores an operating system of a computer device, which may include, but is not limited to: windows system, linux system, etc., which is not limited in this application.
In one embodiment, the processor 1201 is specifically configured to:
when detecting the adjustment operation of the access servers, acquiring initial connection configuration information, wherein the initial connection configuration information comprises a mapping relation between the access servers and the clients, and the number of the access servers allocated for each client is matched with the target number;
when the adjustment operation is that an access server is newly added, determining the current number of the access servers, determining the reference number of clients to be connected of each access server according to the initial connection configuration information and the current number, and determining new connection configuration information according to the reference number and the initial connection configuration information;
and when the adjustment operation is to delete the access server, determining the information of deleting the access server, and determining new connection configuration information according to the information of deleting the access server, the initial connection configuration information and the target number.
In one embodiment, the processor 1201 is specifically configured to:
determining transfer client information corresponding to each access server in the initial connection configuration information according to the reference number and the initial connection configuration information;
The initial connection configuration information is adjusted according to the transfer client information corresponding to each access server, and adjusted connection configuration information is obtained;
determining a mapping relation between each newly added access server and the client according to the transfer client information corresponding to each access server and the reference quantity;
and determining new connection configuration information according to the mapping relation between each newly added access server and the client and the adjusted connection configuration information.
In one embodiment, the processor 1201 is specifically configured to:
determining a first client according to the information of the deletion access server and the initial connection configuration information, wherein the first client is any client with a mapping relation with the deletion access server;
adjusting the initial connection configuration information according to the information of deleting the access server to obtain adjusted connection configuration information;
determining a first access server according to the adjusted connection configuration information and the target number;
and establishing a mapping relation between the first access server and the first client, and determining new connection configuration information according to the mapping relation between the first access server and the first client and the adjusted connection configuration information.
In one embodiment, the processor 1201 is specifically configured to:
determining the current number of access servers allocated to the first client according to the adjusted connection configuration information;
determining one or more first access servers from the at least two access servers according to the target number, the current number and the adjusted connection configuration information;
the first access server is an access server which orders the access servers from small to large according to the number of the corresponding clients and then arranges the access servers in the front M bits, and M is determined according to the target number and the current number.
In one embodiment, the processor 1201 is specifically configured to:
determining, for a second client, a current number of access servers allocated to the second client according to initial connection configuration information;
detecting whether the current quantity is matched with the target quantity;
when the current number is not matched with the target number, determining a second access server according to the initial connection configuration information;
establishing a mapping relation between the second access server and the second client so that the number of the access servers distributed for the second client is matched with the target number;
And determining new connection configuration information according to the mapping relation between the second access server and the second client and the initial connection configuration information.
In one embodiment, the processor 1201 is specifically configured to:
determining whether a load balancing condition is met according to the initial connection configuration information;
if the load balancing condition is not met, determining a third access server and a fourth access server from the at least two access servers according to the initial connection configuration information;
the mapping relation between the third access server and the fourth access server and the clients is adjusted so that the number of the clients associated with the third access server and the fourth access server is matched;
the third access server is the access server with the largest number of the clients associated with the at least two access servers, and the fourth access server is the access server with the smallest number of the clients associated with the at least two access servers.
In one embodiment, the processor 1201 is specifically configured to:
receiving a configuration information acquisition request sent by a client;
Inquiring the matching connection configuration information of the client in response to the configuration information acquisition request;
and sending the matched connection configuration information to the client so that the client performs data operation on the database through the access server indicated by the matched connection configuration information.
In the embodiment of the application, when detecting that an access server is newly added, determining the reference number of clients to be connected of each access server according to the initial connection configuration information and the current number of the access servers, so as to determine new connection configuration information according to the reference number and the initial connection configuration information; when detecting that the access server is deleted, determining new connection configuration information according to the information of deleting the access server, the initial connection configuration information and the target number; the number of access servers allocated for each client indicated by the initial connection configuration information and the new connection configuration information matches the target number. By the method, the number of the access servers associated with the single client can be configured, meanwhile, when the capacity is expanded or contracted, the access servers associated with each client can be updated in time, fault connection is removed, and high availability of database service and load balancing of all the access servers are guaranteed. The indirect effect ensures the rate and success rate of the data operation of the database by the client through the access server configured for the client.
Those skilled in the art will appreciate that implementing all or part of the above-described embodiment methods may be accomplished by computer programs stored on a computer readable storage medium, which when executed, may include embodiments of the above-described data processing methods. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random access Memory (RandomAccess Memory, RAM), or the like.
One or more embodiments of the present application also provide a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps performed in the embodiments of the methods described above.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the claims. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application is to be determined by the claims appended hereto.

Claims (9)

1. A data processing method applied to a data processing system, wherein the data processing system comprises a directory server, at least two access servers, one or more clients and a database, the directory server is used for distributing the access servers for the clients, and the clients perform data operation on the database through the distributed access servers, and the method comprises the following steps:
when detecting the adjustment operation of the access servers, acquiring initial connection configuration information, wherein the initial connection configuration information comprises a mapping relation between the access servers and the clients, and the number of the access servers allocated for each client is matched with the target number;
when the adjustment operation is that an access server is newly added, determining the current number of the access servers, determining the reference number of clients to be connected of each access server according to the initial connection configuration information and the current number, and determining new connection configuration information according to the reference number and the initial connection configuration information;
when the adjustment operation is that the access server is deleted, determining information of deleting the access server, and determining new connection configuration information according to the information of deleting the access server, the initial connection configuration information and the target number;
Wherein the method further comprises: and determining whether a load balancing condition is met or not according to initial connection configuration information, if the load balancing condition is not met, determining a third access server and a fourth access server from the at least two access servers according to the initial connection configuration information, and adjusting the mapping relation between the third access server and the fourth access server and the clients so that the number of the clients associated with the third access server and the fourth access server is matched, wherein the third access server is the access server with the largest number of the clients associated with the at least two access servers, and the fourth access server is the access server with the smallest number of the clients associated with the at least two access servers.
2. The method of claim 1, wherein said determining new connection configuration information based on said reference number and said initial connection configuration information comprises:
determining transfer client information corresponding to each access server in the initial connection configuration information according to the reference number and the initial connection configuration information;
The initial connection configuration information is adjusted according to the transfer client information corresponding to each access server, and adjusted connection configuration information is obtained;
determining a mapping relation between each newly added access server and the client according to the transfer client information corresponding to each access server and the reference quantity;
and determining new connection configuration information according to the mapping relation between each newly added access server and the client and the adjusted connection configuration information.
3. The method of claim 1, wherein said determining new connection configuration information based on said deletion of access server information, said initial connection configuration information, and said target number comprises:
determining a first client according to the information of the deletion access server and the initial connection configuration information, wherein the first client is any client with a mapping relation with the deletion access server;
adjusting the initial connection configuration information according to the information of deleting the access server to obtain adjusted connection configuration information;
determining a first access server according to the adjusted connection configuration information and the target number;
And establishing a mapping relation between the first access server and the first client, and determining new connection configuration information according to the mapping relation between the first access server and the first client and the adjusted connection configuration information.
4. A method according to claim 3, wherein said determining a first access server based on said adjusted connection configuration information and said target number comprises:
determining the current number of access servers allocated to the first client according to the adjusted connection configuration information;
determining one or more first access servers from the at least two access servers according to the target number, the current number and the adjusted connection configuration information;
the first access server is an access server which orders the access servers from small to large according to the number of the corresponding clients and then arranges the access servers in the front M bits, and M is determined according to the target number and the current number.
5. The method according to any one of claims 1-4, further comprising:
determining, for a second client, a current number of access servers allocated to the second client according to initial connection configuration information;
Detecting whether the current quantity is matched with the target quantity;
when the current number is not matched with the target number, determining a second access server according to the initial connection configuration information;
establishing a mapping relation between the second access server and the second client so that the number of the access servers distributed for the second client is matched with the target number;
and determining new connection configuration information according to the mapping relation between the second access server and the second client and the initial connection configuration information.
6. The method according to claim 1, wherein the method further comprises:
receiving a configuration information acquisition request sent by a client;
inquiring the matching connection configuration information of the client in response to the configuration information acquisition request;
and sending the matched connection configuration information to the client so that the client performs data operation on the database through the access server indicated by the matched connection configuration information.
7. A data processing apparatus, the apparatus comprising:
the system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is used for acquiring initial connection configuration information when detecting the adjustment operation of the access servers, the initial connection configuration information comprises a mapping relation between the access servers and the clients, and the number of the access servers allocated for each client is matched with the target number;
The processing module is used for determining the current number of the access servers when the adjustment operation is that the access servers are newly added, determining the reference number of clients which each access server needs to be connected according to the initial connection configuration information and the current number, and determining new connection configuration information according to the reference number and the initial connection configuration information;
the processing module is further configured to determine, when the adjustment operation is to delete the access server, information of deleting the access server, and determine new connection configuration information according to the information of deleting the access server, the initial connection configuration information, and the target number;
the processing module is further configured to determine whether a load balancing condition is met according to initial connection configuration information, if the load balancing condition is not met, determine a third access server and a fourth access server from at least two access servers according to the initial connection configuration information, and adjust mapping relationships between the third access server and the fourth access server and clients so that the number of clients associated with the third access server and the fourth access server is matched, where the third access server is an access server with the largest number of clients associated with the at least two access servers, and the fourth access server is an access server with the smallest number of clients associated with the at least two access servers.
8. A computer device comprising a memory, a communication interface and a processor, wherein the memory, the communication interface and the processor are interconnected, the memory storing computer program code, the processor invoking the computer program code stored in the memory for performing the data processing method of any of claims 1-6.
9. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the data processing method according to any one of claims 1 to 6.
CN202110187719.0A 2021-02-18 2021-02-18 Data processing method, device, equipment and storage medium Active CN112799849B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110187719.0A CN112799849B (en) 2021-02-18 2021-02-18 Data processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110187719.0A CN112799849B (en) 2021-02-18 2021-02-18 Data processing method, device, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN112799849A CN112799849A (en) 2021-05-14
CN112799849B true CN112799849B (en) 2024-03-19

Family

ID=75815180

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110187719.0A Active CN112799849B (en) 2021-02-18 2021-02-18 Data processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112799849B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117407042B (en) * 2023-11-01 2024-04-05 广州国测规划信息技术有限公司 Extension design method and system for homeland space basic information platform

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005092862A (en) * 2003-08-11 2005-04-07 Hitachi Ltd Load distribution method and client-server system
WO2018153218A1 (en) * 2017-02-27 2018-08-30 腾讯科技(深圳)有限公司 Resource processing method, related apparatus and communication system
CN109743392A (en) * 2019-01-07 2019-05-10 北京字节跳动网络技术有限公司 A kind of load-balancing method, device, electronic equipment and storage medium
CN110289999A (en) * 2019-06-21 2019-09-27 深圳前海微众银行股份有限公司 A kind of data processing method, system and device

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5782641B2 (en) * 2012-08-31 2015-09-24 株式会社日立製作所 Computer system and packet transfer method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005092862A (en) * 2003-08-11 2005-04-07 Hitachi Ltd Load distribution method and client-server system
WO2018153218A1 (en) * 2017-02-27 2018-08-30 腾讯科技(深圳)有限公司 Resource processing method, related apparatus and communication system
CN109743392A (en) * 2019-01-07 2019-05-10 北京字节跳动网络技术有限公司 A kind of load-balancing method, device, electronic equipment and storage medium
CN110289999A (en) * 2019-06-21 2019-09-27 深圳前海微众银行股份有限公司 A kind of data processing method, system and device

Also Published As

Publication number Publication date
CN112799849A (en) 2021-05-14

Similar Documents

Publication Publication Date Title
US9052962B2 (en) Distributed storage of data in a cloud storage system
EP3667500A1 (en) Using a container orchestration service for dynamic routing
US20100138540A1 (en) Method of managing organization of a computer system, computer system, and program for managing organization
CN102244685A (en) Distributed type dynamic cache expanding method and system supporting load balancing
CN111352716B (en) Task request method, device and system based on big data and storage medium
CN109144972B (en) Data migration method and data node
EP4068725B1 (en) Topology-based load balancing for task allocation
CN106991008B (en) Resource lock management method, related equipment and system
CN111258627A (en) Interface document generation method and device
US12007848B2 (en) Automatic selection of network path connecting backup clients to a pool of multiple backup storage appliances
CN113900774A (en) Virtual machine control method and device of cloud operating system and storage medium
CN106547790B (en) Relational database service system
CN113377866A (en) Load balancing method and device for virtualized database proxy service
CN104125294A (en) Method and system for big data safety management
CN112468589A (en) Data distribution method and device, computer equipment and storage medium
CN112799849B (en) Data processing method, device, equipment and storage medium
CN114760304B (en) Processing method, processing system and computing gateway of computing information
KR20130130295A (en) System and method for assigining server to terminal and efficiently delivering messages to the terminal
CN113177179B (en) Data request connection management method, device, equipment and storage medium
CN112910796B (en) Traffic management method, apparatus, device, storage medium, and program product
CN114490100B (en) Message queue telemetry transmission load balancing method, device and server
CN110958326B (en) Load balancing method, device, system, equipment and medium
CN116028196A (en) Data processing method, device and storage medium
CN109005071B (en) Decision deployment method and scheduling equipment
KR101997602B1 (en) Resource Dependency Service Method for M2M Resource Management

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 40043475

Country of ref document: HK

GR01 Patent grant
GR01 Patent grant