Disclosure of Invention
The present invention is directed to solving at least one of the technical problems existing in the related art. To this end, the present invention provides an originating node output method in an MPP distributed system.
The invention provides an originating node output method in an MPP distributed system, which comprises the following steps:
s1: collecting the number of nodes and the node IP in the database cluster, and sequencing according to the fragmentation information to obtain a first mapping relation of the node fragmentation and the node IP;
s2: collecting server information, creating an IP distance matrix of a computing node and a management node according to the server information, and obtaining a distance matrix;
S3: distributing a plurality of hash buckets for each computing node, recording the corresponding relation between the computing node and the hash buckets through a nodedatamap table of a management node, and acquiring a second mapping relation between the node fragments and the computing node according to nodedatamap table information and the corresponding relation;
S4: extracting hash keys in filtering conditions of query sentences, and obtaining node IP of a computing node corresponding to the query sentences according to the hash key query by using the second mapping relation and the first mapping relation to obtain a node IP query result;
s5: and inquiring and obtaining minimum values of the calculation node and the management node according to the node IP inquiry result by the distance matrix, and outputting the node corresponding to the minimum value as an initiating node.
According to the method for outputting the initiating node in the MPP distributed system provided by the invention, the first mapping relation in the step S1 is a one-dimensional array.
According to the method for outputting the initiating node in the MPP distributed system, the distance matrix in the step S2 is a two-dimensional matrix.
According to the method for outputting the initiating node in the MPP distributed system, in the distance matrix, when the IP of the computing node is the same as the IP of the management node, the distance between the computing node and the management node is 0, and when the IP of the computing node is different from the IP of the management node, the distance between the computing node and the management node is 1.
According to the method for outputting the initiating node in the MPP distributed system, in the step S3, when a plurality of hash buckets are distributed for each computing node, the distribution mode is uniformly distributed according to the total number of the hash buckets.
According to the method for outputting the initiating node in the MPP distributed system, the total number of hash buckets is 65536.
According to the method for outputting the originating node in the MPP distributed system provided by the present invention, step S4 further includes:
S41: extracting hash values of the distributed columns in the filtering conditions of the query statement;
s42: calculating according to the hash value to obtain a hash key;
s43: acquiring the fragments of the data corresponding to the query statement according to the hash key by the second mapping relation;
s44: and acquiring the node IP corresponding to the computing node according to the fragments of the data by the first mapping relation, and acquiring a node IP query result.
According to the method for outputting the initiating node in the MPP distributed system, in the step S4, the filtering condition is an accurate query statement containing a where condition exists in an equation distribution column.
The invention provides an originating node output method in MPP distributed system, which is used for solving the problem of network load aggravation caused by incorrect selection when the concurrent pressure is high and the network load is heavy, the method can automatically select the optimal originating node according to the MPP characteristic, the selected initiating node and the computing node are ensured to be the same machine, network overhead is saved, performance abnormality caused by incorrect selection of the initiating node is avoided, user service is completed more quickly, and user experience and use efficiency are improved.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention. The following examples are illustrative of the invention but are not intended to limit the scope of the invention.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the embodiments of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
An embodiment of the present invention is described below with reference to fig. 1.
The invention provides an originating node output method in an MPP distributed system, which comprises the following steps:
s1: collecting the number of nodes and the node IP in the database cluster, and sequencing according to the fragmentation information to obtain a first mapping relation of the node fragmentation and the node IP;
The first mapping relationship in step S1 is a one-dimensional array.
In this stage, the mapping relationship between the slices and the ip is first obtained, specifically, the number of nodes and the ip information in the database cluster are obtained, and the sequence is performed according to the slice information, so as to define N columns of one-dimensional arrays a, where N is the number of calculated nodes, and the expression is a [0] =ip1, a [1] =ip … a [ N-1] = ipN.
S2: collecting server information, creating an IP distance matrix of a computing node and a management node according to the server information, and obtaining a distance matrix;
wherein, the distance matrix in step S2 is a two-dimensional matrix.
In the distance matrix, when the IP of the computing node is the same as the IP of the management node, the distance between the computing node and the management node is 0, and when the IP of the computing node is different from the IP of the management node, the distance between the computing node and the management node is 1.
The purpose of this stage is to obtain a distance matrix between the computing node and the initiating node, specifically, firstly, according to server location information or ip information provided by a server administrator, a two-dimensional matrix M of n×n is defined, then, a distance matrix between the computing node and the managing node ip is created, when ip is the same, the distance is 0, and when ip is different, the distance is 1.
S3: distributing a plurality of hash buckets for each computing node, recording the corresponding relation between the computing node and the hash buckets through a nodedatamap table of a management node, and acquiring a second mapping relation between the node fragments and the computing node according to nodedatamap table information and the corresponding relation;
furthermore, the purpose of this stage is to obtain the relation between the fragments and the computing nodes, where N computing nodes of the MPP cluster have id ranges of N1-nN, and 65536/N hash buckets are allocated to each node, and the hash bucket ids are h0-h65535.
And nodedatamap (a system table used for recording the mapping between node fragments and data) in the MPP initiating node records the relation between the fragments and the hash buckets, each node corresponds to a plurality of hash buckets, and then the relation between the fragments and the computing node ip can be obtained through information in nodedatamap.
In step S3, when a plurality of hash buckets are allocated to each computing node, the allocation mode is that the hash buckets are uniformly allocated according to the total number of the hash buckets.
Wherein the total number of hash buckets is 65536.
In some embodiments, the GBase database corresponds to 65536 hash buckets.
S4: extracting hash keys in filtering conditions of query sentences, and obtaining node IP of a computing node corresponding to the query sentences according to the hash key query by using the second mapping relation and the first mapping relation to obtain a node IP query result;
wherein, step S4 further comprises:
S41: extracting hash values of the distributed columns in the filtering conditions of the query statement;
s42: calculating according to the hash value to obtain a hash key;
s43: acquiring the fragments of the data corresponding to the query statement according to the hash key by the second mapping relation;
s44: and acquiring the node IP corresponding to the computing node according to the fragments of the data by the first mapping relation, and acquiring a node IP query result.
Wherein, in step S4, the filtering condition is an accurate query statement containing a where condition and an equation distribution column exists.
In this stage, firstly, the hash value v in the filtering condition in the sql is extracted, the ip of the sql calculation node is determined, specifically, after the sql is obtained, the value v of the hash distribution column in the accurate query filtering condition is found, the sign is that the value v contains the where condition and the distribution column '=', after the value v is found, the crc32 (v)% 65536 is performed to calculate hashkey, wherein the crc is an encryption and decryption algorithm based on the crc checking principle, then the relation between the fragments and hashkey is obtained from nodedatamap information in the step S3, the fragment where the data is located is obtained, and the calculation node ip is obtained according to the mapping relation between the fragments and the ip obtained in the step S1.
S5: and inquiring and obtaining minimum values of the calculation node and the management node according to the node IP inquiry result by the distance matrix, and outputting the node corresponding to the minimum value as an initiating node.
The purpose of this stage is to obtain the optimal initiating node ip, find the minimum value of the distance matrix between the computing node and the initiating node obtained in step S2 according to the computing node ip obtained in step S4, min (M [ k, i ]), k is the slice value of the compute node ip, i is 0....the.n-1, and outputting the M node as a computing node.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.