KR101512647B1 - Method For Choosing Query Processing Engine - Google Patents

Abstract

The present invention discloses a query process engine selection method. The query process engine selection method is used in a data processing system including multiple query process engines. The method includes: a query reception step; a query process engine evaluation step of evaluating the query process engines which are to process the query; a query transmission step of transmitting the query to one of the query process engines selected based on the evaluation result obtained in the query process engine evaluation step; and a query processing step of enabling the selected query process engine to process the query.

Description

A method for selecting a query processing engine {Method For Choosing Query Processing Engine}

The present invention relates to a method for selecting a query processing engine for processing a query in a data processing system including a plurality of query processing engines.

It should be noted that the following description merely provides background information related to the present embodiment and does not constitute the prior art.

With the everyday use of personal computers (PCs), mobile devices and the Internet, the amount of data that an IT service operator has to process is growing exponentially. UCC (User Created Contents) and SNS (Social Network Service) data produced by users are not only in terms of data growth rate but also in form and quality. Therefore, such diverse and vast amounts of data can be used as important factors for future competitiveness of companies and countries. In the past, attempts have been made to analyze large data and find meaningful information, but the current BigData environment is far from the past in terms of data volume and diversity compared to the past.

Hadoop has been developed to handle large-scale unstructured data such as HTML and TEXT in the Internet environment based on Google's GFS (Google File System), a recently introduced big data processing system. Hadoop includes an engine that handles queries such as Hadoop Distributed File System (HDFS) and Structured Query Language (SQL) used in relational databases in HDFS. It is common to query a system that processes big data and take more than several days to get results. Although HDFS has been developed for various queries such as Tajo, Impala, Hive, and MapReduce, each query engine has its own characteristics, so even if the same query is performed, Time to obtain the resource, and resources required to perform it. Therefore, there is a problem in selecting a suitable query processing engine according to the query.

The main object of the present embodiment is to provide a method of selecting a query processing engine when a system for processing data uses a plurality of query processing engines.

According to an aspect of the present invention, there is provided a data processing system including a plurality of query processing engines, the method comprising: receiving a query; A query processing engine evaluation process of evaluating the plurality of query processing engines to perform the query; Transmitting the query to the query processing engine selected as a result of the evaluation in the query processing engine process; And a query processing step of performing a query in which the selected query processing engine performs the query.

According to another aspect of the present invention, there is provided a query processing system comprising: a query processing engine selection unit that receives a query and selects one of the plurality of query processing engines; A query history log storing history information on the query; A dynamic resource allocation unit allocating resources for performing the query; And a data storage unit including a plurality of data nodes.

As described above, according to the present embodiment, the query processing engine combines the same query, similar query, or plural queries with the received query to obtain history information about past queries that are evaluated to be the same or the same, It is possible to provide a method of selecting the most efficient engine among a plurality of query processing engines for processing the big data by evaluating the system situation and the execution plan of the query processing of each query processing engine. The time required for processing the big data is very different from the execution time according to the query processing engine, so it is important to select an appropriate engine according to the query. According to the embodiment of the present invention, the optimum query processing engine can be selected .

If the received query and the queries that have been performed in the history log of the query history are the same and the available system conditions are the same, the history based evaluation is performed. If there is no history information from the query processing engine An execution plan is evaluated by receiving an Explain Plan for the query. In addition, a hybrid assessment can be made that weighs the historical and performance plan based evaluations if the system conditions at the time of receiving the query are changed or if it is difficult to evaluate the optimal engine by history based evaluation alone. If an optimal query processing engine is selected through such evaluation, the time for performing the query can be shortened as much as possible.

According to this embodiment, the dynamic resource allocator allocates a maximum available resource at the time of requesting a resource. However, during the execution of the query, the dynamic resource allocator allocates a resource having a higher priority than the currently executing query When a request is received, the resource performing the query can be readjusted. The dynamic resource allocation unit is capable of retrieving resources when resource utilization is low, and can specify a maximum or minimum range of resources allocated for performing a query.

1 is a block diagram of a HDFS (Hadoop Distributed File System).
2 is a configuration diagram of a big data processing system including a plurality of query processing engines according to the present embodiment.
3 is a flowchart of a process of selecting a query processing engine according to the present embodiment.
FIG. 4 is a flowchart of a process of dynamically allocating resources according to the present embodiment.

Hereinafter, the present embodiment will be described in detail with reference to the accompanying drawings.

Although the present embodiment is described on the basis of Hadoop (Hadoop) and HDFS (Hadoop Distributed File System), the structure for storing data is not limited thereto. As a system for processing big data, there may be various systems such as GFS (Google File System) and MapReduce, and the technical idea of the present invention is not limited to a specific system for processing big data.

1 is a block diagram of a HDFS (Hadoop Distributed File System).

The Hadoop Distributed File System (HDFS) is a technology that allows Hadoop distributed file systems to store large amounts of collected data on multiple servers for processing big data. The HDFS is composed of a NameNode 110 and a data node 120. The name node 110 stores meta information of an actual file stored in a data node (DataNode), and is not a place where actual data is stored. The name node (secondary) 114 includes a name node (master) 112 and a name node (secondary) 114. When a name node (master) 112 fails, (Master) 112 or to recover a name node (master)

Data nodes 01 to 121, data nodes 02 to 122, data nodes 03 to 123, data nodes 04 to 124, and data nodes 05 to 125, which are members of the data node 120, Networked servers or storage. The name node 110 has a file stored in the data node 120 and information of the actually stored data node 120. When an application program or a user wants to access a file, the name node 110 searches for and accesses the data node 120 in which the file is stored.

2 is a configuration diagram of a big data processing system including a plurality of query processing engines according to the present embodiment.

When a client such as a user or an application requests a query to the BigData processing system 200, the query processing engine selection unit 210 of the big data processing system 200 transmits a query request And receives resource information including a memory usage amount and a CPU usage rate of a Hadoop cluster in the dynamic resource allocation unit 220 and receives the resource information. In response to the query received from the query history log 230, And obtains the history information about the user. However, the resource information is not limited to the memory usage amount and the CPU usage rate information. The query restrike log 230 stores resource information including the query, the query execution time, the amount of memory used for query execution, and the CPU occupancy information. The query processing engine selection unit 210 evaluates the query history information, the resource status of the big data processing system, and the query execution plan for each query processing engine, and selects the optimum query processing engine in the query processing engine unit 240. A method for selecting an optimal query processing engine among a plurality of query processing engines included in the query processing engine unit 240 by the query processing engine selection unit 210 will be described in detail with reference to FIG.

Hadoop cluster-based query processing engines include Tajo 242, Impala 244, Hive 246, MapReduce, HBase, Pig, etc. . However, since each engine includes unique characteristics, there is a great difference in the execution time depending on the engine for a specific query. It is common for a BigData processing system to take several days to process the data in batches and generate the results. Therefore, it is important to select an appropriate query processing engine according to the query.

The dynamic resource assignment unit 220 monitors system resource information of the Hadoop cluster, and when the request is made by the query processing engine selection unit 210, the dynamic resource assignment unit 220 allocates the system resource to the query processing engine for performing the query, do. In addition, resources can be reassigned when a new query request arrives while the query is being performed. A method of allocating system resources will be described in detail with reference to FIG.

The query processing engine selected to perform the query processes the data of the Hadoop cluster 250 storing the actual data by allocating resources in the dynamic resource assignment unit 220 to generate a query result, Unit 210, and the query processing engine selection unit transfers it to the client again. The dynamic resource assignment unit 220 receives resource information of the system from the Hadoop cluster 250 while the query is being executed. The Hadoop cluster 250 may be composed of a name node 110 and a data node 120 constituting the HDFS of FIG.

When the query processing engine finishes executing the query, the query processing engine stores resource information including the query, the query execution time, the memory usage used in query execution, and the CPU usage rate in the query history log 230.

The big data processing system 200 according to the embodiment of the present invention may be applied to a personal computer (PC), a notebook computer, a tablet, a personal digital assistant (PDA), a game console, a portable multimedia player A user terminal such as a portable multimedia player (PMP), a PlayStation Portable (PSP), a wireless communication terminal, a smart phone, a TV, a media player, And may be part of the Big Data Processing System 200. The big data processing system 200 according to the embodiment of the present invention may be a server terminal such as an application server and a service server. (I) a communication device such as a communication modem for performing communication with various devices or wired / wireless communication networks, (ii) a communication device for storing data for executing a program, A memory, (iii) a microprocessor for executing and controlling a program, and the like. According to at least one embodiment, the memory may be a computer such as a random access memory (RAM), a read only memory (ROM), a flash memory, an optical disk, a magnetic disk, or a solid state disk Readable recording / storage medium. According to at least one embodiment, a microprocessor can be programmed to selectively perform one or more of the operations and functions described in the specification. In accordance with at least one embodiment, the microprocessor may be implemented in hardware, such as an Application Specific Integrated Circuit (ASIC), in wholly or partially of a particular configuration.

3 is a flowchart of a process of selecting a query processing engine according to the present embodiment.

FIG. 3 illustrates a process in which, when a client requests a query to the big data processing system 200, the big data processing system 200 receiving the query selects a query processing engine suitable for processing the query. The query processing engine selection unit 210 receives the query (S310) and, if the same query as the query received from the query history log has been performed in the past, a similar query or query that can be evaluated to be the same as the history information or the received query And obtains history information about the query that is evaluated to be equal to or identical to the received query (S320). The history information is information about the resource including the query, the execution time of the query, the amount of memory used for executing the query, and the CPU occupancy rate, but is not limited to the listed items. The query processing engine selection unit 210 receives available resource information from the dynamic resource assignment unit 220 for query execution.

If a query that is the same as or identical to the query received by combining the same query, the similar query, and the plurality of queries as the received query has not been performed in the past, the history based evaluation can not be performed, and thus the presence or absence of the history is determined ).

If history-based evaluation can not be performed, the query processing engine selection unit 210 transmits a query to each of the query processing engines of the query processing engine unit 240 and outputs an execution plan for the query Or Execution Plain) (S350). The query processing engine selection unit 210 performs an execution plan based evaluation for evaluating the number of executing steps for performing a query in the execution plan of each engine and the number of operations that can be distributedly processed (S355) , And selects a query processing engine having many distributed processes.

If there is history information about the query received in the query history log and history-based evaluation is performed, whether or not an optimal query processing engine can be selected even if it is evaluated only based on the history base, It is determined whether to perform the hybrid based evaluation (S360). If the same query as the received query has already been performed and remains in the query history log 230 and the available resource information can be evaluated to be the same or the same, there is no need to evaluate the execution plan again. This is because the initial state without the history log of the query is evaluated based on the execution plan. Therefore, in this case, the evaluation is performed based on the history (S370).

However, even if there is a log record for the same query, there is a change in the state of available resources, or there is no evaluation for the same query, but there is a record that has performed similar queries, or a query It is not enough to select an optimal query processing engine by evaluating only based on the history. Therefore, in this case, a hybrid evaluation is performed (S380). In the hybrid evaluation (S380), the execution time of the query based on the history based evaluation, the number of the query execution steps of the execution plan based evaluation, and the number of operations that can be distributedly processed in the query are weighted and evaluated.

The most optimal query processing engine among a plurality of query processing engines is selected through history-based evaluation, execution plan-based evaluation, and hybrid evaluation (S390).

FIG. 4 is a flowchart of a process of dynamically allocating resources according to the present embodiment.

When the query processing engine selecting unit 210 selects a query processing engine to execute a query among the query processing engine unit 240 (S410), the query processing engine receives the query and performs a query to the dynamic resource requesting unit 220 (S420).

In the data node 120 of FIG. 1, each data node (data node 01 (121), ..., data node 125) has the same number of preemptive resource occupations ). The preemption process is a process for preempting resources. When receiving the resource allocation request, the dynamic resource assignment module 210 assigns the largest possible preemption process to allocate the largest usable resource to the query processing engine to perform the query.

The query processing engine that has been allocated resources performs a query (S430). While the query processing engine is executing the query, the query processing engine selection unit 210 receives a new query and selects a query processing engine to process the query When a resource is requested to the dynamic resource assignment unit 220, there is a need to readjust the resource to perform the query according to factors such as the priority of the query to be performed. The dynamic resource assignment unit 220 receives the resource information in use from the process for preemption of the data node 120, recovers the preemption process to reallocate resources when the utilization rate is low (S432). The query processing engine determines whether the execution of the query is terminated (S434). When the execution of the query is completed, the query processing engine stores the query execution history information in the query history log 230 (S440). History information includes, but is not limited to, query content, query execution time, amount of memory used for query execution, and CPU utilization. The query processing engine completes the query execution, generates the result, and transmits the result to the query processing engine selection unit (S460).

The foregoing description is merely illustrative of the technical idea of the present embodiment, and various modifications and changes may be made to those skilled in the art without departing from the essential characteristics of the embodiments. Therefore, the present embodiments are to be construed as illustrative rather than restrictive, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be construed according to the following claims, and all technical ideas within the scope of equivalents thereof should be construed as being included in the scope of the present invention.

110 Namenode 120 data node
200 Big Data Processing System 210 Query Processing Engine Selection Unit
220 Dynamic resource allocation part 230 Query history log
240 query processing engine part 250 Hadoop cluster

Claims (18)

In a data processing system including a plurality of query processing engines,
Receiving a query;
A query processing engine evaluation process of evaluating the plurality of query processing engines to perform the query;
Transmitting the query to the query processing engine selected as a result of evaluation in the query processing engine evaluation process; And
Wherein the selected query processing engine executes a query process
Wherein the query processing engine evaluation process comprises: receiving an execution plan from the plurality of query processing engines; Extracting the number of execution steps of the query and the number of executable operations of the query in the execution plan; And selecting one of the plurality of query processing engines by comparing the number of execution steps with the number of the distributed execution executable operations. delete delete The method according to claim 1,
The query processing engine evaluation process includes:
And an execution plan based evaluation process for evaluating the execution plan based on the execution plan. delete 5. The method of claim 4,
The execution plan-
Wherein the execution plan based evaluation is performed when there is no history information on a query that is the same as or identical to the query by combining the same query, similar query, and multiple queries with the query. In a data processing system including a plurality of query processing engines,
Receiving a query;
A query processing engine evaluation process of evaluating the plurality of query processing engines to perform the query;
Transmitting the query to the query processing engine selected as a result of evaluation in the query processing engine evaluation process; And
Wherein the selected query processing engine executes a query process
Wherein the query processing engine selection process includes a hybrid method of weighting and evaluating the execution time of the query based on the history based evaluation, the number of execution steps of the query of the execution plan based evaluation, And evaluating the query processing engine. 8. The method of claim 7,
In the hybrid evaluation process,
The hybrid evaluation is performed when the information included in the history information is a query similar to the query, a query evaluated to be the same as or identical to the query by combining a plurality of queries, or the resource information is not similar even though the query is the same query A query processing engine selection method. 8. The method of claim 7,
Wherein the step of performing the query includes a step of allocating a resource from a dynamic resource allocator. 10. The method of claim 9,
Wherein the dynamic resource assignment unit allocates a preemption process occupying resources in a plurality of data nodes (DataNode) storing data. 11. The method of claim 10,
Wherein the number of preempting processes is equal to the number of CPUs of the data node. 10. The method of claim 9,
Wherein the dynamic resource assignment unit is capable of retrieving a process for a preemption assigned to the query processing engine. 10. The method of claim 9,
The dynamic resource allocation portion may include information on a minimum number and a maximum number of preemption processes to be allocated to the query processing engine and a minimum time and a maximum time when the preemption process can be used by the query processing engine A query processing engine selecting method characterized by: In a data processing system including a plurality of query processing engines,
Receiving a query;
A query processing engine evaluation process of evaluating the plurality of query processing engines to perform the query;
Transmitting the query to the query processing engine selected as a result of evaluation in the query processing engine evaluation process; And
Wherein the selected query processing engine executes a query process
Wherein the query execution process includes the step of storing the history information of the query including the execution time of the query, the memory information used for executing the query, and CPU information in a query history log . delete delete A query processing engine selection unit which receives a query and selects a query processing engine from among a plurality of query processing engines;
A query history log storing history information on the query;
A dynamic resource allocation unit allocating resources for performing the query; And
A data storage unit including a plurality of data nodes
Wherein the query processing engine selection unit receives an execution plan from the plurality of query processing engines and extracts the number of execution steps of the query and the number of distributed executable operations of the query in the execution plan And an execution plan based evaluation unit for comparing the number of execution steps with the number of distributed execution operations to select any one of the plurality of query processing engines. A query processing engine selection unit which receives a query and selects a query processing engine from among a plurality of query processing engines;
A query history log storing history information on the query;
A dynamic resource allocation unit allocating resources for performing the query; And
A data storage unit including a plurality of data nodes
Wherein the query processing engine selection unit includes a hybrid evaluation unit for weighting and evaluating the execution time of the query based on the history based evaluation, the number of execution steps of the query of the execution plan based evaluation, and the number of the distributed execution executable queries And a query processing engine selection device.

Images