CN115544055A - Calculation engine determination method and device - Google Patents

Abstract

The invention provides a calculation engine determination method and equipment. In an embodiment, the method includes: obtaining a data operation statement to be processed; determining a first memory consumption of the first engine executing the data operation statement; wherein, the first engine is a memory-based engine, and the first engine is used for Processing target data; determining the current remaining memory amount of the total memory allocated to the first engine; if the current remaining memory amount is greater than the first memory consumption amount, determining that the first engine processes the data operation statement. Thus, the intelligent selection of the engine is realized, and the balance between efficiency and stability is realized.

Description

Calculation engine determination method and device

technical field

The invention relates to the technical field of data query, in particular to a calculation engine determination method and device.

Background technique

In big data business scenarios, multiple computing engines are usually required to meet various business needs, which brings about a problem: under the premise of ensuring stability requirements, in order to ensure the best performance, different tasks need to use different computing engines. Computing engine, what is more difficult is that business personnel need to choose a suitable computing engine based on personal experience, which not only increases the workload, but also choosing a computing engine based on experience itself cannot guarantee the best choice.

Fig. 1 shows a schematic flowchart of the selection of computing engines for big data. As shown in Figure 1, the specific implementation process is as follows:

1. The Hive engine acts as a unified SQL entry to receive SQL statements;

2. After the SQL is parsed by the Hive engine, the data size of all tables involved in the SQL is obtained from the generated AST tree;

3. Compare the amount of data in the table with the preset fixed threshold. If it is less than the threshold, it is considered that the amount of data in the task is small and submitted to Trino for execution to obtain the best performance; Hive is executed to ensure stability under large amounts of data.

However, for the above technical solution, considering that the preset fixed threshold cannot be applied to clusters of different sizes, developers need to manually adjust the threshold based on experience to improve the accuracy of engine selection, and it is impossible to truly realize the intelligent selection of computing engines.

Therefore, there is an urgent need for a method that can realize intelligent selection of computing engines.

The information disclosed in this Background section is only for enhancing the understanding of the general background of the present invention and should not be taken as an acknowledgment or any form of suggestion that the information constitutes the prior art that is already known to those skilled in the art.

Contents of the invention

The embodiment of the present invention provides a calculation engine determination method and a cluster, which can determine the memory-based The engine for reading and writing and the engine based on disk reading and writing are selected to execute data operation statements, thereby realizing the intelligent selection of engines and achieving a balance between efficiency and stability.

In the first aspect, the embodiment of the present invention provides a method for determining a computing engine, the method includes: obtaining a data operation statement to be processed; determining the first memory consumption of the first engine executing the data operation statement; wherein, the first engine is based on The memory engine, the first engine is used to process the target data; determine the current memory remaining amount of the total memory allocated to the first engine; if the current memory remaining amount is greater than the first memory consumption, determine that the first engine processes the data operation statement.

In this solution, by comparing the remaining amount of total memory allocated to the memory-based engine and the memory consumption of the memory-based engine executing data operation statements, the first engine based on memory reading and writing and disk-based reading and writing can be compared. In the second engine, the engine that executes the data operation statement is selected, and then the intelligent selection of the engine is realized, and the balance between efficiency and stability is achieved.

In a possible implementation, if the current remaining memory is less than or equal to the first memory consumption, it is determined that the second engine processes the data operation statement; wherein, the second engine is a disk-based engine; the second engine is used to process the target data .

In this solution, when the first memory consumption is greater than the current remaining memory resources, on the one hand, it means that the current data volume is high, and on the other hand, it means that the memory-based engine does not have enough memory to execute data operation statements. The engine executes data manipulation statements.

Optionally, obtaining the data operation statement to be processed includes: using the data operation statement received by the second engine as the data operation statement to be processed.

In this solution, the second engine serves as an interface for unified docking of data operation statements, and can receive various data operation statements, thereby ensuring the normal realization of business.

In a possible implementation manner, determining the first memory consumption of the first engine executing the data operation statement includes: determining a logical execution tree of the data operation statement; where the logical execution tree is used to indicate the data processing represented by the data operation statement A logical flow; based on the metadata of the target data and the logical execution tree, determine the first memory consumption for the first engine to execute the data operation statement.

In this solution, based on the metadata and logical execution tree of the target data, the analysis process of the first engine actually executing the data operation statement is simulated, the resource consumption in the process of the first engine actually executing the data operation statement is obtained, and the execution data operation statement is analyzed. The real memory resource consumption, thus ensuring the intelligent selection of the engine.

In an example, based on the metadata of the target data and the logical execution tree, determining the first memory consumption of the first engine executing the data manipulation statement includes: for each node in the logical execution tree, based on the metadata of the target data, Determine the amount of data corresponding to the node; based on the amount of data corresponding to the node, determine the second memory consumption corresponding to the node; wherein, the second memory consumption is used to indicate the memory consumption of the first engine to execute the task corresponding to the node; based on logic Execute the respective second memory consumption of each node in the tree to determine the first memory consumption for executing the data operation statement by the first engine.

In this solution, based on the metadata of the target data and the memory resource consumption corresponding to each node in the logical execution tree, the analysis of memory consumption is realized.

Optionally, based on the amount of data corresponding to the node, determining the second memory consumption corresponding to the node includes: taking the amount of data to be processed by the task corresponding to the node as the memory consumption to obtain the third memory consumption corresponding to the node; determining the corresponding The data processing operation; based on the data processing operation corresponding to the node, determine the correction value corresponding to the third memory consumption; based on the correction value, correct the third memory consumption, and determine the second memory consumption corresponding to the node.

In this solution, the memory consumption is analyzed based on the correction value to determine the memory consumption that can reflect the actual memory consumption, so that the engine with better performance can be selected to execute the data operation statement.

In one implementation, the data processing operation is a scan, and the correction value is the reciprocal of the data parallelism.

In one implementation, the data processing operation is an operation based on network communication, and the correction value includes a first value and a second value, and the first value is used to indicate the number of times the first engine applies for memory during the process of executing the task corresponding to the node; the first The two values are used to indicate the ratio of the first engine to allocate and release memory during the process of executing the task corresponding to the node.

In one implementation, the data processing operation is a local processing operation, and the correction value is the second value.

Optionally, based on the respective second memory consumption of each node in the logical execution tree, determining the first memory consumption of executing the data operation statement includes: performing the respective second memory consumption of each node in the logical execution tree Summing, use the summed result as the first memory consumption for executing the data manipulation statement.

In an example, determining the logic execution tree of the data operation statement includes: performing syntax analysis on the data operation statement to determine an abstract syntax tree; performing semantic analysis on the abstract syntax tree to determine the logic execution tree.

In a possible implementation manner, obtaining the data operation statement to be processed includes: using the data operation statement of the receiving terminal as the data operation statement to be processed.

In the second aspect, the embodiment of the present invention provides an apparatus for determining a computing engine, which includes several modules, and each module is used to execute each step in the method for determining a computing engine provided in the first aspect of the embodiment of the present invention. Regarding the modules The division is not limited here. Please refer to the functions of each step of the calculation engine determination method provided in the first aspect of the embodiment of the present invention for the specific functions performed by each module in the calculation engine determination device and the beneficial effects achieved, and details will not be repeated here.

Exemplarily, the calculation engine determining means installs a memory-based first engine, and the first engine is used for managing target data; the means includes:

A statement determination module is used to obtain the data operation statement to be processed;

A consumption resource determination module, configured to determine the first memory consumption of the first engine executing the data operation statement; wherein, the first engine is a memory-based engine, and the first engine is used to process target data;

A remaining resource determination module, configured to determine the current remaining amount of memory allocated to the total memory in the first engine;

The engine selection module is configured to determine that the first engine processes the data operation statement if the current remaining amount of memory is greater than the first amount of memory consumption.

In a possible implementation, the engine selection module is further configured to determine that the second engine processes the data operation statement if the current remaining memory is less than or equal to the first memory consumption; wherein, the second engine is a disk-based engine; the second Engines are used to process target data.

Optionally, the statement determination module is configured to use the data operation statement of the receiving terminal as the data operation statement to be processed or the data operation statement received by the second engine as the data operation statement to be processed.

In a possible implementation manner, the consumption resource determination module includes an execution tree determination unit and a consumption determination unit; wherein,

The execution tree determination unit is used to determine the logical execution tree of the data operation statement; wherein, the logical execution tree is used to indicate the logical flow of data processing represented by the data operation statement;

A consumption amount determining unit, configured to determine the first memory consumption amount for the first engine to execute the data operation statement based on the metadata of the target data and the logical execution tree.

In one example, the first consumption amount determination unit includes a first consumption amount determination subunit and a second consumption amount determination subunit;

The first consumption determination subunit is configured to, for each node in the logical execution tree, determine the amount of data corresponding to the node based on the metadata of the target data; determine the task corresponding to the first engine execution node based on the amount of data corresponding to the node The second memory consumption amount; wherein, the second memory consumption amount is used to indicate the memory consumption amount of the task corresponding to the execution node of the first engine;

The second consumption determining subunit is configured to determine the first memory consumption for executing the data operation statement by the first engine based on the respective second memory consumption of each node in the logical execution tree.

Optionally, the first consumption determination subunit specifically executes the following content:

Use the amount of data that needs to be processed by the task corresponding to the node as the memory consumption to obtain the third memory consumption corresponding to the node; determine the data processing operation corresponding to the node; determine the correction value corresponding to the third memory consumption based on the data processing operation corresponding to the node ; Correct the third memory consumption based on the correction value, and determine the second memory consumption corresponding to the node.

In one implementation, the data processing operation is a scan, and the correction value is the reciprocal of the data parallelism.

In one implementation, the data processing operation is an operation based on network communication, and the correction value includes a first value and a second value, and the first value is used to indicate the number of times the first engine applies for memory during the process of executing the task corresponding to the node; the first The two values are used to indicate the ratio of the first engine to allocate and release memory during the process of executing the task corresponding to the node.

In an implementation manner, the data processing operation is a local processing operation, and the correction value is a second value, and the second value is used to indicate a ratio of memory application and release in the process of the first engine executing a task corresponding to the node.

Optionally, the second consumption determination unit is configured to sum the respective second memory consumption of each node in the logical execution tree, and use the summed result as the first memory consumption for executing the data operation statement.

In one example, the execution tree determination unit includes: a syntax analysis unit and a semantic analysis unit; wherein,

The syntax analysis unit is used to perform syntax analysis on the data operation statement and determine the abstract syntax tree;

The semantic analysis unit is used for performing semantic analysis on the abstract syntax tree to determine the logic execution tree.

In a possible implementation manner, the statement determination module is configured to use the data operation statement of the receiving terminal as the data operation statement to be processed.

In a third aspect, an embodiment of the present invention provides a calculation engine determination device, the device includes a processor and a memory; the memory stores program instructions; the processor is used to execute the program instructions, so that the device executes the method as in the first aspect.

In practical applications, the first engine includes a scheduling node and several working nodes, and the calculation engine determines that the device includes a scheduling node. Specifically, the computing engine determines that the device is a device on which the first engine is installed in the device cluster. Specifically, the device cluster includes at least one electronic device, and at least one electronic device is installed with the first engine.

Further, a second engine is also installed in the device cluster. In one example, the first engine and the second engine are installed on the same electronic device, for example, the computing engine determines that the first engine and the second engine are installed on the device at the same time. In another example, the first engine and the second engine are mounted on different electronic devices.

In a fourth aspect, an embodiment of the present invention provides a calculation engine determination device, including: at least one memory for storing programs; at least one processor for executing the programs stored in the memory, when the programs stored in the memory are executed, The processor is configured to execute the method provided in the first aspect. Exemplarily, the program is a program of the first engine, and the calculation engine determining means may be a device.

In a fifth aspect, an embodiment of the present invention provides a computing engine determination device, wherein the device runs computer program instructions to execute the method provided in the first aspect. Exemplarily, the device may be a chip or a processor. Exemplarily, the computer program instructions are programs of the first engine.

In an example, the apparatus may include a processor, and the processor may be coupled to the memory, read instructions in the memory and execute the method provided in the first aspect according to the instructions. Wherein, the memory may be integrated in the chip or the processor, or independent of the chip or the processor.

In a sixth aspect, an embodiment of the present invention provides a computer storage medium, and instructions are stored in the computer storage medium, and when the instructions are run on a computer, the computer is made to execute the method provided in the first aspect. Exemplarily, the instruction is a program of the first engine.

In a seventh aspect, an embodiment of the present invention provides a computer program product containing instructions, and when the instructions are run on a computer, the computer is made to execute the method provided in the first aspect. Exemplarily, the instruction is a program of the first engine.

Description of drawings

Fig. 1 is a schematic flow diagram of a calculation engine determination;

FIG. 2 is a first schematic diagram of the framework of a calculation engine determination system provided by an embodiment of the present invention;

Fig. 3 is a schematic flowchart of a calculation engine determination method provided by an embodiment of the present invention;

FIG. 4 is a schematic flow chart of step 320 provided in FIG. 3;

FIG. 5 is a schematic flow chart of step 322 provided in FIG. 4;

Fig. 6 is a schematic diagram of a logical execution tree provided by an embodiment of the present invention;

7 is a schematic diagram of a Trino engine and a Hive engine selection method provided by an embodiment of the present invention;

Fig. 8 is a schematic flowchart of a calculation engine determination device provided by an embodiment of the present invention;

Fig. 9 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings.

In the description of the embodiments of the present invention, words such as "exemplary", "for example" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present invention shall not be construed as being more preferred or more advantageous than other embodiments or designs. Rather, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific manner.

In the description of the embodiments of the present invention, the term "and/or" is only a kind of association relationship describing associated objects, which means that there may be three kinds of relationships, for example, A and/or B can mean: A exists alone, A exists alone There is B, and there are three cases of A and B at the same time. In addition, unless otherwise specified, the term "plurality" means two or more. For example, multiple systems refer to two or more systems, and multiple terminals refer to two or more terminals.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly specifying indicated technical features. Thus, a feature defined as "first" and "second" may explicitly or implicitly include one or more of these features. The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless specifically stated otherwise.

Since the concept of big data was put forward, enterprises have increasingly strong demands for data information to be transformed into data assets. In order to meet the needs of different big data application scenarios, a variety of big data computing engine services have emerged in the technical field, including offline analysis Hive/Spark for scenarios, Flink/Storm for real-time computing scenarios, Trino/Impala for interactive query scenarios, etc. With the continuous maturity of the big data ecology and the wide application of the integrated lake warehouse architecture, the trend of unified data lake storage at the bottom layer and compatibility with multiple computing engines at the upper layer has gradually become clear. Currently, Hive and Trino are the most commonly used computing engines in offline analysis and interactive query scenarios respectively.

Based on the Trino engine, data storage can be avoided, and its interactive analysis performance under small and medium data volumes is excellent. It has gradually become the most commonly used interactive query engine, but the stability of Trino cannot be guaranteed for analysis tasks under large data volumes. As the most commonly used offline analysis engine, Hive provides powerful data warehouse capabilities and has become the de facto technical standard in the field of offline data warehouses. However, poor offline analysis performance under small and medium data volumes has always been a problem in Hive.

In big data business scenarios, multiple computing engines are usually required to meet various business needs, which brings about a problem: under the premise of ensuring stability requirements, in order to ensure the best performance, different tasks need to use different computing engines. Computing engine, what is more difficult is that business personnel need to choose a suitable computing engine based on personal experience, which not only increases the workload, but also choosing a computing engine based on experience itself cannot guarantee the best choice.

Fig. 1 shows a schematic flowchart of the selection of computing engines for big data. As shown in Figure 1, the specific implementation process is as follows:

1. The Hive engine acts as a unified SQL entry to receive SQL statements;

2. After the SQL is parsed by the Hive engine, the data size of all tables involved in the SQL is obtained from the generated AST tree;

3. Compare the amount of data in the table with the preset fixed threshold. If it is less than the threshold, it is considered that the amount of data in the task is small and submitted to Trino for execution to obtain the best performance; Hive is executed to ensure stability under large amounts of data.

However, there are two following technical problems in the above-mentioned technical solution:

First, intelligent selection of computing engines cannot be truly realized: Considering that the preset fixed thresholds cannot be applied to clusters of different sizes, developers need to manually adjust the thresholds based on experience to improve the accuracy of engine selection.

In the second aspect, the calculation engine's automatic selection accuracy is insufficient: the size of the table data involved in obtaining the SQL from the generated AST tree is the physical storage space of the table, and the data is loaded into the memory in batches during the actual calculation process, and the memory usage and The release is performed at the same time, so the method of judging the resource consumption when the task is running through the physical storage space of the table is also inaccurate.

In order to solve the above technical problems, an embodiment of the present invention provides a method for selecting a computing engine for big data.

Fig. 2 shows an example architecture diagram of a calculation engine selection system provided by an embodiment of the present invention. The embodiment of the present invention provides that the calculation engine determination method can be applied to the system architecture diagram shown in FIG. 2 . As shown in FIG. 2 , the computing engine selection system includes a terminal device 101 and a device cluster 102 .

Wherein, the terminal device 101 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. Exemplary embodiments of terminal devices involved in this solution include but are not limited to electronic devices equipped with iOS, android, Windows, Harmony OS or other operating systems. The embodiment of the present invention does not specifically limit the type of the electronic device.

Wherein, the device cluster 102 may be implemented by an independent electronic device or a device cluster composed of multiple electronic devices. In some possible implementation manners, the electronic devices in the device cluster 102 may be terminals, computers, or servers.

In an example, the server involved in this solution can be used to provide cloud services, which can be a server that can establish a communication connection with other devices and can provide computing functions and/or storage functions for other devices or a HyperTerminal. Wherein, the server involved in this solution may be a hardware server, and may also be embedded in a virtualization environment. For example, the server involved in this solution may be a virtual machine executed on a hardware server including one or more other virtual machines.

Wherein, the terminal device 101 communicates with the device cluster 102 through the network. The network can be a wired network or a wireless network. Exemplarily, the wired network can be a cable network, an optical fiber network, a digital data network (Digital DataNetwork, DDN), etc., and the wireless network can be a telecommunication network, an internal network, the Internet, a local area network (Local Area Network, LAN), a wide area network (Wide Area Network) Network, WAN), wireless local area network (Wireless Local Area Network, WLAN), metropolitan area network (Metropolitan Area Network, MAN), public switched telephone network (Public Service Telephone Network, PSTN), Bluetooth network, ZigBee network (ZigBee), mobile Telephone (Global System for Mobile Communications, GSM), CDMA (Code Division Multiple Access) network, CPRS (General Packet Radio Service) network, etc. or any combination thereof. It can be understood that the network can use any known network communication protocol to realize the communication between different client layers and gateways, and the above network communication protocol can be various wired or wireless communication protocols, such as Ethernet, Universal Serial Bus (universal serial bus, USB), fire wire (firewire), global system for mobile communications (global system for mobile communications, GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access , CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (long term evolution, LTE), new air interface (new radio, NR), Bluetooth (bluetooth), wireless fidelity (wireless fidelity, Wi-Fi) and other communication protocols.

In the embodiment of the present invention, the device cluster 102 can store large data (referred to as target data for ease of description and distinction), and can also install multiple engines, which can read and calculate the target data. Here, the multiple engines include an engine for reading and writing based on memory (for ease of description and distinction, referred to as the first engine) and an engine for read and write based on disk (for ease of description and distinction, referred to as the second engine). The first engine can be a Trino engine or an impala engine, and the second engine can be a Hive engine. Different engines have their own advantages and disadvantages in the query process. For example, the Hive engine needs to perform multiple disk reads and writes during the query process, and each read and write to the disk will cause a delay, but the Hive engine is suitable for the query process with a large amount of data. The Trino engine is a distributed real-time query engine based on memory. There is no delay caused by disk read and write during the query process, and it has a faster query speed, but the amount of query data that can be queried is limited. Therefore, different engines need to be selected based on the actual data processing process.

It should be noted that, in the device cluster 102 provided by the embodiment of the present invention, the electronic devices where the first engine, the second engine, and the target data reside may be different, or may partially overlap. For example, the electronic devices in the device cluster 102 may store part of the target data, and may also perform part of the functions of the first engine, and may also perform part of the functions of the second engine; for another example, the device cluster 102 may be divided into three Clusters, one cluster stores target data, one cluster installs the first engine, and the other cluster installs the second engine. This embodiment of the present invention does not specifically limit this, and the server where the first engine, the second engine, and the target data are located may be specifically set in combination with actual conditions.

Based on this, after the terminal device 101 of the embodiment of the present invention acquires the query command triggered by the user, it can obtain the data operation statement and upload the data operation statement to the device cluster 102, and the first engine in the device cluster 102 can pre-calculate and execute The total memory consumption of the data operation statement is to obtain the current remaining amount of memory in the memory allocated to the first engine, and compare the total memory consumption with the current remaining amount of memory in the memory allocated to the first engine, if If it is less than, the first engine such as the Trino engine will execute the data operation statement; if it is greater than, the second engine such as the Hive engine will execute the data operation statement, so as to select the best data processing engine.

It should be pointed out that when there are multiple first engines installed in the device cluster 102, it is necessary to compare the consumption of memory resources for executing data operation statements with the current remaining memory of each first engine. If the remaining amount of memory is less than the consumption of memory resources for executing the data operation statement, the data operation statement is handed over to the second engine for processing.

The architecture diagram of the calculation engine selection system mentioned above is just an example. In some possible implementation manners, the computing engine selection system further includes an engine selection server, the terminal device 101 communicates with the engine selection server through a network, and the device cluster 102 communicates with the engine selection server through a network. For the detailed introduction of the network, refer to the above, and will not go into details. The engine selection server can be implemented by an independent server or a device cluster composed of multiple servers. In some possible implementation manners, the terminal device 101 in the embodiment of the present invention may obtain the data operation statement after acquiring the query instruction triggered by the user, and upload the data operation statement to the engine selection server. The engine selection server requests the first engine in the device cluster 102 to pre-calculate the total memory consumption for executing the data operation statement, the engine selection server obtains the current remaining memory in the memory allocated to the first engine, and calculates the total memory consumption Compare with the current remaining amount of memory in the memory allocated to the first engine, if it is less than, then hand over to the first engine in the device cluster 102 such as the Trino engine to execute the data operation statement, if it is greater than, then hand over to the device cluster 102 The second engine, such as the Hive engine, executes data manipulation statements to select the best data processing engine.

Next, a calculation engine determination method provided by an embodiment of the present invention is introduced. Wherein, the calculation engine may include a first engine based on memory reading and writing and a second engine based on disk reading and writing. Here, the first engine and the second engine are installed in the aforementioned device cluster 102 . The method can be executed by any device, device, platform, or device cluster that has computing and processing capabilities. For example, the above-mentioned device cluster 102 in Figure 2, considering that the technical solution provided by the embodiment of the present invention involves the analysis of data operation statements and the estimation of consumption, and these functions need a specific engine to determine, therefore, the preferred device cluster In 102, the electronic device on which the first engine is installed (for ease of description and distinction, referred to as a target device) executes. It should be pointed out that the first engine includes a scheduling node and multiple working nodes, and the scheduling node is used to convert data operation statements into multiple tasks and schedule them to multiple working nodes for execution. The scheduling node can be a physical electronic device, or a virtual electronic device, such as a virtual server, a virtual machine, and the like. Worker nodes are similar. For example, an electronic device of an entity may be configured with a scheduling node and a processing node at the same time, and the scheduling node and the processing node may share physical resources of the electronic device. Correspondingly, the scheduling node and the processing node may be virtual machines. Usually, the scheduling node parses the data operation statement and estimates the consumption to implement task scheduling. Therefore, the target device is the electronic device where the scheduling node is located. The following takes the target device as the execution subject as an example for description.

It should be pointed out that the device cluster 102 stores target data. Exemplarily, the target data may be massive data stored in the storage device cluster 103, such as structured database tables, semi-structured text data, and unstructured voice, picture, video and other data. In addition, metadata of object data is stored in the device cluster 102, and the object data is managed based on this. Among them, metadata (Metadata), also known as intermediary data and relay data, is data describing data (data about data), mainly describing information about data attributes (property), used to support such as indicating storage locations, Resource search, file recording and other functions. Metadata is information about the organization of data, data domains and their relationships, in short, metadata is data about data. In practical applications, the metadata of the target data is stored in a distributed file system, such as HDFS (Hadoop Distributed File System, Hadoop Distributed File System), NFS (Network File System, Network File System), etc., but not limited thereto. The deployed distributed file system can manage the target data. Exemplarily, the device cluster 102 may be deployed with a distributed file system, based on which the stored target data is managed.

Fig. 3 shows a schematic flowchart of a calculation engine determination method provided by an embodiment of the present invention. The method includes the following steps:

Step 310, receiving the data operation statement to be processed.

In practical applications, in an example, the target device receives the data operation statement sent by the terminal device 101, and the statement is a data operation statement to be processed. In one example, if the second engine in the device cluster 102 serves as a unified interface for receiving data operation statements, optionally, the second engine in the device cluster 102 receives the data operation statements sent by the terminal device 101, and converts the data operation The statement is sent to the target device where the first engine is installed.

Here, the data operation statement indicates the logic of filtering required data from target data and performing data processing. The data operation statement may be embodied by a query language such as SQL (Structured Query Language, Structured Query Language), HQL (Hibernate Query Language, a query language). Exemplarily, the present invention mainly focuses on the SQL statement to illustrate the method provided by the present invention, but considering the diversity of data manipulation statements, the present invention does not limit the specific types of data manipulation statements. It should be pointed out that both the first engine and the second engine installed in the device cluster 102 can process the same data operation statement.

In one embodiment, the data operation statement may be generated according to the user's target operation on the query interface provided by the terminal device 101 .

Exemplarily, the query interface includes an input box, and correspondingly, the query operation includes but not limited to query content input by the user through the input box. The data operation statement may be generated from the input content in the input box, and optionally, the input content may be a data operation statement.

Exemplarily, the query interface includes a data operation statement component, which refers to a component that encapsulates a data operation statement and can be used repeatedly. Correspondingly, the query operation is that the user assembles a complete data operation statement by dragging and dropping at least one data operation statement component on the query interface.

Further, the query interface includes a data operation statement generation control. When the terminal device 101 detects a click operation acting on the data operation statement generation control, it generates a data operation statement according to the query operations in the above two example scenarios, and converts the data operation statement to Upload to the device cluster 102.

Step 320, determine the first memory consumption of the first engine executing the data operation statement.

It should be noted that the first memory consumption indicates the memory consumed by the first engine to execute the data operation statement. It should be noted that, considering that in the embodiment of the present invention, it is necessary to determine the first memory consumption of the first engine to execute the data operation statement, and the processing logic of different engines is not completely consistent, therefore, usually the first engine in the device cluster 102 An engine performs the evaluation, so that the first memory consumption of the first engine executing the data operation statement can be accurately evaluated.

Step 330, determine the current remaining memory amount of the total memory allocated to the first engine.

In practical applications, the target device manages the use of resources allocated to the first engine by the device cluster 102, so as to determine the use of the total memory allocated to the first engine, and obtain the current remaining amount of memory.

Next, the target device judges whether the first memory consumption is smaller than the current remaining memory value. If yes, send the data manipulation statement to the first engine for execution, in other words, execute step 340a. Otherwise, it is executed by the second engine, in other words, step 340b is executed. As shown in FIG. 3, after step 330, either 340a or 340b is performed. The embodiment of the present invention mainly uses the Trino engine and the Hive engine to process SQL statements as an example to illustrate the method provided by the present invention, but considering the diversity of computing engines, the present invention does not limit the specific types of computing engines.

Step 340a, if the current remaining amount of memory is greater than the first amount of memory consumption, determine that the first engine processes the data operation statement.

Exemplarily, the first engine may be a trino engine. Its working principle is as follows: the trino engine contains a scheduling node and multiple working nodes. The scheduling node is used to parse the data operation statement after receiving the data operation statement, generate a logical execution plan, generate multiple actual tasks, and distribute multiple actual tasks to all working nodes. The working nodes are responsible for actually executing actual tasks, and data transmission can be performed between multiple working nodes, and each working node can interact with the distributed file system and read data stored on the device cluster 102 corresponding to the distributed file system. After the calculation is completed, the working node notifies the scheduling node to end the query, and sends the query result to the scheduling node. It is worth noting that both caller nodes and worker nodes represent threads or processes.

In specific implementation, for the trino engine, a thread pool can be pre-built, and multiple worker threads (created by multiple worker nodes) can be preset in the thread pool. After the trino engine receives the SQL statement, it can call the SQL parser to parse the SQL statement and obtain the AST tree. Afterwards, the AST tree is converted into a logical execution tree by the logical execution plan component. Through the distributed analysis of the logical execution tree through the distributed plan component, multiple plans are obtained, and each plan is converted into a corresponding task. Multiple tasks can be scheduled to multiple worker threads for execution by calling algorithms. Here, the scheduling algorithm is a random algorithm, a round-robin scheduling algorithm, a weighted round-robin algorithm, and the like.

In one embodiment, when the first memory consumption is less than the current remaining memory resources, on the one hand, it indicates that the current data volume is not high, and on the other hand, it indicates that the Trino engine has enough memory to execute the data operation statement. At this time, the Trino engine is called to execute Data manipulation statements to ensure execution efficiency.

It is worth noting that the logical calculation execution tree is obtained by the logical execution plan component in the trino engine. When it is determined that the data operation statement is executed by the Trino engine, the scheduling node can directly parse the logical execution tree to obtain multiple tasks and schedule multiple tasks. to multiple worker nodes for execution.

Step 340b, if the current remaining memory amount is less than or equal to the first memory consumption amount, determine that the first engine processes the data operation statement.

Exemplarily, the second engine may be a Hive engine. The Hive engine is a data warehouse tool based on Hadoop, which can map structured data files into a database table, and provide a simple sql query function, which can convert sql statements into MapReduce (a programming model) tasks for operation.

In specific implementation, for the Hive engine, after the Hive engine receives the SQL statement, it can call the SQL parser to parse the SQL statement to obtain the AST tree. The AST tree is converted into a logical execution tree through the logical execution plan component. Through the distributed analysis of the logical execution tree through the distributed plan component, multiple plans are obtained, and each plan is converted into a corresponding Map task. After each Map task reads data from the disk for processing, it outputs the intermediate results to the disk for storage. Because the Hive engine needs to perform multiple disk reads and writes, there will be a long delay in the query process. But it is also because the Hive engine writes intermediate results to disk, so the Hive engine does not have too many restrictions on the amount of data. It is worth noting that in practical applications, the Hive engine usually also includes a scheduling node and multiple working nodes. The scheduling node is used to parse the data operation statement after receiving the data operation statement, generate a logical execution plan, and generate multiple Map tasks. , Distribute multiple Map tasks to all working nodes.

In one embodiment, when the first memory consumption is greater than the current remaining memory resources, on the one hand, it indicates that the current data volume is high, and on the other hand, it indicates that the Trino engine does not have enough memory to execute the data operation statement. At this time, the Hive engine is called Execute data manipulation statements.

Thus, in the embodiment of the present invention, the memory-based engine and the memory-based The engine that executes the data operation statement is selected among the engines for disk read and write, and the engine based on the memory read and write can quickly realize the processing of the data operation statement, or the engine based on the disk read and write can stably realize the processing of the data operation statement, and then realize the engine The intelligent selection achieves the balance between efficiency and stability.

FIG. 4 shows a schematic flowchart of step 302 in the embodiment shown in FIG. 3 .

As shown in FIG. 4 , on the basis of the above-mentioned embodiment shown in FIG. 3 , in an exemplary embodiment of the present invention, step 320 may specifically include the following steps:

Step 321. Determine the logical execution tree of the data operation statement; wherein, the logical execution tree indicates the logical flow of data processing represented by the data operation statement.

First, the target device performs grammatical analysis on the data operation statement to obtain an AST tree. Here, AST (abstract syntax code, abstract syntax tree) is a tree representation of the abstract syntax structure of the source code, and each node on the tree represents a structure in the source code, which is abstract because the abstract syntax Trees don't represent every detail that occurs in the real grammar, for example, nested parentheses are implicit in the tree structure and not represented as nodes. The abstract syntax tree does not depend on the grammar of the source language, that is to say the context-free grammar used in the parsing phase.

Then semantically analyze the AST tree to obtain a logical execution tree.

Wherein, the logical execution tree is a tree composed of operators, indicating the processing logic of the database table. In practical applications, the logic execution tree is an optimized tree, and the optimization of the logic execution tree is a function of the first engine and the second engine itself. The specific optimization method needs to be determined in conjunction with the design of the engine.

It should be noted that after the semantic analysis of the AST tree, the semantic analysis of the AST tree can be performed to obtain the table name and data processing operation of the data table processed by the SQL statement, and further obtain the logical execution tree. FIG. 5 is a schematic diagram of a logical execution tree provided by an embodiment of the present invention. As shown in Figure 6, data processing operations include data scanning (TatleScan), filtering (Filter), projection (Project), hash join (Hashjion) to realize table join, data aggregation (Aggregete), etc.

Among them, data scanning (TatleScan) means to go through all the data in the table to display the data results. In practical applications, search according to the index and scan a part of the data to get the data results.

Among them, filter (Filter) means to select the qualified data in the table.

Among them, projection (Project) means to convert the data in the table to another memory space for processing.

Among them, table connection (JOIN) can be understood as passing certain connection conditions between multiple tables, so that the tables are associated and data can be obtained from multiple tables. Since different tables are on different servers, the table join process involves communication between servers in the device cluster 102 . Among them, hash join (Hashjion) is only a way to realize table join, which can improve the join efficiency. In practical application, there are other ways of join table.

Among them, data aggregation (Aggregete) can be understood as a process of collecting data located in different servers in a table and expressing it in summary. It should be noted that the data of a table can be stored in different servers, therefore, data aggregation is required.

It should be noted that, considering the need to determine the first memory consumption of the first engine to execute the data operation statement in the embodiment of the present invention, and the logic of the logic execution tree generated by different engines is not completely consistent, therefore, usually by the device cluster The first engine in 102 determines the logical execution tree, ensuring that the first memory consumption of the first engine executing the data operation statement can be accurately evaluated.

Afterwards, the first memory consumption for the first engine to execute the data operation statement may be determined based on the metadata of the target data and the logical execution tree.

Among them, based on the metadata, you can know the size of the data table, how many rows, the data storage space of each row, etc., and then get the data that needs to be processed when executing according to the logical execution tree, so as to analyze and execute the data operation statement more accurately. Memory consumption. As shown in Figure 4, it specifically includes the following steps:

Step 322 , for each node in the logical execution tree, based on the metadata of the target data, determine the data volume corresponding to the node; based on the data volume corresponding to the node, determine the second memory consumption corresponding to the node.

Wherein, the second memory consumption is used to indicate the memory consumption of the task corresponding to the execution node of the first engine.

First, for each node in the logical execution tree, the target device determines the amount of data corresponding to the node based on the metadata of the target data and the name of the data table processed by the node, and determines the amount of data corresponding to the node based on the amount of data corresponding to the node. The second memory consumption. The second memory consumption indicates the memory consumption of the task corresponding to the execution node of the first engine. In practical applications, the task corresponding to each node can be generated based on the logical execution tree, and the task indicates the implementation of the data operation indicated by the node. plan. Among them, based on the metadata, you can know the size of the data table, how many rows, the data storage space of each row, etc., and then get the amount of data processed by each node in the logical execution tree, so as to get the second memory consumption corresponding to each node quantity. Specifically, the amount of data may be the product of the number of data rows and the data storage space of each row.

Step 323 , based on the respective second memory consumption of each node in the logic execution tree, determine the first memory consumption for executing the data operation statement by the first engine.

Next, based on the respective second memory consumption of each node in the logical execution tree, the first memory consumption for executing the data operation statement is determined. Specifically, the second memory consumption corresponding to each node in the logical execution tree is summed to obtain the first memory consumption for executing the data operation statement.

Therefore, in the embodiment of the present invention, by restoring the analysis process of the first engine actually executing the data operation statement, the resource consumption in the process of the first engine actually executing the data operation statement is obtained, and the real memory resources for executing the data operation statement are analyzed Consumption, thereby ensuring intelligent selection of engines.

FIG. 5 shows a schematic flowchart of step 322 in the embodiment shown in FIG. 4 .

As shown in FIG. 5, on the basis of the embodiment shown in FIG. 3 above, in an exemplary embodiment of the present invention, as shown in step 322, based on the metadata and logical execution tree of the target data, the first step to execute the data operation statement is determined. The step of memory consumption may specifically include the following steps:

Step 3221, taking the amount of data to be processed by the task corresponding to the node as the memory consumption, and obtaining the third memory consumption corresponding to the node.

During specific implementation, the device cluster 102 may determine the amount of data that needs to be processed by the task corresponding to the node, and use the amount of data as the third memory consumption of the node.

Step 3222, determine the data processing operation under the task corresponding to the node.

In the embodiment of the present invention, there are multiple types of data processing operations represented by multiple nodes in the logical execution tree, and generally one node indicates a type of data processing operation. Specifically, there are three types, type 1: data scanning (TatleScan); type 2: local processing, such as filtering (Filter), projection (Project); type 3, network communication, such as table connection (JOIN), data aggregation (Aggregete).

As shown in Figure 6, the data processing operations include data scanning (TatleScan), filtering (Filter), projection (Project), hash join (Hashjion) to realize table join, and data aggregation (Aggregete). Correspondingly, the third memory consumption is cost(TatleScan), cost(Filter), cost(Project), cost(Hashjion), cost(Aggregete).

Step 3223, based on the data processing operation under the task corresponding to the node, determine the correction value corresponding to the third memory consumption.

For type 1: data scan.

Considering that the data is not loaded into the memory all at once, but the proportion of data loaded into the memory at the same time is determined according to maxPartition (maximum parallelism).

Therefore, when the data processing operation of the node is data scanning, that is, type 1, the correction value corresponding to the node is used to represent the degree of data parallelism. For example, the correction value may be W1, W1=1/maxPartition, so as to reflect the actual memory usage.

For type 2: local processing.

Considering that the tasks corresponding to the processing nodes of the first engine are processed in parallel, the occupation and release of memory resources are also performed at the same time. For example, for the trino engine, tasks corresponding to the nodes are processed in parallel through multiple working nodes. Therefore, when calculating memory resource consumption, it is inaccurate to only consider the total amount of memory resource applications, and it is necessary to take into account the release of memory during the parallel processing of tasks corresponding to nodes by the first engine.

Therefore, when the data processing operation of the node is a local processing operation, which is type 2, the correction value corresponding to the node is used to indicate the ratio of memory application and release in the process of the first engine executing the task corresponding to the node, thus reflecting The actual memory usage. For example, this may be W (also referred to as the second value), W=r1/T1, where r1 represents the real memory consumption of the process of the actual task corresponding to the first engine execution node, and T1 represents the first engine execution node The total amount of memory resources requested by the process corresponding to the actual task. As shown in FIG. 7 , the correction value corresponding to the node representing Filter (Filter) is W (Filter), and the correction value corresponding to the node representing Project (Project) is W (Project).

For type 3: network communication.

On the one hand, the actual tasks corresponding to the nodes are executed in parallel by multiple nodes. On the other hand, considering that the network communication will receive a large amount of data, the actual occupied memory is N (also called the first value) times of the input data. Here, N can be represented by the number of times the task corresponding to the node applies for memory during execution.

Therefore, when the data processing operation of a node is an operation based on network communication, which is type 3, the correction value corresponding to the node is used to indicate that the first engine applies for and releases memory during the process of executing the task corresponding to the node. On the other hand, it is used to indicate the new memory usage (indicated by the number of times of memory application) through network communication during the process of the first engine executing the corresponding task of the node, so as to reflect the real memory usage.

Exemplarily, the correction value may include a second value W and a first value N, wherein W refers to the above description and will not be repeated here. Specifically, the correction value may be W*N. As shown in Figure 7, the correction value corresponding to the node that implements the hash connection (Hashjion) is W(Hashjion)*N(Hashjion), and the correction value corresponding to the node that implements the data aggregation (Aggregete) is W(Aggregete)*N (Aggregete).

Step 3224: Correct the third memory consumption based on the correction value, and determine the second memory consumption corresponding to the node.

Specifically, the device cluster 102 uses the product of the correction value corresponding to the node and the third memory consumption as the second memory consumption corresponding to the node.

As shown in Figure 6, the second memory consumption of type 1 nodes is cost(TatleScan)/maxPartition; the second memory consumption of type 2 nodes is cost(Filter)*W(Filter); cost(Project)* W(Project); the second memory consumption of the type 3 node is cost(Hashjion)*W(Hashjion)*N(Hashjion); cost(Aggregete)*W(Aggregete)*N(Aggregete).

Therefore, in the embodiment of the present invention, the correction value is obtained by considering the memory resource consumption of each node during the actual task execution process, and the memory resource consumption is corrected based on the correction value, so that the memory resource consumption of each node during the actual task execution process is obtained. The situation of resource consumption ensures the real memory resource consumption of executing the data operation statement, thereby ensuring the intelligent selection of the engine.

It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of each process should be determined by its functions and internal logic, and should not constitute any limitation to the implementation process of the embodiment of the present invention.

Based on the calculation engine determination method provided above, the specific application of the method will be described. FIG. 6 is a schematic diagram of a method for selecting a Trino engine and a Hive engine provided by the implementation of the present invention. As shown in Figure 6, the specific contents include:

1. The Hive engine acts as a unified SQL entry to receive SQL statements. Hand over the SQL statement to the Trino engine for processing.

2. The Trino engine parses the SQL statement, generates an AST tree, and generates a logical execution tree based on the AST tree.

3. The Trino engine counts the memory consumption of executing SQL statements.

4. The Trino engine compares the memory consumption with the remaining memory of the total memory allocated to itself. If it is less than that, it considers that the data volume of the task is small, and submits it to Trino for execution to obtain the best performance and efficiency; if it is greater, it considers it The task has a large amount of data, and the SQL statement is executed by the Hive engine to ensure stability under a large amount of data.

In the embodiment of the present invention, it is possible to select the engine for executing the SQL statement from the Trino engine and the Hive engine based on the comparison result of the memory remaining amount of the Trino engine and the memory consumption of the Trino engine executing the SQL statement, and the SQL statement can be quickly implemented based on the Trino engine Or, based on the Hive engine, the stable processing of SQL statements is realized, and then the intelligent selection of the engine is realized, and the balance between efficiency and stability is achieved.

Based on the same idea as the method embodiment of the present invention, the embodiment of the present invention also provides a computing engine determining device. The device includes several modules, and each module is used to execute each step in the calculation engine determination method provided in the first aspect of the embodiment of the present invention, and there is no limitation on the division of the modules. For the specific functions performed by each module in the calculation engine determination device and the beneficial effects achieved, please refer to the functions of each step 310 to step 340b of the calculation engine determination method provided by the embodiment of the present invention described above, and will not be repeated here. .

Fig. 8 is a schematic structural diagram of an apparatus for determining a computing engine provided by an embodiment of the present invention.

As shown in FIG. 8, an embodiment of the present invention provides a calculation engine determination device. The calculation engine determination device installs a memory-based first engine, and the first engine is used to manage target data, including:

Statement determination module 801, configured to acquire data operation statements to be processed;

The resource consumption determination module 802 is configured to determine the first memory consumption of the first engine executing the data operation statement; wherein, the first engine is a memory-based engine, and the first engine is used to process target data;

A remaining resource determining module 803, configured to determine the current remaining amount of memory allocated to the total memory in the first engine;

The engine selection module 804 is configured to determine that the first engine processes the data operation statement if the current remaining amount of memory is greater than the first amount of memory consumption.

For the detailed functions of the statement determination module 801 , the resource consumption determination module 802 , the remaining resource determination module 803 and the engine selection module 804 , please refer to the description of steps 310 to 340 b above, and will not repeat them here.

Those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above-mentioned modules is used for illustration. The internal structure of the device is divided into different modules to complete all or part of the functions described above. Each module in the embodiment can be integrated into one processing unit, or each unit can exist separately physically, or two or more units can be integrated into one unit, and the above-mentioned integrated units can be implemented in the form of hardware, For example, the means for determining a computing engine may be several electronic devices in the device cluster 102, and may also be implemented in the form of a software functional unit. For example, the means for determining a computing engine may be deployed in the first engine. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present invention. For the specific working process of the modules in the above-mentioned device, reference may be made to the corresponding process in the aforementioned method embodiments, which will not be repeated here.

Based on the same idea as the method embodiment of the present invention, the embodiment of the present invention also provides an electronic device.

Fig. 9 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

As shown in FIG. 9 , an electronic device 900 includes a processor 901 , a memory 902 and a network interface 903 .

The processor 901 may be a central processing unit (Central Processing Unit, CPU), and may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), on-site Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

Memory 902 may include one or more computer program products, which may include various forms of computer-readable storage media, may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile Both sexual memories. Among them, the non-volatile memory can be read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically programmable Erases programmable read-only memory (electrically EPROM, EEPROM) or flash memory. Volatile memory can be random access memory (RAM), which acts as external cache memory. By way of illustration and not limitation, many forms of RAM are available such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), Double data rate synchronous dynamic random access memory (double data date SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (synchlinkDRAM, SLDRAM) and direct memory Bus random access memory (direct rambus RAM, DR RAM). One or more computer programs may be stored on the computer-readable storage medium. When the processor 901 executes the computer programs, the steps in the above embodiments of each calculation engine determination method are implemented, such as steps 310 to 340b shown in FIG. 3 .

Exemplarily, the computer program can be divided into one or more modules/units, and the one or more modules/units are stored in the memory 902 and executed by the processor 901 to implement the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions. For example, the computer program can be divided into a sentence determination module 801, a resource consumption determination module 802, a remaining resource determination module 803, and an engine selection module 804, and the specific functions of each module refer to the above description.

The network interface 903 is used for sending and receiving data, for example, sending data processed by the processor 901 to other electronic devices, or receiving data sent by other electronic devices.

Of course, for simplicity, only some components related to the present invention in the electronic device 900 are shown in FIG. 9 , and components such as bus, input/output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 900 may further include any other appropriate components. In addition, the electronic device may be computing devices such as desktop computers, notebooks, palmtop computers, and cloud servers. Those skilled in the art can understand that FIG. 9 is only an example of an electronic device 900, and does not constitute a limitation to the electronic device. It may include more or less components than those shown in the figure, or combine certain components, or different components, For example, the electronic device may also include an input device, an output device, a network access device, a bus, and the like. Exemplarily, the input device may be a microphone array, and may also include, for example, a keyboard, a mouse, and the like. Exemplarily, the output device may output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

In addition to the above-mentioned method, device, and electronic equipment, an embodiment of the present invention may also provide a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the processor executes the implementation of the present invention. The calculation engine of the example determines the steps in the method. Wherein, the computer program product can be written in any combination of one or more programming languages to execute the computer program codes for performing the operations of the embodiments of the present invention, and the programming languages include object-oriented programming languages, such as Java , C++, etc., and also includes conventional procedural programming languages such as the "C" language or similar programming languages. Wherein, the computer program code may be in the form of source code, object code, executable file or some intermediate form. The computer program code may be executed entirely on the electronic device, partly on the electronic device, as a stand-alone software package, partly on the electronic device and partly on a remote electronic device, or entirely on the remote electronic device implement.

In addition, the present invention also provides a device cluster 102 . Device cluster 102 includes a plurality of electronic devices. Exemplarily, the electronic device is the above-mentioned electronic device 900 . The device cluster 102 is configured to execute the steps in the calculation engine determining method of the embodiment of the present invention.

In practical application, the processors 901 in the several electronic devices 900 installed with the first engine in the device cluster 102 run the program of the first engine to execute the calculation engine determination method provided by the embodiment of the present invention. More specifically, the processor 901 in the target device where the scheduling node in the first engine is located runs the program of the first engine to execute the calculation engine determination method provided by the embodiment of the present invention.

Optionally, the device cluster 102 installs the first engine and the second engine at the same time. For example, the above-mentioned target device is equipped with the first engine and the second engine at the same time.

Optionally, the device cluster 102 may be divided into a first cluster and a second cluster, the first cluster is installed with the first engine, and the second cluster is installed with the second engine.

The installation here can be understood as storing the program of the engine in the memory 902, so that the processor 901 can run the program of the engine to realize the functions that the engine can realize.

In addition, an embodiment of the present invention may also provide a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the processor executes the computing engine of the embodiment of the present invention Identify the steps in the method. The computer readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, computer-readable media Excludes electrical carrier signals and telecommunication signals.

In the above-mentioned embodiments, the descriptions of each embodiment have their own emphases, and for parts that are not detailed or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.

The basic principles of the present invention have been described above in conjunction with specific embodiments, but it should be pointed out that the advantages, advantages, effects, etc. mentioned in the present invention are only examples rather than limitations, and these advantages, advantages, effects, etc. Every embodiment of the invention must have. In addition, the specific details disclosed above are only for the purpose of illustration and understanding, rather than limitation, and the above details do not limit the present invention to be implemented by using the above specific details.

The block diagrams of devices, devices, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, devices, systems may be connected, arranged, configured in any manner. Words such as "including", "comprising", "having" and the like are open-ended words meaning "including but not limited to" and may be used interchangeably therewith. As used herein, the words "or" and "and" refer to the word "and/or" and are used interchangeably therewith, unless the context clearly dictates otherwise. As used herein, the word "such as" refers to the phrase "such as but not limited to" and can be used interchangeably therewith.

It should also be pointed out that in the apparatus, equipment and method of the present invention, each component or each step can be decomposed and/or reassembled. These decompositions and/or recombinations should be considered equivalents of the present disclosure.

The foregoing description has been presented for purposes of illustration and description. In addition, the above description is not intended to limit the embodiments of the present invention to the above description. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

It can be understood that the various numbers involved in the embodiments of the present invention are only for convenience of description, and are not used to limit the scope of the embodiments of the present invention.

Claims (10)

1. A method for computing engine determination, the method comprising:

acquiring a data operation statement to be processed;

determining a first memory consumption of the first engine for executing the data operation statement; the first engine is a memory-based engine and is used for processing target data;

determining a current remaining amount of memory allocated to the total memory of the first engine;

and if the current memory surplus is larger than the first memory consumption, determining that the first engine processes the data operation statement.

2. The method of claim 1, further comprising:

if the current memory surplus is less than or equal to the first memory consumption, determining that a second engine processes the data operation statement;

wherein the second engine is a disk-based engine; the second engine is to process the target data.

3. The method of claim 1 or 2, wherein determining the first memory consumption amount of the first engine to execute the data operation statement comprises:

determining a logical execution tree of the data operation statement; wherein the logic execution tree is used for indicating the logic flow of the data processing represented by the data operation statement;

and determining a first memory consumption amount of the first engine for executing the data operation statement based on the metadata of the target data and the logic execution tree.

4. The method of claim 3, wherein determining a first amount of memory consumption for a first engine to execute the data operation statement based on the metadata of the target data and the logic execution tree comprises:

for each node in the logic execution tree, determining a data amount corresponding to the node based on the metadata of the target data; determining a second memory consumption corresponding to the node based on the data amount corresponding to the node; the second memory consumption is used for indicating the consumption of the memory of the task corresponding to the node executed by the first engine;

and determining a first memory consumption of the first engine for executing the data operation statement based on the respective second memory consumption of each node in the logic execution tree.

5. The method according to claim 4, wherein the determining the second memory consumption amount corresponding to the node based on the data amount corresponding to the node comprises:

taking the data quantity which needs to be processed by the task corresponding to the node as memory consumption, and obtaining a third memory consumption quantity corresponding to the node;

determining data processing operation corresponding to the node;

determining a correction value corresponding to the third memory consumption based on the data processing operation corresponding to the node;

and correcting the third memory consumption based on the correction value, and determining a second memory consumption corresponding to the node.

6. The method according to claim 4 or 5, wherein the determining a first memory consumption amount for executing the data operation statement based on a respective second memory consumption amount of each node in the logic execution tree comprises:

and summing the respective second memory consumption of each node in the logic execution tree, and taking the summed result as the first memory consumption for executing the data operation statement.

7. The method according to claim 5 or 6, wherein the determining a correction value corresponding to the third memory consumption amount based on the data processing operation corresponding to the node comprises:

if the data processing operation is scanning, the correction value is the reciprocal of the data parallelism; or

If the data processing operation is based on network communication, the correction value comprises a first numerical value and a second numerical value, and the first numerical value is used for indicating the number of times of applying for the memory in the process of executing the task corresponding to the node by the first engine; the second value is used for indicating the proportion of applying for and releasing the memory in the process of executing the task corresponding to the node by the first engine; or

And if the data processing operation is a local processing operation, the correction value is the second numerical value.

8. The method according to any one of claims 1 to 7, wherein the obtaining the data operation statement to be processed comprises:

and taking the data operation statement of the receiving terminal as the data operation statement to be processed, or taking the data operation statement received by the second engine as the data operation statement to be processed.

9. The method according to any one of claims 1 to 8, wherein after determining that the first engine processes the data operation statement if the current amount of memory remaining is greater than the first amount of memory consumption, further comprising:

determining a task for each node in the logical execution tree, the task being executed by the first engine.

10. A computing engine determination device, comprising: the apparatus includes a processor and a memory;

the memory stores program instructions;

the processor is configured to execute the program instructions to cause the apparatus to perform the method of claims 1 to 9.

Images