TWI778924B - Method for big data retrieval and system thereof - Google Patents

Abstract

A method for big data retrieval and system thereof are adapted to solve the poor execution efficiency of the conventional big data analysis platform. The method includes: making a data clustering and proceeding a corresponding distributed indexed method by applying a stacked sparse autoencoder and an Elasticsearch techniques; in a state that multiple date-search jobs are pending to be executed, obtaining a corresponding predicted working time for each date-search job by a time computation model, and determining the executing orders for the multiple date-search jobs based on the corresponding predicted working times.

Description

Big data retrieval method and system

The present invention relates to a retrieval method and system, especially a big data retrieval method and system for improving execution efficiency.

In recent years, due to the rapid growth of data volume and various objective conditions such as the cost reduction of storage devices, the evolution of software technology and the increasingly mature cloud environment, big data analysis has developed rapidly. In order to support the processing of big data, Hadoop or Spark, a cloud computing architecture based on Java language, is a common commercial big data platform. Since Hadoop uses the first-in-first-out (FIFO, First Input First Output) technology by default, it assigns priority to the work according to the order in which it arrives. Although this scheduling is relatively fair, it does not take into account the different processing time required for each work. It is easy to lead to low data processing and long execution time.

In view of this, it is still necessary to improve the conventional big data retrieval method and system.

In order to solve the above problem, the purpose of the present invention is to provide a big data retrieval method and system, which can reduce the average waiting time of the work flow and improve the data processing efficiency by estimating the size sequence of the work time as the work scheduling basis.

Another object of the present invention is to provide a big data retrieval method and system, which can improve the grouping effect of data through the application of stacked sparse autoencoders.

Another object of the present invention is to provide a big data retrieval system and system, which can perform distributed indexing on grouped data through a distributed search engine based on elastic search technology, so as to improve the query performance of the system.

The use of the quantifier "a" or "an" for the elements and components described throughout the present invention is only for convenience and provides a general meaning of the scope of the present invention; in the present invention, it should be construed as including one or at least one, and a single The concept of also includes the plural case unless it is obvious that it means otherwise.

The term "coupling" mentioned throughout the present disclosure includes electrical and/or signal direct or indirect connection, which can be selected by those with ordinary knowledge in the art according to the usage requirements.

"Computer" as used throughout the present invention refers to various data processing devices with specific functions and implemented by hardware or hardware and software, especially having a processor to process analysis information and/or generate corresponding data Control information, such as a server, a virtual machine, a desktop computer, a notebook computer, a tablet computer or a smart phone, etc., can be understood by those with ordinary knowledge in the technical field to which the present invention pertains.

The big data retrieval method of the present invention performs the following steps through a computer, including: establishing a learning model through a neural network of a stacked sparse auto-encoder formed by stacking a plurality of sparse auto-encoders after layer-by-layer training; Inputting multiple raw data with multiple dimensions into the learning model, projecting the multiple raw data into eight quadrants of the three-dimensional space through the learning model, and adding a quadrant corresponding to each raw data to the learning model raw data to form clustered data; based on elastic search technology and using a distributed search engine, the clustered data is distributed indexed to generate corresponding index values corresponding to the quadrature values of each of the clustered data; The corresponding index value is stored in a distributed file system for a data retrieval work; based on deep neural The network pre-establishes a time operation model. The time operation model is based on the amount of data, the number of data columns, the number of data columns, the time complexity of the program, the search engine environment used and the remaining memory size of the system in the data retrieval work. obtaining an estimated working time for predicting the time spent on executing the data retrieval job; in a state where a plurality of data retrieval jobs are pending, obtain the estimated working time corresponding to each data retrieval job by using the time operation model The estimated working time is determined, and the execution sequence of the corresponding multiple data retrieval jobs is determined according to the magnitude of the multiple estimated working time.

The big data retrieval system of the present invention includes: a computer with a processing unit, a memory unit and a hard disk unit, the processing unit has a processor, the memory unit and the hard disk unit are coupled to the processing unit A grouping index module, which has a grouping unit and an indexing unit coupled to the grouping unit; the grouping unit is stacked through a neural network of a stacked sparse auto-encoder after layer-by-layer training through a plurality of sparse auto-encoders. establishing a learning model; through the learning model, projecting a plurality of original data of multiple dimensions into eight quadrants of the three-dimensional space, and adding a quadrant corresponding to each original data to the original data to form grouping data; The indexing unit receives the grouping data obtained by the grouping unit, and applies a distributed search engine based on elastic search to perform a distributed index on the grouping data, so as to generate a corresponding index corresponding to the quad value of each of the grouping data The corresponding index values are then stored in a distributed file system for a data retrieval work; and a scheduling module having a time estimation module and a sorting unit; the time estimation module It has a pre-established time operation model based on deep neural network. The time operation model is based on the amount of data, the number of data columns, the number of data columns, the time complexity of the program, and the search engine environment and system used in the data retrieval work. Remaining memory size to obtain an estimated work time for predicting the time spent performing the data retrieval job; in a state in which there are multiple data retrieval jobs to be processed, the sorting unit receives a plurality of corresponding estimated jobs time, and the execution sequence of the corresponding plurality of data retrieval jobs is determined according to the size sequence of the plurality of estimated work times.

Accordingly, the big data retrieval method and system of the present invention can automatically The encoder can improve the grouping effect of data; the distributed indexing of grouped data through elastic search technology can improve the query performance of the system; through a time operation model based on deep neural network, an estimated working time can be obtained, and The corresponding execution order is determined according to the magnitude order of the plurality of estimated work times, so as to reduce the average waiting time of the work flow and improve the data processing efficiency.

Wherein, the execution order of the plurality of data retrieval tasks can be determined according to a plurality of priorities corresponding to the plurality of data retrieval tasks and the plurality of estimated working times; each priority is a A value that is predefined or set in a predefined manner. In this way, the corresponding execution order can be determined according to the priority and the estimated work time, so as to reduce the average waiting time of the work flow and improve the data processing efficiency.

The time complexity can be calculated by calculating a total number of execution times as a calculation index of the time complexity according to a relationship between the loop commands and/or call letters included in the data retrieval job. In this way, an accurate estimated working time can be obtained by calculating the time complexity.

Wherein, the time complexity can be calculated according to a relationship between the loop instructions and/or call letters included in the data retrieval work to calculate a total number of executions, and take the power of the total number of executions, and ignore its All coefficients, and the one with the largest power value is used as the calculation index of time complexity. In this way, the time complexity calculation can be appropriately simplified by exponentiating the time complexity, which can improve the calculation efficiency, and at the same time still have the effect of reflecting the execution time of a program.

Wherein, when the data retrieval work is performed, the retrieval engine is selected according to a percentage value of available memory between the remaining memory of a memory unit of the computer and all the memory; when the percentage value of available memory is less than 25%, Select the Hive engine; when the available memory percentage value is not less than 25% and not greater than 50%, select the Impala engine; when the available memory percentage value is greater than 50%, select the SparkSQL engine; among them, the capacity of the memory unit for 20 GB. In this way, an appropriate retrieval engine can be selected according to the size of the available memory percentage value, so as to exert the best performance of the retrieval engine, and has the effect of improving the overall performance of the system.

Wherein, when the computer receives a data retrieval job, it first inquires whether there is a corresponding data from a memory unit of the computer. If the corresponding data is hit from the memory unit, the computer reads the corresponding data from the memory unit. and write it into an output file; if the corresponding data is not hit from the memory unit, the computer inquires whether there is the corresponding data from the hard disk unit; if the corresponding data is hit from the hard disk unit, the computer The hard disk unit reads the corresponding data and writes it into the output file; if the corresponding data is not hit from the hard disk unit, the data retrieval work will first predict a corresponding estimated working time through the time operation model; The priority and the estimated work time are arranged in the corresponding work execution sequence; and when the data retrieval work is executed, the corresponding retrieval engine is selected according to a relationship between the currently available memory percentage value to select the corresponding retrieval engine from the distributed file system. Search for the corresponding data. In this way, the mechanism of retrieving the memory unit and the disk unit first can save excessive hardware resource consumption caused by repeated retrieval, and has the effect of saving retrieval system resources.

1: Computer

11: Processing unit

12: Memory unit

13: Hard disk unit

2: Clustering index module

21: Grouping unit

22: Index unit

3: Scheduling module

31: Time Estimation Module

32: Sort unit

4: Search engine selection module

S1: Steps to build a sparsity model

S2: Data grouping step

S3: Data index decentralization step

S4: Time operation model establishment and estimation steps

S5, S5': work scheduling steps

S6: Search engine selection step

S7, S7': data retrieval execution steps

[FIG. 1] A system architecture diagram of an embodiment of the present invention.

[FIG. 2] A flow chart of a method according to the system of the present invention.

In order to make the above-mentioned and other objects, features and advantages of the present invention more obvious and easy to understand, the preferred embodiments of the present invention are given below and described in detail with the accompanying drawings; in addition, the same symbols are marked in different drawings. are considered to be the same, and their descriptions will be omitted.

Please also refer to FIG. 1, which is an embodiment of the big data retrieval system of the present invention The block diagram includes a computer 1, a grouping module 2, a scheduling module 3 and a search engine selection module 4, wherein the computer 1 is coupled to the grouping index module 2, the scheduling module 3 and the The search engine selects module 4 .

The computer 1 includes a processing unit 11 , a memory unit 12 and a hard disk unit 13 . The processing unit 11 includes one or more arithmetic elements such as microprocessors, microcontrollers, digital signal processors, central processing units, programmable logic controllers, analog circuits, digital circuits, or circuit boards with the aforementioned functions. The memory unit 12 and the hard disk unit 13 are coupled to the processing unit 13 to cooperate with the processing unit 13 to perform signal processing and storage. It should be noted that the execution/calculation of each element, each unit, each module, each function or each logic, etc. of the present invention is not directly stated to be executed by the processor in the processing unit of the computer, but this part is performed. Those with ordinary knowledge in the technical field to which the present invention pertains can understand, and will not be described in detail.

The grouping index module 2 includes a grouping unit/sub-module 21 and an indexing unit/sub-module 22 , wherein the indexing unit 22 is coupled to the grouping unit 21 . The grouping unit 21 includes a stacked sparse autoencoder (SSAE, Stacked Sparse Autoencoder), and the stacked sparse autoencoder includes a plurality of sparse autoencoders (SAE, Sparse Autoencoder), that is, the stacked sparse autoencoder is composed of multiple A learning model of a neural network composed of a stack of sparse autoencoders trained layer by layer.

Among them, the known data clustering learning model is trained with an autoencoder (AE, Autoencoder). When training the aforementioned autoencoder, the nodes of the hidden layer will be activated too frequently, and there are too many degrees of freedom, which is easy to cause excessive fitting result. In order to reduce the activation ratio of hidden layer nodes, this embodiment adds a sparse constraint function to the encoder, so that each node can only be activated by a specific type of input signal, not all input signals can activate each node, so as to extract more relevant Characteristics. In detail, the auto-encoder with sparse constraints is the above-mentioned sparse auto-encoder, and its loss function is defined as shown in the following formulas (1)~(3).

其中，在公式(1)中，L代表無稀疏約束時的損失函數，KL則是KL散度(KL divergence)。P表示網路中神經元的期望激活程度；若激活函數為Sigmoid函數，此值可設為0.05，表示大部分神經元未被激活。

表示第j個神經元的平均激活程度。在公式(2)中，使用KL散度來衡量隱藏層節點的平均激活輸出和稀疏度ρ之間的相似性。在公式(3)中，

定義為訓練樣本集上的平均激活程度，m為訓練樣本個數，a _j 與為x ⁱ 隱藏層第j個節點對i個樣本的響應輸出。

Among them, in formula (1), L represents the loss function without sparse constraints, and KL is the KL divergence. P represents the expected activation degree of neurons in the network; if the activation function is a sigmoid function, this value can be set to 0.05, indicating that most neurons are not activated.

represents the average activation of the jth neuron. In Equation (2), the KL divergence is used to measure the similarity between the average activation output of hidden layer nodes and the sparsity ρ. In formula (3),

It is defined as the average activation degree on the training sample set, m is the number of training samples, and a _j is the response output of the jth node of the xi hidden layer to the ⁱ sample.

After being trained layer by layer by the aforementioned multilayer sparse autoencoder (SAE), the stacked neural network is called a stacked sparse autoencoder (SSAE). Among them, the appropriate training mechanism is: the encoded output of the previous stage is the input of the next stage. In detail, after the input layer is trained by the first-stage sparse auto-encoder, n nodes of the first hidden layer are obtained; and the n nodes of the first hidden layer are trained by the second-stage sparse auto-encoder to obtain the second k nodes of the hidden layer; then k nodes of the second hidden layer are trained by the third sparse stage sparse autoencoder to obtain several nodes of the third hidden layer, and analogous to the training of multiple sparse stages according to these patterns. In other words, the hidden layer nodes of each layer can be regarded as a new set of features generated by the previous layer. After such layer-by-layer training, more layers can be trained, and finally all the training results are combined to obtain a learning model after training.

In this way, the data can be grouped by using the learning model of the stacked sparse autoencoder after training. Among them, the dimension of the input layer and output layer of the stacked sparse autoencoder The vector is the number of data strokes multiplied by the data field, and the encoder of the four-layer stacked sparse auto-encoder can be used as an unsupervised learning architecture. The output of each layer is connected to the input of successive layers to reduce the data dimension to three dimensions, and the decoder of each stacked sparse autoencoder is run in reverse order to restore the data dimension to the original dimension. In one example, the loss function used by the stacked sparse autoencoder may be MSE, the activation function may be Sigmoid, and the optimizer may be Adam. After the training of the stacked sparse autoencoder is completed, the data grouping method is to take out the output of the last hidden layer of the encoding end, which means that each data is mapped to a point in the three-dimensional space through the encoding end, and according to the X in the aforementioned three-dimensional space , Y and Z axes automatically divide the data into eight groups, and insert the corresponding quadrature value (1~8) into the last column of the original data table to form grouped data.

Specifically, the learning model has an encoding end that can be used to group big data. Specifically, the learning model is composed of an input layer, a hidden layer, and an output layer, and the input layer and the output layer have neurons respectively. ), which is equivalent to the dimension of the original data contained in the big data. In this embodiment, the learning model is established by using an autoencoder algorithm in deep learning. For example, when the raw data has 108 dimensions, eg, the raw data has 108 columns of data, the input layer and the output layer respectively have 108 neurons. On the other hand, in this embodiment, the number of the hidden layers can be set to seven, and the number of hidden layers is 128, 64, 32, 3, 32, 64, and 128 neurons in sequence, but not limited thereto. Among the seven hidden layers, the first four hidden layers can form the coding end of the learning model, and the coding end is used for compressing several input original data, and respectively generating a three-dimensional space coordinate. This type of neural network module 1 converts the respective three-dimensional space coordinates of the plurality of original data into a corresponding quadrant value in the eight quadrants according to the eight quadrants corresponding to the three-dimensional space coordinates, and converts the image The limit value is stored in the corresponding raw data to complete the grouping of the plurality of raw data. In other words, a hidden layer in the learning model has three neurons to convert the original data with several dimensions The data are projected into eight quadrants in the three-dimensional space, and a quadrant corresponding to each original data is added to the original data to form grouping data.

In an experiment, the results of each training 50 epochs (epochs) between the traditional auto-encoder (AE) and the stacked sparse auto-encoder (SSAE) of this embodiment are compared. Since the auto-encoder has no sparsity restriction, it is easy to perform clustering. Some data are mapped to the same area, resulting in uneven clustering and affecting the effect of clustering. For example, it is possible to have two datasets mapped to the same region entangled with no clear demarcation between clusters; however, stacked sparse autoencoders with sparse constraint functions can make the points of the same cluster more compact and The separation between clusters is more obvious to obtain better clustering effect.

The indexing unit 22 is based on an application based on the technology of a distributed search engine/system/architecture, especially the elastic search (Elasticsearch) technology. The technology and application of the distributed search engine are those in the technical field to which the present invention belongs Usually knowledgeable people can understand it, so I won't go into details. The indexing unit 22 receives the grouped data obtained by the grouping unit 21, and applies a distributed search engine to perform a distributed index on the grouped data. To be more specific, under the architecture of the distributed search engine, it includes a cluster (Cluster), the cluster includes a plurality of nodes, each node will join the cluster when it is started and elect a (node) node to become the master node (Master node). Node), index the data through distributed search, and then store the corresponding data in the Distributed File System (HDFS, Hadoop Distributed File System). In other words, the indexing unit 22 may include the distributed file system. In detail, when a complete data set needs to be indexed, if the current node is the primary node, the indexing can be started, otherwise, it is forwarded to the primary node for indexing. Among them, the distributed search engine based on elastic search technology will first determine whether the data has been indexed; if not, the distributed search engine will start to build the index and write the result into the Lucene index, and use the hash algorithm to ensure that the data will be indexed. The index data is evenly distributed and stored in the designated primary shard and replica shard, and a corresponding version number is created and stored in Translog. If the data has been indexed However, the distributed search engine will first compare the existing version numbers to check whether there is a conflict: if there is no conflict, the indexing can be started; if it is judged that there is a conflict, it will return an error result that the indexing failed. With the above technical structure, the multi-group grouping data (sets) processed by the grouping unit 21 can be uploaded to a distributed search engine based on elastic search technology for distributed indexing, so as to correspond to each of the grouped data The corresponding index value is generated by the quadrature value of , and the relevant data (corresponding index value) processed by the distributed search engine are stored in a distributed file system for a data retrieval work. Wherein, the data retrieval work can be a corresponding command or operation input by a user, or by any other means, such as the work implicit in the computer 1 performing a work task (preferably, it can be automatically generated/inputted) corresponding data retrieval work), or the computer 1 receives corresponding instructions for data retrieval work from other systems. In this way, the clustered indexing module 2 integrates the stacked sparse autoencoder and the distributed search engine based on elastic search technology to establish the SSAE-ES fast indexing method, which is compared with the prior art (especially as disclosed in the ROC Patent Publication No. I696082). : DAE-SOLR technology that groups data through Deep Autoencoder and Solr retrieval technology), which can improve the grouping effect and the average waiting time of job execution, thereby greatly increasing the query performance by about 40%.

The scheduling module 3 includes a time estimation module 31 and a sorting unit 32 . The time estimation module 31 has a time operation model based on Deep Neural Network (DNN, Deep Neural Network) technology for obtaining an estimated working time, and the estimated working time refers to a work (or an instruction ) is an estimate of the total execution time required. In detail, in the time operation model of applying deep neural network, the corresponding input layer has six dimension vectors, namely: data volume, data column number, data column number, program time complexity, used retrieval The engine environment and the remaining memory size of the system; the data of the corresponding output layer is a dimension used to predict the time spent performing data retrieval. The activation function used by the time operation model is ReLU, the loss function is MSE, and the optimizer is Adam, but not limited to this.

Preferably, the calculation of the time complexity of the aforesaid program is achieved by executing a time complexity calculation process with a designed analysis program. In the time complexity calculation process, it is first determined whether there is at least one loop instruction in the code of a data retrieval job, and the number of execution times TA ₁ of a loop is defined by the operator in the at least one loop instruction (if there is N, then it is TA ₁ ~TA _N ); then it is judged whether the function call part in the code of the data retrieval work has at least one recursive function of repeated calls, so as to define the execution times TB of a call call ₁ (if there are M, it is TB ₁ ~TB _m ); finally, consider the relationship between each loop instruction and/or each call letter to calculate a total execution times Ts to represent the sum of all execution times, And use this as the calculation index of time complexity. For example, in a relationship in which a loop instruction A1 includes a loop instruction A2, in a relationship in which a loop instruction A1 includes a call letter B1, a call letter B1 includes a call letter B2. In the relationship, the corresponding total execution times Ts are respectively A1 x A2, A1 x B1, and B1 x B2. In another illustrative example, for example, each loop instruction and/or each call letter is in a relationship not included in any other one, then the total execution times Ts is the sum of the respective times; for example, for example, a In the case that the loop command A1, a loop command A2, and a call notification B1 are not included in each other, the total execution times Ts is A1+A2+B1. In yet another illustrative example, a loop command A1 includes a call signal B1, the call signal B1 further includes a loop command A2, and another call signal B2 is not included in the loop In the case of instruction A1, the total execution times Ts is (A1 x B1 x A2)+A1. More preferably, the power among the total execution times Ts may be taken, and all its coefficients are ignored, and the maximum value of the power may be used as the calculation index of the time complexity. In the aforementioned exponentiation method, the corresponding exponentiation values can be, for example, 1, logn, n, nlogn, n ² , n ³ , 2 ⁿ , n ⁿ in sequence, and are given in sequence, for example, 0 to 7. Calculate metrics. Among them, calculating the time complexity through the above exponentiation can moderately simplify the time complexity calculation, achieve the improvement of the calculation efficiency, and at the same time still have the performance of reflecting the execution time of a program.

With the time operation model of the time estimation module 31, there are multiple data checks. In a state in which the job is to be processed, a plurality of corresponding estimated working hours can be obtained; the sorting unit 32 receives the plurality of estimated working hours, and determines a plurality of corresponding estimated working hours according to the size of the plurality of estimated working hours The order in which data retrieval work is performed. Preferably, the sorting unit 32 prioritizes the one whose estimated working time is smaller.

In another example, in a state in which multiple data retrieval jobs are pending (ie, when multiple data retrieval jobs are queued to form a pending state), the sorting unit 32 first considers executing each data retrieval job The priority of the job is the first filter condition, and then the estimated working time of each data retrieval job is regarded as the second filter condition, so as to determine the execution order of several data retrieval jobs, so as to improve the work processing of the overall system per unit time. quantity. The priority may be a predefined value or a value set in a predefined manner in each data retrieval work; preferably, the priority may be at least divided into a high priority and a low priority. In the first filter condition, the material retrieval work with high priority will be executed in priority to the material retrieval work with low priority. In the second filter bar, when at least two of the multiple data retrieval jobs have the same priority, the one with the smaller estimated working time is prioritized. Wherein, if the two data retrieval jobs have the same priority and the same estimated working time, the job sequence is scheduled according to the FIFO scheduling mechanism.

In this way, the scheduling module 3 combines the deep neural network (DNN) to predict the time required for each job to execute, and then schedules according to the shortest work time first (SJF, Sortest-Job-First), which is collectively referred to as DNNSJF scheduling. Compared with prior art such as FIFO scheduling mechanism, DNNSJF scheduling can effectively reduce the average waiting time of job execution by about 3~5%; in addition, compared with prior art such as MSHEFT (Memory-Sensitive Heterogeneous Early Finish Time) scheduling In terms of mechanism, DNNSJF scheduling can effectively reduce the average waiting time of job execution by about 1~3%. Among them, based on HEFT (Heterogeneous Early Finish Time), each work will be assigned to the appropriate CPU according to the sorted priority list, so as to complete the work as soon as possible. The MSHEFT mechanism includes the following steps: first consider the work priority, and then based on the qualifications of the work Material size is regarded as the second filter condition.

The search engine selection module 4 includes Hive, Impala and SparkSQL engines/interfaces/platforms that support Structured Query Language (SQL) commands, and has a search engine selection logic for selecting an appropriate search engine according to the current available memory size engine. The search engine selection logic selects the Hive engine, the Impala engine and the SparkSQL engine respectively according to a percentage value of available memory between the remaining memory and the total memory of the computer from small to large. Preferably, when the available memory percentage value is less than 25%, the Hive engine is selected; when the available memory percentage value is not less than 25% and not greater than 50%, the Impala engine is selected; when the available memory percentage value is greater than 50%, choose the SparkSQL engine. Wherein, after an actual measurement of critical data is performed by the present invention, the size of the above-mentioned percentage value is particularly suitable for the state where the memory 12 of the computer 1 is 20 GB. However, the aforementioned threshold can be appropriately adjusted according to the hardware performance and actual usage requirements, and is not limited thereto.

Accordingly, according to the system of the present invention, in an example, when a data retrieval job/command is input through the computer 1, the data retrieval job will first pass through the scheduling module 3 (according to the estimated working time or At the same time, the priority and the estimated working time are considered in order to arrange the corresponding job execution order, and when the data retrieval job is executed, the retrieval engine selection module 4 selects the corresponding job according to a relationship between the currently available memory percentage values. A search engine is used to query whether there is corresponding data from the grouping index module 2 . In this way, the processor module can select an appropriate retrieval engine according to the size of the available memory percentage value, so as to exert the best performance of the retrieval engine, which has the effect of improving the overall performance of the system.

In another example, when a data retrieval job is input through the computer 1 , the data retrieval job will firstly inquire whether there is corresponding data from the memory unit 1 through the processing unit 11 . If the corresponding data is hit from the memory unit 12, the processing unit 11 reads the corresponding data from the memory unit 12 and writes it to an output file; if the memory unit 12 misses the corresponding data For corresponding data, the processing unit 11 inquires from the hard disk unit 13 whether there is corresponding data. If the corresponding data is hit from the hard disk unit 13, the processing unit 11 reads the corresponding data from the hard disk unit 13 and writes the corresponding data into an output file; if the corresponding data is not hit from the hard disk unit 13, the corresponding data The data retrieval work will arrange the corresponding work execution order through the scheduling module 3, and when the data retrieval work is executed, the corresponding retrieval engine will be selected through the retrieval engine selection module 4 to be selected from the grouping index module 2. Check to see if there is any corresponding information. In this way, through the two-layer cache mechanism formed by the memory unit 12 and the disk unit 13, excessive hardware resource consumption caused by repeated retrieval is saved, and the retrieval system resources are saved.

Please refer to FIG. 2 , according to the aforementioned system, the big data retrieval method of the present invention executes all the following steps through a computer: a sparse model building step S1 : stacking after layer-by-layer training through a plurality of sparse auto-encoders A neural network consisting of a stack of sparse autoencoders is constructed to build a learning model.

A data grouping step S2: Inputting a plurality of original data with multiple dimensions into the learning model, and projecting the plurality of original data into the eight quadrants of the three-dimensional space through the learning model, and assigning a corresponding one of the original data to the learning model Quadrature values are appended to this raw data to form cluster data.

A data index decentralization step S3: based on elastic search technology and using a distributed search engine, the grouped data are distributed indexed to generate corresponding index values corresponding to the quadratic values of each of the grouped data; The corresponding index value is stored in a distributed file system for a user to perform a data retrieval job.

A time operation model establishment and estimation step S4: a time operation model is pre-established based on the deep neural network, and the time operation model is based on the amount of data, the number of data columns, the number of data columns, and the program time complexity in the data retrieval work. , Use the search engine environment and the remaining memory size of the system, and predict an estimated working time spent performing the data search.

In an example, a job scheduling step S5: in a state in which there are a plurality of data retrieval jobs to be processed, an estimated working time corresponding to each of the data retrieval jobs is obtained by the time operation model, and according to the The size sequence of the plurality of estimated work times determines the execution sequence of the corresponding plurality of data retrieval jobs.

In another example, a job scheduling step S5 ′: in a state where there are a plurality of data retrieval jobs to be processed, obtain an estimated working time corresponding to each of the data retrieval jobs by using the time operation model, and The execution order of the plurality of corresponding data retrieval jobs is determined according to the plurality of priorities and the plurality of estimated working times corresponding to the plurality of data retrieval jobs respectively.

A retrieval engine selection step S6 : selecting an appropriate retrieval engine according to the current available memory size when performing the data retrieval job. Specifically, the retrieval engine is selected according to a percentage of available memory between the remaining memory of the memory unit 12 of the computer 1 and the total memory. Preferably, when the available memory percentage value is less than 25%, the Hive engine is selected; when the available memory percentage value is not less than 25% and not greater than 50%, the Impala engine is selected; when the available memory percentage value is greater than 50%, choose the SparkSQL engine. More preferably, the memory unit has a capacity of 20GB.

In an example, a data retrieval execution step S7: when the computer 1 receives a data retrieval job, the data retrieval job will first predict an estimated working time spent performing the data retrieval job through the time operation model; The estimated working time or both the priority and the estimated working time are considered to arrange the corresponding job execution sequence; and when the data retrieval job is executed, the corresponding retrieval engine is selected according to a relationship between the currently available memory percentage value, To search whether there is corresponding data from the decentralized file system.

In another example, a data retrieval executes step S7': when the computer 1 receives a data retrieval job, the memory unit 12 of the computer 1 is first inquired about whether there is corresponding data. If the corresponding data is hit from the memory unit 12 , the computer 1 reads the pair from the memory unit 12 The data is applied and written to an output file; if the corresponding data is not hit from the memory unit 12 , the computer 1 inquires from the hard disk unit 13 whether there is corresponding data. If the corresponding data is hit from the hard disk unit 13, the computer 1 reads the corresponding data from the hard disk unit 13 and writes the corresponding data into an output file; if the corresponding data is not hit from the hard disk unit 13, the data The retrieval job will first predict an estimated working time spent on executing the data retrieval job through the time operation model; arrange the corresponding job execution sequence according to the estimated working time or considering the priority and the estimated working time at the same time; And when the data retrieval work is performed, a corresponding retrieval engine is selected according to a relationship between the currently available memory percentage values, so as to retrieve whether there is corresponding data from the distributed file system.

To sum up, the big data retrieval method and system of the present invention can effectively improve the grouping effect and establish a fast index through the grouping index module, which can reduce the search range of the database. In addition, through the scheduling module, the estimated working hours can be obtained, and jobs with shorter working hours can be given higher priority scheduling, thereby shortening the average waiting time of the overall job. In addition, an appropriate retrieval engine is selected according to the size of the currently available memory, so as to select an appropriate retrieval engine to execute the scheduling work, so as to achieve the effect of optimizing the execution efficiency of the system. In addition, the results of data retrieval are stored by the memory unit or hard disk unit cache mechanism. If the same data needs to be retrieved repeatedly in a short period of time, the data can be directly searched and read in the cache storage, and there is no need to re-use the retrieval engine for retrieval. It can greatly save execution time and reduce consumption of hardware resources.

Although the present invention has been disclosed by the above-mentioned preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make various changes and modifications relative to the above-mentioned embodiments without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall include all changes within the meaning and equivalent scope recorded in the appended patent application scope.

S1: Steps to build a sparsity model

S2: Data grouping step

S3: Data index decentralization step

S4: Time operation model establishment and estimation steps

S5, S5': work scheduling steps

S6: Search engine selection step

S7, S7': data retrieval execution steps

Claims (10)

A method for retrieving big data, which executes the following steps through a computer, comprising: building a learning model through a neural network of stacked sparse auto-encoders formed by stacking a plurality of sparse auto-encoders after layer-by-layer training; A plurality of raw data of multiple dimensions are input into the learning model, and the plurality of raw data are projected into the eight quadrants of the three-dimensional space through the learning model, and a quadrant corresponding to each raw data is added to the raw data , to form grouped data; based on elastic search technology and using a distributed search engine, the grouped data will be indexed in a distributed manner to generate corresponding index values corresponding to the quadrature values of each of the grouped data; The index value is stored in a distributed file system for a data retrieval work; a time operation model is pre-established based on the deep neural network, and the time operation model is based on the amount of data and the number of data columns in the data retrieval work. , the number of data columns, the program time complexity, the search engine environment used, and the remaining memory size of the system to obtain an estimated work time to predict the time spent performing the data search; In a state of processing, the estimated working time corresponding to each data retrieval job is obtained by the time operation model, and the execution order of the corresponding plurality of data retrieval jobs is determined according to the magnitude order of the plurality of estimated working times. The method of claim 1, wherein the execution order of the plurality of data retrieval jobs is determined according to a plurality of priorities corresponding to the plurality of data retrieval jobs and the plurality of estimated working times; each of the priorities is A value that is predefined or set in a predefined manner in each data retrieval job. The method of claim 1 or 2, wherein the time complexity is calculated according to a relationship between loop instructions and/or call letters included in the data retrieval job to calculate a total execution time The number of lines is used as a calculation index of the time complexity. The method of claim 2, wherein, when performing the data retrieval job, a retrieval engine is selected according to a percentage value of available memory between the remaining memory of a memory unit of the computer and the total memory; when the available memory is When the percentage value is less than 25%, the Hive engine is selected; when the free memory percentage value is not less than 25% and not greater than 50%, the Impala engine is selected; when the free memory percentage value is greater than 50%, the SparkSQL engine is selected; , the memory unit has a capacity of 20GB. The method of claim 4, wherein when the computer receives a data retrieval job, it first inquires whether there is a corresponding data from a memory unit of the computer. If the corresponding data is hit from the memory unit, the computer retrieves the data from the memory unit. The body unit reads the corresponding data and writes it into an output file; if the corresponding data is not hit from the memory unit, the computer inquires from the hard disk unit whether there is the corresponding data; if it hits the corresponding data from the hard disk unit Corresponding data, the computer reads the corresponding data from the hard disk unit and writes it into the output file; if the corresponding data is not hit from the hard disk unit, the data retrieval work will first predict a corresponding data through the time operation model Estimate the working time; consider the priority and the estimated working time at the same time, and arrange the corresponding work execution sequence; Retrieve whether the corresponding data exists in the distributed file system. A big data retrieval system, comprising: a computer with a processing unit, a memory unit and a hard disk unit, the processing unit has a processor, the memory unit and the hard disk unit are coupled to the processing unit; a The grouping index module has a grouping unit and an indexing unit coupled to the grouping unit; the grouping unit is formed by stacking a neural network of a stacked sparse auto-encoder after a plurality of sparse auto-encoders are trained layer by layer to establish a Learning model; through the learning model, multiple original data of multiple dimensions are projected into the eight quadrants of the three-dimensional space, and a quadrant corresponding to each original data is added to the raw data to form grouping data; the indexing unit receives the grouping data obtained by the grouping unit, and applies a distributed search engine based on elastic search to perform a distributed index on the grouping data to correspond to each of the grouping data The corresponding index values are generated from the quadratic value of the user; the corresponding index values are then stored in a distributed file system for a data retrieval work; and a scheduling module has a time estimation module and a sorting unit; the time estimation module has a time operation model pre-established based on the deep neural network, and the time operation model is based on the amount of data, the number of data columns, the number of data columns, and the complexity of the program time in the data retrieval work degree, the search engine environment used, and the remaining memory size of the system to obtain an estimated work time to predict the time spent performing the data search work; in a state where multiple data search work is pending, the sorting The unit receives a plurality of corresponding estimated working hours, and determines the execution order of the corresponding plurality of data retrieval jobs according to the magnitude order of the plurality of estimated working hours. The system of claim 6, wherein the time complexity is calculated based on a relationship between loop instructions and/or call letters included in the data retrieval job to calculate a total number of executions as the time complexity A calculation indicator. The system of claim 6, wherein the time complexity is based on a relationship between loop instructions and/or call letters included in the data retrieval job to calculate a total execution times, and obtain the total execution times The middle power is ignored, and all its coefficients are ignored, and the maximum value of the power is used as the calculation index of the time complexity. The system of any one of claims 6 to 8 further comprises a retrieval engine selection module, the retrieval engine selection module has Hive, Impala and SparkSQL engines supporting structured query language commands, and the retrieval engine selection module It has a retrieval engine selection logic; the retrieval engine selection logic is defined as: when performing the data retrieval work, select retrieval according to a percentage value of available memory between the remaining memory of a memory unit of the computer and the total memory engine; when the available memory percentage value is less than 25%, select the Hive engine; when the available memory percentage is less than 25% When the percentage value is not less than 25% and not greater than 50%, the Impala engine is selected; when the available memory percentage value is greater than 50%, the SparkSQL engine is selected; the capacity of the memory unit is 20GB. The system of claim 9, wherein when the computer receives a data retrieval job, it first inquires whether there is a corresponding data from a memory unit of the computer, and if the corresponding data is hit from the memory unit, the computer retrieves the data from the memory unit. The body unit reads the corresponding data and writes it into an output file; if the corresponding data is not hit from the memory unit, the computer inquires from the hard disk unit whether there is the corresponding data; if it hits the corresponding data from the hard disk unit Corresponding data, the computer reads the corresponding data from the hard disk unit and writes it into the output file; if the corresponding data is not hit from the hard disk unit, the data retrieval work will first predict a corresponding data through the time operation model Estimate the working time; consider the priority and the estimated working time at the same time, and arrange the corresponding work execution sequence; Retrieve whether the corresponding data exists in the distributed file system.

Images