CN113535677A

CN113535677A - Data analysis query management method and device, computer equipment and storage medium

Info

Publication number: CN113535677A
Application number: CN202110724031.1A
Authority: CN
Inventors: 陈鹏; 付成龙; 高金星
Original assignee: Ping An International Financial Leasing Co Ltd
Current assignee: Ping An International Financial Leasing Co Ltd
Priority date: 2021-06-29
Filing date: 2021-06-29
Publication date: 2021-10-22
Anticipated expiration: 2041-06-29
Also published as: CN113535677B

Abstract

The invention relates to the technical field of big data, and discloses a data analysis query management method, a device, computer equipment and a storage medium, wherein the method comprises the following steps: integrating at least one authority registry into an authority library; determining a target library in a database according to the analysis request, and acquiring authority information of the user side in the target library; acquiring target data from a target library according to the analysis request based on the authority information, and performing analysis operation on the target data to obtain an analysis result; determining authority information of the target library and the user side in the target library according to the data query request; and acquiring query data from the target library according to the data query request based on the authority information. The invention also relates to a blockchain technique, where information can be stored in blockchain nodes. The invention avoids the problems of slow business data acquisition rate and low efficiency caused by frequent access to a plurality of databases, realizes the technical effect of automatically extracting and analyzing the business data of a plurality of data sources, and improves the data analysis efficiency.

Description

Data analysis query management method and device, computer equipment and storage medium

Technical Field

The invention relates to the technical field of data processing of big data, in particular to a data analysis query management method and device, computer equipment and a storage medium.

Background

When a user side analyzes and queries data, the user side generally needs to access a plurality of databases to obtain complete and accurate data and analyze the data; however, the rules and types of the databases are different, so that when a user needs to acquire service data from the databases, the user needs to frequently access the databases and call different data extraction methods to extract the service data from the databases, which results in slow acquisition rate of the service data and low efficiency. Moreover, the current multi-data source data analysis usually requires manual intervention to extract and process data, which is not only slow in efficiency, but also low in accuracy of analysis.

Disclosure of Invention

The invention aims to provide a data analysis query management method, a data analysis query management device, computer equipment and a storage medium, which are used for solving the problems that in the prior art, due to different rules and types of databases, a user side needs to frequently access the databases and needs to call different data extraction methods to extract service data, so that the service data acquisition rate is slow, the efficiency is low, and the current multi-data-source data analysis is slow in efficiency and low in analysis accuracy due to manual extraction and processing.

In order to achieve the above object, the present invention provides a data analysis query management method, which includes:

acquiring at least one authority registry of a database with service data, and integrating at least one authority registry into an authority library; the authority registry defines the authority level for acquiring the service data and records authority information reflecting the authority level of the user side;

receiving an analysis request sent by a user side, determining a target library in the database according to the analysis request, and acquiring authority information of the user side in the target library; acquiring target data from the target library according to the analysis request based on the authority information, performing analysis operation on the target data to obtain an analysis result, and sending the analysis result to the user side;

receiving a data query request sent by a user side, and determining a target library and authority information of the user side in the target library according to the data query request; and acquiring query data from the target library according to the data query request based on the authority information, and sending the query data to the user side.

In the above scheme, the step of acquiring at least one authority registry of a database having service data and integrating the at least one authority registry into an authority library includes:

constructing an acquisition component, and establishing communication connection between the acquisition component and at least one database;

acquiring service data and a permission registry from at least one database through the acquisition component;

extracting the source name and the type name of the authority registry, constructing an authority index according to the source name and the type name, and associating the authority index with at least one authority registry to form an authority library; the source name is the name of the database to which the authority registry belongs, and the type name is the name of the type of the service data in the database of the authority registry.

In the foregoing solution, the step of obtaining target data from the target library according to the analysis request based on the permission information includes:

identifying the service data corresponding to the authority information in the target library, setting the service data as comparison data, extracting access information in the analysis request, and judging whether all the access information is the comparison data;

if all the access information is the comparison data, acquiring target data from the target library according to the access information;

if part of the access information is the comparison data, setting the access information belonging to the comparison data as an access target, and acquiring target data from the target library according to the access target;

and if all the access information is not the comparison data, sending unauthorized analysis information to the user side.

In the foregoing solution, the step of performing analysis operation on the target data to obtain an analysis result, and sending the analysis result to the user side includes:

extracting an operation rule in the analysis request, and calling a preset input cache component to cache the target data;

calling a preset analysis engine to run the operation rule, and analyzing and operating the target data to obtain a request analysis result;

and inputting the request analysis result into a preset output queue component, and sending the request analysis result to the user side through the output queue component.

In the above solution, after integrating at least one of the rights registries into a rights repository, the method further includes:

and carrying out batch processing on the service data in the database to obtain an analysis result, determining authority information of the analysis result through the authority library, and storing the analysis result into a preset data center.

In the above solution, the step of obtaining query data from the target repository according to the data query request based on the permission information and sending the query data to the user side includes:

identifying the business data corresponding to the authority information in the target library and setting the business data as comparison data; extracting access information in the query request, and judging whether all the access information is the comparison data;

if all the access information is the comparison data, acquiring query data from the target library according to the access information, and sending the query data to the user side;

if part of the access information is the comparison data, setting the access information belonging to the comparison data as an access target, acquiring query data from the target library according to the access target, and sending the query data to the user side;

and if all the access information is not the comparison data, sending the unauthorized query information to the user side.

In the foregoing solution, after the storing the analysis result in a preset data center, the method further includes:

receiving a result query request sent by a user side, and determining authority information of the user side through the authority library; based on the authority information, obtaining an analysis result from the data center according to the result query request, and sending the analysis result to the user side;

after the analysis result is sent to the user side, the method further includes:

uploading the analysis result to a blockchain.

In order to achieve the above object, the present invention further provides a data analysis query management apparatus, including:

the authority integration module is used for acquiring at least one authority registry of a database with service data and integrating the at least one authority registry into an authority library; the authority registry defines the authority level for acquiring the service data and records authority information reflecting the authority level of the user side;

the data analysis module is used for receiving an analysis request sent by a user side, determining a target library in the database according to the analysis request and acquiring authority information of the user side in the target library; acquiring target data from the target library according to the analysis request based on the authority information, performing analysis operation on the target data to obtain an analysis result, and sending the analysis result to the user side;

the data query module is used for receiving a data query request sent by a user side and determining a target library and authority information of the user side in the target library according to the data query request; and acquiring query data from the target library according to the data query request based on the authority information, and sending the query data to the user side.

In order to achieve the above object, the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor of the computer device implements the steps of the data analysis query management method when executing the computer program.

In order to achieve the above object, the present invention further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of the data analysis query management method.

According to the data analysis query management method, the data analysis query management device, the computer equipment and the storage medium, at least one authority registry of at least one database with service data is obtained and integrated into the authority library, so that a user can access a plurality of databases based on authority only through one authority library, the plurality of databases are integrated, the user can obtain the service data from the plurality of databases only by uniformly accessing the user, and the problems that the service data obtaining speed is slow and the efficiency is low due to the fact that the user needs to frequently access the plurality of databases and call different data extraction methods to extract the service data from the databases when the user needs to obtain the service data from the plurality of databases due to the fact that rules and types of the databases are different at present are solved.

Determining a target library in the database according to the analysis request, and acquiring authority information of the user side in the target library to ensure acquisition authority and analysis authority of the analysis request of the user side on the service data, and avoiding data leakage; the target data are obtained from the target library through the analysis request, and the analysis result is obtained by analyzing and calculating the target data, so that the technical effect of automatically extracting and analyzing the service data of multiple data sources is realized, the data are extracted and processed without manual intervention, and the data analysis efficiency, and the accuracy and the comprehensiveness of the data analysis are improved. And based on the authority information, the safety of the service data of each database is ensured by a mode of acquiring the query data from the target database according to the data query request.

Drawings

FIG. 1 is a flowchart of a first embodiment of a data analysis query management method according to the present invention;

FIG. 2 is a schematic diagram of an environmental application of a data analysis query management method according to a second embodiment of the data analysis query management method of the present invention;

FIG. 3 is a flowchart of a data analysis query management method according to a second embodiment of the data analysis query management method of the present invention;

FIG. 4 is a schematic diagram of program modules of a third embodiment of a data analysis query management apparatus according to the present invention;

fig. 5 is a schematic diagram of a hardware structure of a computer device according to a fourth embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The invention provides a data analysis query management method, a data analysis query management device, computer equipment and a storage medium, which are suitable for the technical field of data processing of big data and provide a data analysis query management method based on an authority integration module, a data analysis module and a data query module. The method comprises the steps of integrating at least one authority registry of a database with service data into an authority library by acquiring the authority registry of the database with the service data; receiving an analysis request sent by a user side, determining a target library in a database according to the analysis request, and acquiring authority information of the user side in the target library; acquiring target data from a target library according to the analysis request based on the authority information, and performing analysis operation on the target data to obtain an analysis result; receiving a data query request sent by a user side, and determining a target library and authority information of the user side in the target library according to the data query request; and acquiring query data from the target library according to the data query request based on the authority information.

The first embodiment is as follows:

referring to fig. 1, a data analysis query management method of the present embodiment includes:

s101: acquiring at least one authority registry of a database with service data, and integrating at least one authority registry into an authority library; the authority registry defines the authority level for acquiring the service data and records the authority information reflecting the authority level of the user side.

S102: receiving an analysis request sent by a user side, determining a target library in the database according to the analysis request, and acquiring authority information of the user side in the target library; and acquiring target data from the target library according to the analysis request based on the authority information, performing analysis operation on the target data to obtain an analysis result, and sending the analysis result to the user side.

S104: receiving a data query request sent by a user side, and determining a target library and authority information of the user side in the target library according to the data query request; and acquiring query data from the target library according to the data query request based on the authority information, and sending the query data to the user side.

In an exemplary embodiment, at least one authority registry of at least one database with service data is obtained and integrated into an authority database, so that a user can access a plurality of databases based on authority by only one authority database, and the plurality of databases are integrated, so that the user can obtain the service data from the plurality of databases only by uniformly accessing the user, and the problems of slow service data obtaining rate and low efficiency caused by frequent access to the plurality of databases and the need of calling different data extraction methods to extract the service data from the databases when the user needs to obtain the service data from the plurality of databases due to different rules and types of the databases are solved.

Determining a target library in the database according to the analysis request, and acquiring authority information of the user side in the target library to ensure acquisition authority and analysis authority of the analysis request of the user side on the service data, and avoiding data leakage; the target data are obtained from the target library through the analysis request, and the analysis result is obtained by analyzing and calculating the target data, so that the technical effect of automatically extracting and analyzing the service data of multiple data sources is realized, the data are extracted and processed without manual intervention, and the data analysis efficiency, and the accuracy and the comprehensiveness of the data analysis are improved.

Determining authority information of a target library and the user side in the target library according to the data query request; and based on the authority information, the safety of the service data of each database is ensured by the mode of acquiring the query data from the target database according to the data query request.

Example two:

the embodiment is a specific application scenario of the first embodiment, and the method provided by the present invention can be more clearly and specifically explained through the embodiment.

Next, the method provided in this embodiment will be specifically described by taking an example in which, in a server running a data analysis query management method, information synchronization and password return identification are performed on a cache and a database in the server, target data is obtained from a target library according to an analysis request based on authority information, an analysis operation is performed on the target data to obtain an analysis result, and query data is obtained from the target library according to a data query request. It should be noted that the present embodiment is only exemplary, and does not limit the protection scope of the embodiments of the present invention.

Fig. 2 schematically shows an environment application diagram of the data analysis query management method according to the second embodiment of the present application.

In an exemplary embodiment, the server 2 where the data analysis query management method is located is respectively connected with the database 3 and the user terminal 4 through a network; the server 2 may provide services through one or more networks, which may include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and/or the like. The network may include physical links, such as coaxial cable links, twisted pair cable links, fiber optic links, combinations thereof, and/or the like. The network may include wireless links, such as cellular links, satellite links, Wi-Fi links, and/or the like; the user terminal 4 may be a computer device such as a smart phone, a tablet computer, a notebook computer, and a desktop computer.

Fig. 3 is a flowchart of a specific method of a data analysis query management method according to an embodiment of the present invention, where the method specifically includes steps S201 to S205.

S201: acquiring at least one authority registry of a database with service data, and integrating at least one authority registry into an authority library; the authority registry defines the authority level for acquiring the service data and records the authority information reflecting the authority level of the user side.

The method and the device have the advantages that the user end can acquire the service data from the databases only by performing uniform access for integrating the databases, and the problems that the user end needs to frequently access the databases and needs to call different data extraction methods to extract the service data from the databases when the user end needs to acquire the service data from the databases due to different rules and types of the databases, so that the service data acquisition rate is slow and the efficiency is low are solved.

In a preferred embodiment, the step of obtaining at least one authority registry with a database of business data and integrating at least one of the authority registries into an authority library includes:

s11: constructing an acquisition component, and establishing communication connection between the acquisition component and at least one database;

in this step, the collection assembly includes:

and the offline synchronization tool is used for realizing data synchronization between the server running with the data acquisition, analysis and management method and the database, and further realizing stable and efficient data synchronization between heterogeneous data sources. The offline synchronization tool is DataX, and the DataX is a heterogeneous data source offline synchronization tool and is used for realizing a stable and efficient data synchronization function among various heterogeneous data sources such as a relational database (MySQL, Oracle and the like), an HDFS (Hadoop distributed file system), a Hive (Hive), a MaxCommute (original ODPS), an HBase (HBase), an FTP (File transfer protocol) and the like. The DataX is used as an offline data synchronization Framework and is constructed by adopting a Framework + plugin architecture. Data source reads and writes are abstracted into Reader/Writer plug-ins, incorporated into the overall synchronization framework.

The ETL tool is used for realizing the technical effects of extracting and converting the service data in the database and loading the processed service data into a server running a data acquisition, analysis and management method, wherein the ETL is an abbreviation of English Extract-Transform-Load and is used for describing the process of extracting (Extract), converting (Transform) and loading (Load) the data from a source end to a destination end. The ETL is an important ring for constructing a data warehouse, and a user extracts required data from a data source, and finally loads the data into the data warehouse according to a predefined data warehouse model after data cleaning. The ETL tool is a key, and the key is an open-source ETL tool and is called KDE Extraction, Transformation and Loading Environment.

The data transmission tool is used for transmitting business data in a relational database into a server running a data acquisition analysis management method, wherein the data transmission tool is Sqoop, the Sqoop is a tool used for transmitting data between Hadoop (hive) and a traditional database (MySQL, postgresql), and the data in one relational database (such as MySQL, Oracle, Postgres and the like) can be guided into an HDFS of Hadoop and the data of the HDFS can be guided into the relational database.

And the data collection tool is used for collecting the business data in the database, loading the business data to a server running the data acquisition analysis management method, and also used for performing customized collection and processing on the business data in the database. The data collection tool is a flash, the flash is a high-availability, high-reliability and distributed system for collecting, aggregating and transmitting massive logs, and the flash supports various data senders customized in the log system and used for collecting data; at the same time, flash provides the ability to simply process data and write to various data recipients.

S12: acquiring service data and a permission registry from at least one database through the acquisition component;

in this step, the authority registry defines the authority level for acquiring the service data, and records the authority information reflecting the authority level of the user.

S13: extracting the source name and the type name of the authority registry, constructing an authority index according to the source name and the type name, and associating the authority index with at least one authority registry to form an authority library; the source name is the name of the database to which the authority registry belongs, and the type name is the name of the type of the service data in the database of the authority registry.

In this step, a permission index is constructed by using the type name as a primary index and the source name as a secondary index, and the type name is used as the primary index and the source name is used as the secondary index, so that a permission registry of a database storing service data corresponding to the type is obtained according to the type of the service data required by the analysis request and the query request, and the permission identification operation of the analysis request and the query request is facilitated.

And associating the source name in the secondary index with the database through a hyperlink, so that a server running with a data acquisition, analysis and management method can access the corresponding authority registry only by calling the secondary index, and the acquisition and use efficiency of the authority registry is improved.

S202: receiving an analysis request sent by a user side, determining a target library in the database according to the analysis request, and acquiring authority information of the user side in the target library; and acquiring target data from the target library according to the analysis request based on the authority information, performing analysis operation on the target data to obtain an analysis result, and sending the analysis result to the user side.

In order to avoid a user side which is not authorized to acquire and analyze the business data and to acquire and analyze the business data to cause data leakage, the step determines a target library in the database according to the analysis request and acquires the authority information of the user side in the target library to ensure the acquisition authority and the analysis authority of the analysis request of the user side on the business data and avoid data leakage.

In order to realize the technical effects of acquiring service data from at least one database and comprehensively analyzing the service data to obtain data analysis based on multiple data sources, the step determines a target database in the database according to the analysis request and acquires the authority information of the user side in the target database so as to ensure the acquisition authority and the analysis authority of the analysis request of the user side on the service data and avoid data leakage; the target data are obtained from the target library through the analysis request, and the analysis result is obtained by analyzing and calculating the target data, so that the technical effect of automatically extracting and analyzing the service data of multiple data sources is realized, the data are extracted and processed without manual intervention, and the data analysis efficiency, and the accuracy and the comprehensiveness of the data analysis are improved.

In a preferred embodiment, the step of determining a target repository in the database according to the analysis request and acquiring the authority information of the user side in the target repository includes:

s21: extracting access information in the analysis request, identifying an authority registry corresponding to the access information from the authority library, and setting a database corresponding to the authority registry as a target library;

in this step, the analysis request includes access information, where the access information is description information of the service data required by the analysis request, and includes an access name and an access type, and the access type is used to represent a type of the service data corresponding to the access name.

Specifically, the step of identifying the authority registry with the access information from the authority library includes: extracting the data type of the access information; acquiring an authority index from the authority library, identifying a primary index corresponding to the data type, setting the primary index as a primary target index, and setting a secondary index under the primary target index as a secondary target index; and setting an authority registry associated with the secondary target index as an authority registry corresponding to the access information, and setting a database to which the authority registry belongs as the target library.

S22: and extracting the identity information in the analysis request, identifying the authority information corresponding to the identity information in the authority registry, and setting the authority information as the authority information of the user side in the target library.

In this step, identity information and authority information corresponding to the identity information are recorded in an authority registry of the target repository, the authority information corresponding to the identity information is acquired from the authority registry according to the identity information of the analysis request, the authority information reflects the authority level of the identity information, and the authority level represents the authority of the user side corresponding to the identity information for acquiring the service data in the target repository, so that the user side which does not have the authority to acquire the service data is prevented from acquiring unauthorized service data, the data security of the database is ensured, and the data leakage is prevented.

In a preferred embodiment, the step of obtaining target data from the target repository according to the analysis request based on the authority information includes:

s23: and identifying the service data corresponding to the authority information in the target library, setting the service data as comparison data, extracting access information in the analysis request, and judging whether all the access information is the comparison data.

In this step, in order to ensure that the acquired target data conforms to the authority of the user side, the service data corresponding to the authority level is acquired through the authority level corresponding to the authority information, and the service data is set as comparison data, so that the technical effect of locking the service data corresponding to the authority information is achieved, the subsequent operation of acquiring the target data does not exceed the range of the comparison data, and the technical effect of data locking is achieved. The access information refers to the name of the service data required to be acquired by the query request.

S24: if all the access information is the comparison data, acquiring target data from the target library according to the access information;

in this step, for the case that the service data required by the analysis request are all comparative data, which indicates that the requirements of the analysis request all conform to the authority level of the user side, at this time, only the target data needs to be acquired through the access information, and the target data is operated through the operation rule to obtain the analysis result.

S25: and if part of the access information is the comparison data, setting the access information belonging to the comparison data as an access target, and acquiring target data from the target library according to the access target.

In this step, for the case that part of the service data required in the analysis request is comparison data and part of the service data is not comparison data, and for considering both data security and reliability of the request return of the user side, the access information belonging to the comparison data is set as an access target, the target data is obtained from the target library according to the access target, the analysis operation is performed on the target data according to the operation rule to obtain an analysis result, and the analysis result is sent to the user side.

S26: and if all the access information is not the comparison data, sending unauthorized analysis information to the user side.

In this step, in the case that the service data required in the analysis request is not the comparison data, the data security is ensured by sending unauthorized analysis information to the user side.

In a preferred embodiment, the step of performing an analysis operation on the target data to obtain an analysis result, and sending the analysis result to the user side includes:

s27: and extracting the operation rule in the analysis request, and calling a preset input cache component to cache the target data.

In this step, the operation rule is a computer rule preset in the analysis request for performing analysis operation on the access information; the input cache component can be kafka, or kudu, or redis;

the kafka is a high-throughput distributed publish-subscribe messaging system that can handle all the action flow data of consumers in the website, unify online and offline message processing through the parallel loading mechanism of Hadoop, and also provide real-time messages through clustering.

The Kudu is a novel column-type storage system, is used for quickly analyzing quickly-changing data, fills the gap of the conventional Hadoop storage layer, and realizes faster data analysis on the basis of more timely updating;

the redis is a high-performance key-value database that supports relatively more stored value types, including string, list, set, and zset.

S28: and calling a preset analysis engine to run the operation rule so as to analyze and operate the target data to obtain a request analysis result.

In this step, the analysis engine comprises an Apache Flink component, a SparkStreaming component and an Impala component;

the target data input into the cache component is batched through the Apache Flink component, so that the analysis and operation efficiency of the target data is improved;

if the operation rule is that the target data is calculated to obtain the specified operation result, calling an Apache flight component and a spark streaming component, and carrying out batch real-time processing on the target data to obtain the analysis result.

If the operation rule is to query the specified data in the target data and collect the specified data to obtain a specified collection result, calling an Apache flight component and an Impala component, performing query processing on the specified data in the target data to obtain the specified data, and collecting the specified data to obtain the analysis result.

It should be noted that Apache Flink is an open source stream processing framework, and the core is a distributed stream data stream engine written in Java and Scala, which executes any stream data program in a data parallel and pipelined manner, and the pipelined runtime system of Flink can execute batch processing and stream processing programs.

SparkStreaming is an extension of a Spark core API, can realize high throughput, has a fault-tolerant mechanism, and supports real-time streaming data processing, and multiple data sources acquire data.

Impala is a new type of query system that provides SQL semantics for querying PB-level big data stored in Hadoop's HDFS and HBase.

S29: and inputting the request analysis result into a preset output queue component, and sending the request analysis result to the user side through the output queue component.

In this step, MQ is used as an output queue component, where the MQ is a message queue, which is a communication method of an application program to the application program. Applications communicate by writing and retrieving queued data (messages) for the application without requiring a dedicated connection to link them. Message passing refers to the communication between programs by sending data in a message, rather than communicating with each other through direct calls, which are often used for techniques such as remote procedure calls. Queuing refers to the application communicating through a queue. The use of queues removes the requirement that the receiving and sending applications execute simultaneously.

S203: and carrying out batch processing on the service data in the database to obtain an analysis result, determining authority information of the analysis result through the authority library, and storing the analysis result into a preset data center.

In the step, the analysis result is obtained by batch processing the service data in the database so as to realize deep mining of the service data.

In order to avoid data leakage caused by the fact that analysis results of deep mining are obtained by a user side without access authority, authority information of the analysis results is determined through the authority library, and the analysis results are stored in the data center.

In a preferred embodiment, the step of performing batch processing on the service data in the database to obtain an analysis result, determining authority information of the analysis result through the authority library, and storing the analysis result in a preset data center includes:

s31: and installing an offline analysis engine through a preset scheduling system.

In this step, the scheduling system is used to construct a computer operation engine and a data scheduling model based on the operation engine to construct a job flow, in this embodiment, Control-M is used as the scheduling system, Control-M is a cross-platform batch job scheduling management software, a C/S mode is used, an Enterprise Manager and a server are installed on the server, an agent (i.e., an offline analysis engine in this application) is installed on a controlled host, and the agent can submit a job flow defined by Control-M on the host and return an operation result. The Enterprise Manager can comprehensively monitor the running conditions of all batch jobs, and can control the running conditions and process intervention of the job in various ways.

Further, the installed offline analysis engine is a computer module formed by combining a hive component, an Apache Spark component and a Kylin component as the offline analysis engine; the Hive component is a data warehouse tool based on Hadoop and is used for realizing a data processing mechanism for storing, inquiring and analyzing data through data extraction, conversion and loading processing. The Hive component is also used for mapping the structured data file into a database table, providing an SQL query function and converting an SQL statement into a MapReduce task to be executed.

The Apache Spark component is a rapid and general computing engine designed for large-scale data processing, is a general parallel framework similar to Hadoop MapReduce, and is widely applied to data mining and machine learning.

The Kylin component is used for solving the data analysis requirement of TB level data in a big data system, provides an SQL (structured query language) query interface and multi-dimensional analysis (OLAP) capability above Hadoop/Spark to support super-large-scale data, and can query a huge Hive table in a sub-second. The core of the method is pre-calculation, and the calculation result is stored in HBase.

S32: and defining the workflow of the offline analysis engine through preset offline rules.

In this step, the job flow is a computer flow describing the starting and ending conditions of the job set formulated according to the offline rule, and the specific operation content of each job therein.

The offline rule includes: data consanguinity rules, data tagging rules, and metadata management rules,

the data blood relationship rule is a computer rule used for combing data blood relationship among business data, and the data blood relationship is a relationship similar to human social blood relationship formed among data in the processes of generation, processing, circulation to extinction.

The data label rule is a computer rule for performing classification labeling on the category to which the business data belongs, and the classification labeling is generally to select a label corresponding to the business data from given labels. For example: there may be many categories/labels of a business data that describes a natural person: adult, female, yellow-rooted, and long-rooted.

The metadata management rule is a computer rule used for extracting metadata in the service data and summarizing the metadata; the Metadata (Metadata), also called intermediary data and relay data, is data (data about data) describing data, and is mainly information describing data attribute (property), and is used to support functions such as indicating storage location, history data, resource search, file record, and the like.

S33: and acquiring the service data in the database according to the workflow through the offline analysis engine, and analyzing and calculating the service data to obtain an analysis result.

In this step, business data is obtained from a database through a hive component of an offline analysis engine, and a workflow corresponding to the data blood margin rule is executed through an Apache Spark component, so as to perform data blood margin mining operation on the business data and obtain an analysis result. And the Kylin component executes the workflow corresponding to the data tag rule and the metadata management rule and is used for carrying out classification labeling processing, metadata extraction and summary processing on the business data to obtain an analysis result.

S34: and acquiring the authority level of the service data, associating the authority information corresponding to the authority level with the analysis result, and storing the analysis result in a preset data center.

In this step, the authority level of the service data is obtained through the authority registry, the authority information reflecting the authority level of the user side is obtained according to the authority level, and the authority mapping table between the authority information and the analysis result is constructed, so that the association between the authority information and the analysis result is realized.

S204: receiving a data query request sent by a user side, and determining a target library and authority information of the user side in the target library according to the data query request; and acquiring query data from the target library according to the data query request based on the authority information, and sending the query data to the user side.

In order to ensure the safety of the service data of the database, the step determines the authority information of a target library and the user side in the target library according to the data query request; and based on the authority information, the safety of the service data of each database is ensured by the mode of acquiring the query data from the target database according to the data query request.

In a preferred embodiment, the step of determining the authority information of the target repository and the user side in the target repository according to the data query request includes:

s41: extracting access information in the query request, identifying an authority registry with the access information from the authority library, and setting a database corresponding to the authority registry as a target library;

in this step, the query request includes access information, where the access information is description information of service data required by the query request, and includes an access name and an access type, and the access type is used to represent a type of the service data corresponding to the access name.

S42: and extracting the identity information in the query request, identifying the authority information corresponding to the identity information in the authority registry, and setting the authority information as the authority information of the user side in the target library.

In this step, identity information and authority information corresponding to the identity information are recorded in an authority registry of the target repository, the authority information corresponding to the identity information is acquired from the authority registry according to the identity information of the query request, the authority information reflects the authority level of the identity information, and the authority level represents the authority of the user side corresponding to the identity information for acquiring the service data in the target repository, so that the user side which does not have the authority to acquire the service data is prevented from acquiring unauthorized service data, the data security of the database is ensured, and the data leakage is prevented.

In a preferred embodiment, the step of acquiring query data from the target repository according to the data query request based on the authority information and sending the query data to the user side includes:

s43: identifying the business data corresponding to the authority information in the target library and setting the business data as comparison data; and extracting access information in the query request, and judging whether all the access information is the comparison data.

In this step, in order to ensure that the acquired query data conforms to the authority of the user side, the service data corresponding to the authority level is acquired through the authority level corresponding to the authority information, and the service data is set as comparison data, so that the technical effect of locking the service data corresponding to the authority information is achieved, the subsequent operation of acquiring the query data does not exceed the range of the comparison data, and the technical effect of data locking is achieved. The access information refers to the name of the service data required to be acquired by the query request.

S44: and if all the access information is the comparison data, acquiring query data from the target library according to the access information, and sending the query data to the user side.

In this step, for the case that the service data required by the query request are all comparative data, which indicates that the requirements of the query request all conform to the authority level of the user side, at this time, only the target data needs to be acquired through the access information, and the target data is operated through the operation rule to obtain the query result.

S45: and if part of the access information is the comparison data, setting the access information belonging to the comparison data as an access target, acquiring query data from the target library according to the access target, and sending the query data to the user side.

In this step, for the case that part of the service data required in the query request is comparison data and part of the service data is not comparison data, and for considering both data security and reliability of the request return of the user side, the access information belonging to the comparison data is set as an access target, the target data is obtained from the target library according to the access target, the query operation is performed on the target data according to the operation rule to obtain a query result, and the query result is sent to the user side.

S46: and if all the access information is not the comparison data, sending the unauthorized query information to the user side.

In this step, in the case that the service data required in the query request is not the comparison data, the data security is ensured by sending the unauthorized query information to the user side.

S205: receiving a result query request sent by a user side, and determining authority information of the user side through the authority library; and acquiring an analysis result from the data center according to the result query request based on the authority information, and sending the analysis result to the user side.

In order to ensure the safety of the analysis result of the data center, the step determines the authority information of a target center and the user side in the target center according to the data query request; and based on the authority information, the safety of the analysis result of each data center is ensured according to the mode of acquiring the query result from the target center according to the data query request.

In a preferred embodiment, the step of determining the target center and the right information of the user terminal in the target center according to the data query request includes:

s51: and extracting access information in the query request, and acquiring an authority registry with the access information.

In this step, the query request includes access information, where the access information is description information of an analysis result required by the query request, and includes an access name and an access type, and the access type is used to characterize a type of the analysis result corresponding to the access name.

Specifically, the step of identifying the authority registry with the access information from the authority library includes: extracting the data type of the access information; acquiring an authority index from the authority library, identifying a primary index corresponding to the data type, setting the primary index as a primary target index, and setting a secondary index under the primary target index as a secondary target index; and setting an authority registry associated with the secondary target index as an authority registry corresponding to the access information, and setting a data center to which the authority registry belongs as the data center.

S52: and extracting the identity information in the query request, identifying the authority information corresponding to the identity information in the authority registry, and setting the authority information as the authority information of the user side in the data center.

In this step, identity information and authority information corresponding to the identity information are recorded in an authority registry of the data center, the authority information corresponding to the identity information is obtained from the authority registry according to the identity information of the query request, the authority information reflects the authority level of the identity information, and the authority level represents the authority of a user side corresponding to the identity information to obtain an analysis result in the data center, so that the user side which does not have the right to obtain the analysis result is prevented from obtaining an unauthorized analysis result, the data security of the data center is ensured, and the data leakage is prevented.

In a preferred embodiment, the step of obtaining a query result from the data center according to the data query request based on the permission information and sending the query result to the user side includes:

s53: identifying an analysis result corresponding to the authority information in the data center and setting the analysis result as comparison data; and extracting access information in the query request, and judging whether all the access information is the comparison data.

In this step, in order to ensure that the obtained query result conforms to the authority of the user side, the analysis result corresponding to the authority level is obtained through the authority level corresponding to the authority information, and the analysis result is set as comparison data, so that the technical effect of locking the analysis result corresponding to the authority information is achieved, the subsequent operation of obtaining the query result does not exceed the range of the comparison data, and the technical effect of data locking is achieved. The access information refers to the name of the analysis result required to be obtained by the query request.

S54: and if all the access information is the comparison data, acquiring a query result from the data center according to the access information, and sending the query result to the user side.

In this step, for the case that the analysis results required by the query request are all comparative data, which indicates that the requirements of the query request all conform to the authority level of the user side, at this time, only the target data needs to be obtained through the access information, and the target data is operated through the operation rule to obtain the query result.

S55: and if part of the access information is the comparison data, setting the access information belonging to the comparison data as an access target, acquiring a query result from the data center according to the access target, and sending the query result to the user side.

In this step, for the case that part of the analysis results required in the query request is comparison data and part of the analysis results is not comparison data, and for considering both data security and reliability of the request return of the user side, the access information belonging to the comparison data is set as an access target, the target data is obtained from the data center according to the access target, the query result is obtained by performing query operation on the target data according to the operation rule, and the query result is sent to the user side.

S56: and if all the access information is not the comparison data, sending the unauthorized query information to the user side.

In this step, under the condition that the analysis results required in the query request are not compared with the data, the data security is ensured by sending the unauthorized query information to the user side.

Preferably, after the analysis result is sent to the user side, the method further includes:

uploading the analysis result to a blockchain.

The corresponding digest information is obtained based on the analysis result, and specifically, the digest information is obtained by hashing the analysis result, for example, using the sha256s algorithm. Uploading summary information to the blockchain can ensure the safety and the fair transparency of the user. The user equipment may download the summary information from the blockchain to verify whether the analysis result is tampered. The blockchain referred to in this example is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.

Example three:

referring to fig. 4, a data analysis query management apparatus 1 of the present embodiment includes:

the authority integration module 11 is used for acquiring at least one authority registry of a database with service data and integrating the at least one authority registry into an authority library; the authority registry defines the authority level for acquiring the service data and records authority information reflecting the authority level of the user side;

the data analysis module 12 is configured to receive an analysis request sent by a user, determine a target library in the database according to the analysis request, and acquire authority information of the user in the target library; acquiring target data from the target library according to the analysis request based on the authority information, performing analysis operation on the target data to obtain an analysis result, and sending the analysis result to the user side;

the data query module 14 is configured to receive a data query request sent by a user, and determine a target repository and authority information of the user in the target repository according to the data query request; and acquiring query data from the target library according to the data query request based on the authority information, and sending the query data to the user side.

Optionally, the right integration module 11 further includes:

the component connecting unit 111 is used for constructing an acquisition component and establishing communication connection between the acquisition component and at least one database;

an information obtaining unit 112, configured to obtain service data and an authority registry from at least one database through the collection component;

the authority library construction unit 113 is configured to extract a source name and a type name of the authority registry, construct an authority index according to the source name and the type name, and associate the authority index with at least one authority registry to form an authority library; the source name is the name of the database to which the authority registry belongs, and the type name is the name of the type of the service data in the database of the authority registry.

Optionally, the data analysis module 12 further includes:

an analysis target identification unit 121, configured to extract access information in the analysis request, identify an authority registry corresponding to the access information from the authority library, and set a database corresponding to the authority registry as a target library;

an analysis permission identifying unit 122, configured to extract the identity information in the analysis request, identify permission information corresponding to the identity information in the permission registry, and set the permission information as permission information of the user terminal in the target repository.

An analysis comparing unit 123, configured to identify service data corresponding to the authority information in the target library, set the service data as comparison data, extract access information in the analysis request, and determine whether all the access information is the comparison data;

a comprehensive analysis output unit 124, configured to obtain target data from the target library according to the access information when all the access information is the comparison data;

a partial analysis output unit 125, configured to set, when part of the access information is the comparison data, access information belonging to the comparison data as an access target, and obtain target data from the target library according to the access target;

an unauthorized analysis output unit 126, configured to send unauthorized analysis information to the user side when all the access information is not the comparison data;

the rule and data processing unit 127 is configured to extract an operation rule in the analysis request, and call a preset input cache component to cache the target data;

the online data operation unit 128 is configured to invoke a preset analysis engine to run the operation rule, so as to perform analysis operation on the target data to obtain a request analysis result;

and the data output unit 129 is configured to record the request analysis result into a preset output queue component, and send the request analysis result to the user side through the output queue component.

Optionally, the data analysis query management apparatus 1 further includes:

and the batch processing module 13 is configured to perform batch processing on the service data in the database to obtain an analysis result, determine authority information of the analysis result through the authority library, and store the analysis result in a preset data center.

Optionally, the batch processing module 13 further includes:

an engine installation unit 131, configured to install an offline analysis engine through a preset scheduling system;

a job flow definition unit 132, configured to define a job flow of the offline analysis engine by preset offline rules;

an offline data operation unit 133, configured to obtain, by the offline analysis engine, service data in the database according to the workflow, and perform analysis operation on the service data to obtain an analysis result;

a result permission associating unit 134, configured to obtain a permission level of the service data, associate permission information corresponding to the permission level with the analysis result, and store the analysis result in a preset data center.

Optionally, the data query module 14 further includes:

a query target identification unit 141, configured to extract access information in the query request, identify an authority registry with the access information from the authority library, and set a database corresponding to the authority registry as a target library;

the query authority identifying unit 142 is configured to extract the identity information in the query request, identify the authority information corresponding to the identity information in the authority registry, and set the authority information as the authority information of the user side in the target library;

a query comparison unit 143, configured to identify service data in the target repository corresponding to the authority information and set the service data as comparison data; extracting access information in the query request, and judging whether all the access information is the comparison data;

a comprehensive query output unit 144, configured to, when all the access information is the comparison data, obtain query data from the target library according to the access information, and send the query data to the user side;

and the partial query output unit 145 is configured to set, when part of the access information is the comparison data, the access information belonging to the comparison data as an access target, acquire query data from the target library according to the access target, and send the query data to the user side.

An unauthorized query output unit 146, configured to send unauthorized query information to the user side when all the access information is not the comparison data.

Optionally, the data analysis query management apparatus 1 further includes:

the result query module 15 is configured to receive a result query request sent by a user side, and determine permission information of the user side through the permission library; and acquiring an analysis result from the data center according to the result query request based on the authority information, and sending the analysis result to the user side.

Optionally, the result query module 15 further includes:

a result permission obtaining unit 151, configured to extract access information in the query request, and obtain a permission registry with the access information;

a result permission identifying unit 152, configured to extract the identity information in the query request, identify permission information corresponding to the identity information in the permission registry, and set the permission information as permission information of the user terminal in the data center;

a result comparing unit 153 configured to identify an analysis result corresponding to the authority information in the data center and set the analysis result as comparison data; extracting access information in the query request, and judging whether all the access information is the comparison data;

and the comprehensive result output unit 154 is configured to, when all the access information is the comparison data, obtain a query result from the data center according to the access information, and send the query result to the user side.

And the partial result output unit 155 is configured to set, when part of the access information is the comparison data, the access information belonging to the comparison data as an access target, acquire a query result from the data center according to the access target, and send the query result to the user side.

An unauthorized result output unit 156, configured to send unauthorized query information to the user side when all the access information is not the comparison data.

The technical scheme is applied to the field of data processing of big data, and at least one authority registry of a database with business data is acquired and integrated into an authority library; receiving an analysis request sent by a user side, determining a target library in a database according to the analysis request, and acquiring authority information of the user side in the target library; acquiring target data from a target library according to the analysis request based on the authority information, and performing analysis operation on the target data to obtain an analysis result; receiving a data query request sent by a user side, and determining a target library and authority information of the user side in the target library according to the data query request; and based on the authority information, performing traversal query on the service data in the target library according to the data query request to obtain query data.

Example four:

in order to achieve the above object, the present invention further provides a computer device 5, where components of the data analysis query management apparatus in the third embodiment may be distributed in different computer devices, and the computer device 5 may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a rack server (including an independent server or a server cluster formed by multiple application servers) that executes a program, and the like. The computer device of the embodiment at least includes but is not limited to: a memory 51, a processor 52, which may be communicatively coupled to each other via a system bus, as shown in FIG. 5. It should be noted that fig. 5 only shows a computer device with components, but it should be understood that not all of the shown components are required to be implemented, and more or fewer components may be implemented instead.

In this embodiment, the memory 51 (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 51 may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory 51 may be an external storage device of a computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the computer device. Of course, the memory 51 may also include both internal and external storage devices of the computer device. In this embodiment, the memory 51 is generally used for storing an operating system and various application software installed in the computer device, for example, the program code of the data analysis query management apparatus in the third embodiment. Further, the memory 51 may also be used to temporarily store various types of data that have been output or are to be output.

Processor 52 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 52 is typically used to control the overall operation of the computer device. In this embodiment, the processor 52 is configured to run the program code stored in the memory 51 or process data, for example, run a data analysis query management apparatus, so as to implement the data analysis query management method of the first embodiment and the data analysis query management method of the second embodiment.

Example five:

to achieve the above objects, the present invention also provides a computer readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, which when executed by a processor 52, implements corresponding functions. The computer-readable storage medium of this embodiment is used for storing a computer program for implementing the data analysis query management method, and when executed by the processor 52, implements the data analysis query management method of the first embodiment and the second embodiment.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner.

The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims

1. A data analysis query management method is characterized by comprising the following steps:

2. The method for managing data analysis and query as claimed in claim 1, wherein the step of obtaining at least one authority registry of a database with service data and integrating at least one of the authority registries into an authority library comprises:

3. The data analysis query management method according to claim 1, wherein the step of obtaining target data from the target repository according to the analysis request based on the authority information comprises:

4. The data analysis, query and management method according to claim 1, wherein the step of performing analysis operation on the target data to obtain an analysis result and sending the analysis result to the user side includes:

5. The data analysis query management method of claim 1, wherein after integrating at least one of the rights registries into a rights repository, the method further comprises:

6. The data analysis, query and management method according to claim 1, wherein the step of obtaining query data from the target repository according to the data query request based on the authority information and sending the query data to the user side comprises:

7. The data analysis query management method according to claim 5, wherein after saving the analysis result to a preset data center, the method further comprises:

uploading the analysis result to a blockchain.

8. A data analysis query management apparatus, comprising:

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the data analysis query management method according to any one of claims 1 to 7 are implemented when the computer program is executed by the processor of the computer device.

10. A computer-readable storage medium, on which a computer program is stored, wherein the computer program stored in the computer-readable storage medium, when being executed by a processor, implements the steps of the data analysis query management method according to any one of claims 1 to 7.