TWI669668B - Data management device and data management method - Google Patents

Abstract

The invention provides a data management device and a data management method. The data management device includes a processor and a memory coupled to the processor. The processor obtains multiple data entities and imports multiple meta-data of the corresponding data entity; parses the data operation language to establish a data relationship of the data entity; and establishes a node relationship diagram of the data relationship according to the graph cluster algorithm, wherein the node relationship diagram Multiple nodes correspond to data entities; and export node relationship diagrams.

Description

Data management device and data management method

The present invention relates to a data management apparatus and a data management method, and more particularly to a data management apparatus and a data management method capable of clearly showing the relationship of data and the importance of data.

When dealing with a large amount of data, in order to understand the source of the data and carry out subsequent processing procedures, a Data Lineage is usually established to achieve the purpose of information transparency. At the same time, the data history can also be used as an impact analysis caused by error finding and partial data errors.

In a database, there are often thousands of Data Entities, and each data entity can have multiple data fields. The creation of a data history usually needs to be based on the relationship between the data entities described in the Data Manipulation Language (DML). However, it is often difficult to quickly understand the relationship between data entities from the established textual material history. Therefore, how to establish a data resume makes it possible for managers to clearly understand the importance of data relationships and materials is a goal that those skilled in the art should strive for.

The invention provides a data management device and a data management method, which can clearly display the data relationship and the importance of the data.

The invention provides a data management device, which comprises a processor and a memory. The memory is coupled to the processor and stores a plurality of data entities. The processor obtains a data entity and imports a plurality of metadata of the corresponding data entity; parses a Data Manipulation Language (DML) to establish a data relationship of the data entity; and establishes a data relationship according to the graph cluster algorithm A node relationship diagram in which a plurality of nodes of a node relationship diagram correspond to a data entity; and a node relationship diagram.

In an embodiment of the invention, the processor parses the data manipulation language to establish a data relationship of a plurality of hierarchical data entities, wherein the hierarchy includes a system hierarchy, a work hierarchy, a task hierarchy, an instruction hierarchy, a data table hierarchy, and a field. Level.

In an embodiment of the invention, the processor divides the node relationship graph into at least one cluster according to the graph clustering algorithm, each cluster includes a center point, and the center point is a node having the highest degree in each cluster.

In an embodiment of the invention, the node relationship graph is a field relationship diagram of a plurality of data fields of each data entity.

In an embodiment of the invention, the processor calculates an importance index of the data entity of the corresponding node according to the degree of the node of the node relationship graph.

The invention provides a data management method, comprising: obtaining a data entity and importing a plurality of meta-data of a corresponding data entity; analyzing a data operation language to establish a data relationship of the data entity; and establishing a node relationship of the data relationship according to the graph cluster algorithm A graph in which a plurality of nodes of a node relationship graph correspond to a data entity; and a graph of a node relationship.

In an embodiment of the present invention, the data management method further includes: parsing a data operation language to establish a data relationship of a plurality of hierarchical data entities, wherein the hierarchical level includes a system level, a working level, a task level, an instruction level, and a data table level. And the level of the field.

In an embodiment of the invention, the data management method further comprises: dividing the node relationship graph into at least one cluster according to the graph cluster algorithm, each cluster including a center point, and the center point has a maximum degree in each cluster Node.

In an embodiment of the invention, the node relationship graph is a field relationship diagram of a plurality of data fields of each data entity.

In an embodiment of the present invention, the data management method further includes: calculating an importance index of the data entity of the corresponding node according to the degree of the node of the node relationship graph.

Based on the above, the data management apparatus and the data management method of the present invention analyze the data operation language to establish a data relationship of the data entity and reuse the graph cluster algorithm to image the data relationship of the data entity, thereby displaying the data node. Information such as clusters and central points, and the importance index of the data entity can be obtained from the node relationship diagram of the data relationship.

The above described features and advantages of the invention will be apparent from the following description.

1 is a block diagram of a material management apparatus according to an embodiment of the present invention.

Referring to FIG. 1 , the data management device 100 of the present invention includes a processor 110 , a memory 120 , and a communication interface 130 . The data management device 100 can be one or more servers that receive user data through the communication interface 130 and store or temporarily store the user data in the memory 120. The data management device 100 can also process user data through the processor 110. The data management device 100 can also manage the user data through the processor 110 and clearly display the relationship between the data.

The processor 110 can be a central processing unit (CPU), or other programmable general purpose or special purpose microprocessor (Microprocessor), digital signal processor (DSP), programmable A controller, an Application Specific Integrated Circuit (ASIC) or other similar component or a combination of the above components.

The memory 120 can be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk. (Hard Disk Drive, HDD), Solid State Drive (SSD) or the like or a combination of the above.

In an embodiment, the communication interface 130 can support Global System for Mobile communication (GSM), Personal Handy-phone System (PHS), and Code Division Multiple Access (Code Division Multiple Access). CDMA) system, Wideband Code Division Multiple Access (WCDMA) system, Long Term Evolution (LTE) system, Worldwide interoperability for Microwave Access (WiMAX) system, Wireless Fidelity (Wireless Fidelity, Wi-Fi) system or wireless communication interface for signal transmission in systems such as Bluetooth. In another embodiment, the communication interface 130 can also be any wired communication interface capable of providing transmission or reception of data, such as a Universal Serial Bus (USB) interface or an inter-integrated bus (Inter-Integrated bus). , I2C bus) and other wired communication interfaces. This novel creation does not limit the type of communication interface 130.

[First Embodiment]

2 is a schematic diagram showing the relationship between a data manipulation language and a data entity according to an embodiment of the invention. FIG. 3 is a schematic diagram showing the relationship of a data table according to an embodiment of the present invention. 4 is a schematic diagram showing the relationship between a data table and a processing program according to an embodiment of the invention.

Referring to FIG. 2 to FIG. 4, in the embodiment, the data operation language may correspond to a connection relationship between a data entity (for example, the data entity 200) and other data entities. The data entity 200 can include a data table 210 and a data field 220. When the number of data entities is large, the data management device 100 will make a data table relationship diagram for the data table (for example, the data table 310 and other data tables) to describe the target relationship and source relationship of the data table. In the present embodiment, the material management apparatus 100 also generates a relationship table between the data table (for example, the data table 410, which is represented by a rectangle in the figure) and the processing program (for example, the processing program 430, which is represented by an ellipse in the figure). . Based on the data sheet and the processing diagram of the processing program, the system administrator can know which data sheets a processing program will obtain data from and which data sheets to output.

FIG. 5 is a flow chart of a data management method according to an embodiment of the invention.

Referring to FIG. 5, in step S501, a data entity is obtained.

In step S502, metadata is obtained. Metadata can be used as a data dictionary to explain the content of each data entity.

In step S503, the metadata is imported.

In step S504, a material operation language is obtained. The data manipulation language can be a SQL script.

In step S505, the material operation language is parsed.

In step S506, a data relationship is established.

In step S507, the data relationship is remitted.

In step S508, a material history is obtained. Specifically, after the complete data history is established, the material management apparatus 100 can trace the source of the data from the bottom layer to the upper layer.

In step S509, an impact analysis is obtained. Specifically, after the data of the upper layer is changed, the data management apparatus 100 can determine from the data relationship which lower layer data is affected by the change of the upper layer data.

In the first embodiment, the number of data sources and data processing programs is quite large, so the connection of the data sheet relationship diagram is very complicated and it is difficult to clarify the relationship of the data fields. Therefore, the second embodiment is proposed below to improve the deficiencies of the first embodiment.

[Second embodiment]

FIG. 6 is a flowchart of a data management method according to an embodiment of the present invention. FIG. 7 is a diagram of a node scatter relationship of a data entity according to an embodiment of the invention. 8 is a schematic diagram of an outlier node in accordance with an embodiment of the present invention.

Referring to FIG. 6, steps S601 to S606 are similar to steps S501 to S506 of FIG. 5, and therefore are not described again.

In step S607, the data relationship is extracted using the network analysis. Specifically, the data management device 100 can establish a node relationship diagram of the data relationship according to the graph cluster algorithm and remit the node relationship graph. The graph clustering algorithm is, for example, a Highly Connected Subgraphs (HCS) clustering algorithm, or other clustering algorithm. Multiple nodes of the node relationship graph correspond to data entities.

In step S608, the data center of the node is established. In particular, the material management apparatus 100 can divide the nodes of the data entity into a plurality of clusters. Taking FIG. 7 as an example, a cluster will have a center point 700 and a sub-cluster 710, a sub-cluster 720, and the sub-cluster 710 will also have a sub-center point 711. The central point 700 can be a node having the highest degree in a cluster. In other words, the center point is a node in a cluster that can be directly connected to the most nodes. Similarly, subcenter point 711 is also the node with the highest height in subcluster 710. After establishing the data center of the node, the administrator can clearly understand the cluster relationship between the data fields to analyze the data characteristics. From the data cluster, managers can also confirm the importance of the data. For example, a node with a higher degree will have a higher importance index in a cluster.

In step S609, a clique and a sub-cluster are obtained. Specifically, "group" is defined as a complete subgraph, that is, any two nodes in a "group" are directly adjacent to each other. When the material management device 100 obtains sub-cluster or group information, it is possible to confirm the closeness of the data relationship in the sub-cluster or group.

In step S610, the location and role of the node are obtained. Specifically, when the material management device 100 generates the node relationship map, the distance between the nodes can be determined according to the degree of relationship between the nodes. The closer the relationship between the two nodes, the shorter the distance in the node graph. Therefore, it is also possible to generate the outlier node 810 and the outlier node 820, as shown in FIG. The outlier node 810 and the outlier node 820 do not have any relationship with other nodes in the node relationship diagram, and therefore there is no edge to connect the outlier node 810 and the outlier node 820 and other nodes.

FIG. 9 is a schematic diagram of hierarchically establishing a data entity relationship diagram according to an embodiment of the invention.

In step S901, a data entity relationship diagram of an Extract Transform Load (ETL) system is established.

In step S902, a data entity relationship table of different ETL execution frequencies is established. Specifically, depending on the business or execution characteristics, the data entities can be divided into different frequencies such as monthly execution, weekly execution, and daily execution. Therefore, the data management apparatus 100 separately establishes a data entity relationship diagram for data entities of different execution frequencies.

In step S903, a data entity relationship diagram of the ETL execution group is established. The ETL execution group belongs to the job (Job) level.

In step S904, a data entity relationship diagram of the ETL execution file is established. ETL execution files (for example, SQL files) belong to the task hierarchy.

In step S905, a data entity relationship diagram of each instruction in the SQL file is established.

In step S906, a data entity relationship diagram of each virtual or entity table in the SQL instruction is established. Each virtual or entity table belongs to a SQL block.

In step S907, a data entity relationship diagram of each target field of the SQL block is established.

In step S908, a data entity relationship diagram of each source field of the SQL block is established.

In summary, the data management apparatus and the data management method of the present invention analyze the data operation language to establish a data relationship of the data entity and reuse the graph cluster algorithm to image the data relationship of the data entity, thereby displaying the data. Information such as the cluster and the central point of the node, and the importance index of the data entity can be obtained from the node relationship diagram of the data relationship.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

100‧‧‧Data management device

110‧‧‧ processor

120‧‧‧ memory

130‧‧‧Communication interface

200‧‧‧Information entity

210, 310, 410‧‧‧Information Sheet

220‧‧‧Information field

430‧‧‧Processing program

S501~S509‧‧‧Steps for data management methods

Steps of the S601~S610‧‧‧ data management method

700‧‧‧ center point

710, 720‧‧‧ sub-cluster

711‧‧‧Subcenter points

810, 820‧‧‧ outlier nodes

S901~S908‧‧‧Steps for layering data entity relationship diagram

1 is a block diagram of a material management apparatus in accordance with an embodiment of the present invention. 2 is a schematic diagram showing the relationship between a data manipulation language and a data entity according to an embodiment of the invention. FIG. 3 is a schematic diagram showing the relationship of a data table according to an embodiment of the present invention. 4 is a schematic diagram showing the relationship between a data table and a processing program according to an embodiment of the invention. FIG. 5 is a flow chart of a data management method according to an embodiment of the invention. FIG. 6 is a flowchart of a data management method according to an embodiment of the present invention. FIG. 7 is a diagram of a node scatter relationship of a data entity according to an embodiment of the invention. 8 is a schematic diagram of an outlier node in accordance with an embodiment of the present invention. FIG. 9 is a schematic diagram of hierarchically establishing a data entity relationship table according to an embodiment of the invention.

Claims (9)

A data management device includes: a processor; and a memory coupled to the processor and storing a plurality of data entities, wherein the processor obtains the data entities and imports the corresponding data entities a plurality of meta-data; parse a Data Manipulation Language (DML) to establish a data relationship of the data entities; and establish a node relationship diagram of the data relationship according to a graph clustering algorithm, The plurality of nodes of the node relationship graph correspond to the data entities; and the node relationship graph is exported. The data management device of claim 1, wherein the processor parses the data operation language to establish the data relationship of the plurality of hierarchical data entities, wherein the hierarchical levels comprise a system level and a working level , a task level, an instruction level, a data level, and a column level. The data management device of claim 1, wherein the processor divides the node relationship graph into at least one cluster according to the graph cluster algorithm, and each of the at least one cluster includes a center point, where the center point is Each of the at least one group is centralized with a node having the highest degree. The data management device of claim 1, wherein the node relationship diagram is a field relationship diagram of a plurality of data fields of each of the data entities. The data management device of claim 1, wherein the processor calculates an importance index of the data entities corresponding to the nodes according to the degrees of the nodes of the node relationship graph. A data management method, applicable to a data management device, the data management device includes a processor and a memory coupled to the processor and storing a plurality of data entities, the data management method comprising: obtaining the The data entities are merged into a plurality of metadata corresponding to the data entities; the processor analyzes a data operation language to establish a data relationship of the data entities; and the processor performs a cluster algorithm according to a graph And establishing a node relationship diagram of the data relationship, wherein the plurality of nodes of the node relationship graph correspond to the data entities; and the node relationship diagram is remitted by the processor. The data management method of claim 6, further comprising: parsing the data operation language by the processor to establish the data relationship of the plurality of hierarchical data entities, wherein the hierarchical levels comprise a system level , a work level, a task level, an instruction level, a data level, and a column level. The data management method of claim 6, further comprising: dividing, by the processor, the node relationship graph into at least one cluster according to the graph cluster algorithm, each of the at least one cluster including a center point, The center point is a node having the highest degree for each of the at least one group. The data management method of claim 6, wherein the node relationship diagram is a field relationship diagram of a plurality of data fields of each of the data entities. The data management method of claim 6, further comprising: calculating, by the processor, an importance index of the data entities corresponding to the nodes according to degrees of the nodes of the node relationship graph.